Re: [Xen-devel] [Xen-users] vif-bridge errors when creating and destroying dozens of VMs simultaneously

[George Dunlap <george.dunlap@xxxxxxxxxx>]

On 17/05/17 10:45, Roger Pau Monné wrote:
> On Wed, May 17, 2017 at 10:04:40AM +0100, George Dunlap wrote:
>> cc'ing xen-devel & some relevant people
> 
> Please bear with me, my knowledge of iptables is 0.
> 
>> On Tue, May 16, 2017 at 4:21 PM, Antony Saba <awsaba@xxxxxxxxx> wrote:
>>> Hello xen-users,
>>>
>>> We are seeing the following errors repeatedly while trying to create
>>> domains using a script, with the end result that 2 or 3 out of about
>>> 20 VMs fail to start, and there are stale entries in the iptables for
>>> domains that have been destroyed.
>>>
>>>
>>>    2017-05-10 11:45:40 UTC libxl: error:
>>> libxl_exec.c:118:libxl_report_child_exitstatus:
>>> /etc/xen/scripts/vif-bridge remove [18767] exited with error status 4
>>>    2017-05-10 11:50:52 UTC libxl: error:
>>> libxl_exec.c:118:libxl_report_child_exitstatus:
>>> /etc/xen/scripts/vif-bridge offline [1554] exited with error status 4
>>>
>>> I've been testing the following patch of vif-common.sh over the last
>>> day and it appears to resolve the issue.  iptables exits with status 4
>>> when "Another app is currently holding the xtables lock."
> 
> So, an iptables command can fail randomly because there's someone else holding
> an iptables internal lock?
> 
> Isn't there anyway to tell the iptables command to just block until it can get
> the lock? This seems extremely racy, isn't people then forced to use something
> like:
> 
> while true; do
>       iptables <...>
>       if [ $? == 0 ]; then
>               break;
>       elif [ $? != 4 ]; then
>               error ...
>       fi
> done
> 
> When dealing with iptables?

This seems to be a common problem ([1][2][3] come up right away).

The basic solution seems to be to add the '-w' option to have it wait
for the lock.  It does seem like that should be the default though.
Having commands normally run inside of scripts randomly fail unless you
add the special "don't randomly fail" option seems a bit mad.

 -George


[1] https://github.com/kubernetes/kubernetes/issues/7370
[2] https://github.com/docker/for-mac/issues/285
[3]
https://serverfault.com/questions/805718/iptables-another-app-is-currently-holding-the-xtables-lock


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel