Xen project Mailing List

On 18 February 2013 12:50, Ian Campbell <ijc@xxxxxxxxxxxxxx> wrote:

On Mon, 2013-02-18 at 12:04 +0200, Gavin wrote:

> Firstly I apologise for the cross-post,

I've added xen-users since you also bounced this there.

Thanks. :-/ Thanks too for your quick reply.

> however I don't expect to get as quick a response from the package
> maintainers as I do from the Debian community, and this issue affects
> a service that I've got scheduled to go live at midnight this
> evening. :(
>
>
> A recent update from xen-hypervisor-4.1-amd64 version 4.1.3-7, to
> version 4.1.3-8 on Debian Wheezy has caused all vm's on this host to
> not receive their arp replies anymore and as such they cannot reach
> their gateways and are now isolated from the network.
>
>
> There was a more recent update as well (4.1.4-2) which I have now
> since applied however this particular issue persists.

Networking level stuff is all done by the dom0 (or driver domain) kernel
rather than the hypervisor so it is far more likely that a kernel level
change rather than a hypervisor change would be responsible. What kernel
version are you running? Did it also change?

This makes sense, although when I did the apt-get upgrade, there was no kernel update, however there may have been packages/drivers that required a kernel mod.

Here is the apt history which details what was upgraded when this broke:-

Upgrade: dnsmasq-base:amd64 (2.62-3, 2.62-3+deb7u1), tasksel-data:amd64 (3.14, 3.14+nmu1), xen-hypervisor-4.1-amd64:amd64 (4.1.3-7, 4.1.3-8), xen-utils-common:amd64 (4.1.3-7, 4.1.3-8), perl:amd64 (5.14.2-16, 5.14.2-17), firmware-linux-free:amd64 (3.1, 3.2), perl-base:amd64 (5.14.2-16, 5.14.2-17), xen-utils-4.1:amd64 (4.1.3-7, 4.1.3-8), libgnutls26:amd64 (2.12.20-2, 2.12.20-4), perl-modules:amd64 (5.14.2-16, 5.14.2-17), psmisc:amd64 (22.19-1, 22.19-1+deb7u1), python2.6:amd64 (2.6.8-0.2, 2.6.8-1.1), libxenstore3.0:amd64 (4.1.3-7, 4.1.3-8), python2.6-minimal:amd64 (2.6.8-0.2, 2.6.8-1.1), coreutils:amd64 (8.13-3.4, 8.13-3.5), libvirt0:amd64 (0.9.12-5, 0.9.12-6), libcurl3:amd64 (7.26.0-1, 7.26.0-1+wheezy1), manpages:amd64 (3.42-1, 3.44-1), tasksel:amd64 (3.14, 3.14+nmu1), libperl5.14:amd64 (5.14.2-16, 5.14.2-17), libsystemd-login0:amd64 (44-7, 44-8), libxen-4.1:amd64 (4.1.3-7, 4.1.3-8), libcurl3-gnutls:amd64 (7.26.0-1, 7.26.0-1+wheezy1), host:amd64 (9.8.4.dfsg.P1-1, 9.8.4.dfsg.P1-4), libvirt-bin:amd64 (0.9.12-5, 0.9.12-6), rinse:amd64 (2.0-1, 2.0.1-1), ca-certificates:amd64 (20120623, 20130119), xenstore-utils:amd64 (4.1.3-7, 4.1.3-8)

The kernel I am using is: 3.2.0-2-amd64, also tried 3.2.0-4-amd64 on another host with no success.

Would the upgrade above of xen-hypervisor-4.1-amd64 on this Debian system not cause the Dom0 kernel to be changed in any way ??

> The arp replies are received by the host and passed all the way up to
> the bridge (br200) being used by Xen, however they are not seen on the
> vif (vif2.0) created for the particular vm.

Do you have any firewall or ebfilter entries which might have either
been discarded or reintroduced by the reboot? (i.e. a manual settings
modification which wasn't propagated to the startup scripts). Or perhaps
sysctl tweaks?

Nope, not using iptables/ebtables, this was working 100% until the apt upgrade above.

After this broke I did try and add the following to /etc/sysctl.conf but it made no difference:-

net.bridge.bridge-nf-call-ip6tables = 0

net.bridge.bridge-nf-call-iptables = 0

net.bridge.bridge-nf-call-arptables = 0

It did add iptables rules but made no difference to this issue.

> 1) Please let me know if I should roll-back this particular xen
> update, kernel and all, and what those steps may be, or if this is a
> known issue with a particular workaround that I can apply.

I'd certainly be tempted to try the older kernel, assuming that was also
upgraded. It may even still be installed and in your grub menu already.

The problem is now we are using grub2 and it appears that on boot grub loads a Linux menu, then the Xen Menu with configs in /etc/grub.d/ so I'm battling to figure out how to do this.

I also do not have physical access to this host at the moment so need to set the boot order 'correctly' prior to a reboot.

> 2) Would moving to openvswitch be another possible workaround?

Without knowing what the underlying issue is it is hard to predict
whether it will also affect ovs.

Agreed.

> My config:-

Looks correct to me.

Ian.

Thanks Ian.

Re: [Xen-users] [Pkg-xen-devel] Recent hypervisor update on Debian Wheezy breaks domU networking