Re: [Xen-users] network problems

[Mike Lovell <mike@xxxxxxxxxxxx>]

I did 'echo 1 > /proc/sys/xen/independent_wallclock' on the host and several of the guests. Then I started the ping flood and the same problem showed up. The warnings about time were not happening every 10 minutes like the networking problem is. But I haven't seen the clock errors since making the change.

mike

Nathan Eisenberg wrote:

Yea, I believe they are.  Take a look at the independant wallclock setting, it should help.
Best Regards,
Nathan Eisenberg
Sr. Systems Administrator
Atlas Networks, LLC

Sent from my BlackBerry

-----Original Message-----
From: Mike Lovell <mike@xxxxxxxxxxxx>

Date: Wed, 22 Jul 2009 17:07:03 
To: Nathan Eisenberg<nathan@xxxxxxxxxxxxxxxx>
Subject: Re: [Xen-users] network problems


All of the mac addresses are unique. During my ping floods, I do what
look like clock errors. They say something like:

Warning: time of day goes back (-27109us), taking countermeasures.

I wasn't sure if these were related so I didn't mention them before. Is
it part of the problem?

mike

Nathan Eisenberg wrote:
  

Have you checked to make sure all of the MAC addresses are unique?  Are there any time/clock went backwards messages on the console/dmesg output?

Best Regards
Nathan Eisenberg
Sr. Systems Administrator
Atlas Networks, LLC
support@xxxxxxxxxxxxxxxx
http://support.atlasnetworks.us/portal


-----Original Message-----
From: xen-users-bounces@xxxxxxxxxxxxxxxxxxx [mailto:xen-users-bounces@xxxxxxxxxxxxxxxxxxx] On Behalf Of Mike Lovell
Sent: Wednesday, July 22, 2009 3:49 PM
To: xen-users@xxxxxxxxxxxxxxxxxxx
Subject: [Xen-users] network problems

So, I have been running into network problems for a while on 4 boxes
that I installed xen on so that some engineers have places to test code.
This particular problem is happening on all 4 of these boxes. (although,
it isn't happening on an older box running xen from debian etch).

What appears to be the problem is that traffic is getting dropped
between the vif#.0 interface in dom0 and the eth0 interface in the
guest. To find this out, I started a ping flood from one domU that was
pinging another domU. About every 10 minutes, there will be a lot of
ping requests going out but no replies coming back. I think it really
weird that it happens like ever 10 minutes plus about 2 seconds. While
the ping was going, I did tcpdumps from the domU starting the ping, from
the vif#.0 of the pinging machine, from the virtual bridge, from the
vif#.0 for the receiving guest, and then from the receiving domU. The
packets are making it all the way to the dom0 vif for the receiving
guest but not making it to the eth0 in the guest. I have no clue why
this is happening and it happens in rather regular intervals. The same
thing happens in pinging a different guest and it happens in about the
same interval but at different times. Also, during the ping flood, there
is never a pause in the sending of packets out of the guest. Only a
pause on the packets going from the host to the guest.

I am running this on 64 bit Debian Lenny using the distribution's
packages. xen-hypervisor-3.2-1-amd64 version 3.2.1-2 and
linux-image-2.6.26-2-xen-amd64 version 2.6.26-17. Here are the
networking configs.

---------
dom0# cat /etc/network/interfaces
# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet static
    address 10.135.7.34
    netmask 255.255.255.224
    network 10.135.7.32
    broadcast 10.135.7.63
    gateway 10.135.7.33
    # dns-* options are implemented by the resolvconf package, if installed
    dns-nameservers 10.135.7.34
    dns-search qa1.mozyops.com

auto vmnet
iface vmnet inet static
        address 10.135.2.71
        netmask 255.255.255.224
        bridge_ports eth1
#        bridge_stp off
#        bridge_fd 9
#        bridge_hello 2
#        bridge_maxage 12

---------
DomU# cat /etc/network/interfaces
# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet dhcp
   post-up ethtool -K eth0 tx off

---------
Dom0# brctl show vmnet
bridge name    bridge id        STP enabled    interfaces
vmnet        8000.003048c8166d    no        eth1
                            vif1.0
                            vif10.0
                            <other interfaces>
---------

Does anyone have any ideas as to what is going on here? Or more
importantly, any ideas on how to solve this? I have tried building a
newer domU kernel from scratch but I haven't been able to make any
progress there. The guest fails to boot without showing anything on the
console. It then goes into this loop of trying to reboot the guest but
failing. I would really like to stay with the debian kernels.

I have been banging my head against a wall for a week or so on this and
desperately need some help to get this working. I have engineers that
are getting held up by this bug.

Thanks for any insight you guys can give.

mike

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users





    

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users