Re: [Xen-users] Xen dom0/domU networking performance issue

[Adam Goryachev <mailinglists@xxxxxxxxxxxxxxxxxxxxxx>]

Hi,

Just wondering if anybody had any suggestions or ideas on what I could
try to resolve this issue.

Thanks,
Adam

On 02/06/14 10:11, Adam Goryachev wrote:
> Hi,
>
> I have a group of physical xen servers (dom0) which have been working
> well for some time. However, recently I converted an existing Linux
> machine to a domU and have been having networking issues with this domU.
>
> Specifically, if from a physical machine on the network, I ping the
> domU, then I get normal ping response times, but for some periods
> (assuming when under a heavier load) the ping time is slow, though no
> packets are ever dropped. This impacts all domU's and the dom0 at the
> same time, moving the domU (live migrate) to a different dom0 causes
> the other dom0 to have the same problem (ie, the problem follows the
> domU).
>
> ie,
> 20140529-14:53:18
> x.x.10.9
> 50 pings in the range 0.18 to 0.27ms
>  0.27
>  157.61
>  117.40
>  472.00
>  623.85
>  956.83
>  1360.31
>  1472.61
>  2045.73
>  2528.93
>  2304.32
>  2651.66
>  2853.93
>  -
> -
> -
> The last three pings were lost (ie, more than three seconds, and fping
> didn't wait for them). When testing manually, if I wait long enough,
> no packets are ever lost, but the delay can get very high).
>
> This is the exact same period pinging to the dom0 on the xen bridge
> interface
> x.x.10.34
>  0.16
>  0.16
>  372.27
>  453.27
>  820.93
>  985.35
>  1358.14
>  1747.34
>  2000.70
>  2370.45
>  2660.26
>  2356.98
>  2859.98
>  -
> -
> -
>
> Pinging to other physical network interfaces on the same dom0 is not
> affected.  eg in the same period of time:
> 2997 packets transmitted, 2997 received, 0% packet loss, time 2996002ms
> rtt min/avg/max/mdev = 0.044/0.091/0.410/0.018 ms
> No slow/delayed pings at all on the iSCSI network interface (not
> bridged for xen, not exposed to the domU's
> 2998 packets transmitted, 2998 received, 0% packet loss, time 2997024ms
> rtt min/avg/max/mdev = 0.084/6.858/1232.196/69.792 ms, pipe 2
> Some delayed pings, up to 1.2 seconds as seen here:
> 64 bytes from x.x.10.34: icmp_seq=2211 ttl=64 time=0.294 ms
> 64 bytes from x.x.10.34: icmp_seq=2212 ttl=64 time=0.287 ms
> 64 bytes from x.x.10.34: icmp_seq=2213 ttl=64 time=0.288 ms
> 64 bytes from x.x.10.34: icmp_seq=2214 ttl=64 time=317 ms
> 64 bytes from x.x.10.34: icmp_seq=2215 ttl=64 time=724 ms
> 64 bytes from x.x.10.34: icmp_seq=2216 ttl=64 time=1155 ms
> 64 bytes from x.x.10.34: icmp_seq=2217 ttl=64 time=1232 ms
> 64 bytes from x.x.10.34: icmp_seq=2218 ttl=64 time=1202 ms
> 64 bytes from x.x.10.34: icmp_seq=2219 ttl=64 time=838 ms
> 64 bytes from x.x.10.34: icmp_seq=2220 ttl=64 time=556 ms
> 64 bytes from x.x.10.34: icmp_seq=2221 ttl=64 time=229 ms
> 64 bytes from x.x.10.34: icmp_seq=2222 ttl=64 time=0.266 ms
> 64 bytes from x.x.10.34: icmp_seq=2223 ttl=64 time=0.239 ms
> 64 bytes from x.x.10.34: icmp_seq=2224 ttl=64 time=0.238 ms
> 64 bytes from x.x.10.34: icmp_seq=2225 ttl=64 time=0.265 ms
>
> System details of dom0 (Debian Stable):
> root@pm04:~# uname -an
> Linux pm04 3.2.0-4-amd64 #1 SMP Debian 3.2.57-3+deb7u1 x86_64 GNU/Linux
> root@pm04:~# dpkg -l | grep xen
> ii  libxen-4.1 4.1.4-3+deb7u1               amd64        Public libs
> for Xen
> ii  libxenstore3.0 4.1.4-3+deb7u1               amd64        Xenstore
> communications library for Xen
> ii  xen-hypervisor-4.1-amd64 4.1.4-3+deb7u1               amd64       
> Xen Hypervisor on AMD64
> ii  xen-linux-system-3.2.0-4-amd64 3.2.57-3+deb7u1             
> amd64        Xen system with Linux 3.2 on 64-bit PCs (meta-package)
> ii  xen-linux-system-amd64 3.2+46                       amd64       
> Xen system with Linux for 64-bit PCs (meta-package)
> ii  xen-system-amd64 4.1.4-3+deb7u1               amd64        Xen
> System on AMD64 (meta-package)
> ii  xen-utils-4.1 4.1.4-3+deb7u1               amd64        XEN
> administrative tools
> ii  xen-utils-common 4.1.4-3+deb7u1               all          Xen
> administrative tools - common files
> ii  xenstore-utils 4.1.4-3+deb7u1               amd64        Xenstore
> utilities for Xen
>
> domU config file:
> name        = "mail"
> hostname    = 'mail'
> kernel        = "/mnt/images/server_images/mail/vmlinuz-3.2.0-4-686-pae"
> ramdisk        =
> "/mnt/images/server_images/mail/initrd.img-3.2.0-4-686-pae"
> root        = "/dev/xvda1"
> disk        = ['phy:/dev/mapper/mpathmail,xvda,w' ]
> memory        = 2048
> cpus        = "2,3"
> vcpus        = 2
> vif        = ['bridge=xenbr5, mac=00:16:3e:43:a8:09', 'bridge=xenbr0,
> mac=00:16:3e:43:d8:09']
>
> The dom0 network config:
> cat /etc/network/interfaces
> # The primary network interface
> allow-hotplug eth0
> auto xenbr0
> iface xenbr0 inet static
>     address x.x.10.34
>     netmask 255.255.240.0
>     gateway x.x.10.254
>     bridge_maxwait 5
>     bridge_ports regex eth0
>
> # This is the vlan which has the DSL modem plus the problem domU/VM
> auto xenbr5
> iface xenbr5 inet manual
>     bridge_ports eth0.5
>
> # Used by dom0 for iscsi SAN only
> auto eth1
> iface eth1 inet static
>     address x.x.16.34
>     netmask 255.255.255.0
>     mtu 9000
>     post-up /etc/network/ifup.eth1
>
> # Used by dom0 for iscsi SAN only
> auto eth2
> iface eth2 inet static
>     address x.x.16.44
>     netmask 255.255.255.0
>     mtu 9000
>     post-up /etc/network/ifup.eth2
>
> xm vcpu-list
> Name                                ID  VCPU   CPU State   Time(s) CPU
> Affinity
> Domain-0                             0     0     0   -b-     965.7 0
> Domain-0                             0     1     1   r--     662.4 1
> mail                                 4     0     2   -b-    1861.0 2
> mail                                 4     1     3   -b-    1134.2 3
> oldcal                               5     0     4   -b-      80.9 4-5
> oldcal                               5     1     5   -b-      32.7 4-5
>
> xm network-list mail
> Idx BE     MAC Addr.     handle state evt-ch tx-/rx-ring-ref BE-path
> 0   0  00:16:3e:43:a8:09    0     4      14    768  /769
> /local/domain/0/backend/vif/4/0
> 1   0  00:16:3e:43:d8:09    1     4      15    770  /1280
> /local/domain/0/backend/vif/4/1
>
> The domU is running:
> # uname -an
> Linux mail 3.2.0-4-686-pae #1 SMP Debian 3.2.57-3+deb7u1 i686 GNU/Linux
>
> The domU is the gateway/firewall for the network which routes between
> the internal network and Internet, also runs email/proxy servers etc.
>
> Can anyone suggest what I could do to resolve this issue? Additional
> information I should provide?
>
> Thanks,
> Adam
>


-- 
Adam Goryachev
Website Managers
www.websitemanagers.com.au


_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users