|
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] Re: [Xen-API] XCP 1.5 BETA BUG: Slow performance with 10Gbit network cards
Did you try to disable all power saving functions in bios? Power saving has a big impact on interrupt handling :( ________________________________________ Von: Sébastien Riccio [sr@xxxxxxxxxxxxxxx] Gesendet: Montag, 30. April 2012 10:38 Bis: Uli Stärk; xen-api@xxxxxxxxxxxxxxxxxxx Betreff: Re: AW: [Xen-API] XCP 1.5 BETA BUG: Slow performance with 10Gbit network cards Hi, Yes I've almost tried everything that is on the network tweaking guide: http://wiki.xen.org/wiki/Network_Throughput_and_Performance_Guide But the bottleneck seems to be the xen hypervisor used in xcp and xenserver. On the same machine but with xcp-xapi (over debian) I get almost normal 10gbit/s speeds from dom0. What is kinda relevant is that even doing an iperf "localhost to localhost" it's stuck to that 3gbit in a xcp or xenserver dom0... XCP: [root@xen-blade13 ~]# iperf -c localhost ------------------------------------------------------------ Client connecting to localhost, TCP port 5001 ------------------------------------------------------------ [ 3] local 127.0.0.1 port 43138 connected with 127.0.0.1 port 5001 [ 3] 0.0-10.0 sec 3.43 GBytes 2.94 Gbits/sec Kronos (xapi on debian) root@xen-blade11:~# iperf -c localhost ------------------------------------------------------------ Client connecting to localhost, TCP port 5001 ------------------------------------------------------------ [ 3] local 127.0.0.1 port 34438 connected with 127.0.0.1 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 18.9 GBytes 16.3 Gbits/sec So I'll never get more than 3gbit/s in one thread on xenserver or xcp... and that is kinda ... bad :/ Not sure yet if it's a limitation of the dom0 kernel or the hypervisor itself... I tend to think it's the hypervisor. xcp 1.5 host : xen-blade13 release : 2.6.32.12-0.7.1.xs1.4.90.530.170661xen version : #1 SMP Sat Apr 28 21:26:23 CEST 2012 machine : i686 nr_cpus : 16 nr_nodes : 2 cores_per_socket : 4 threads_per_core : 2 cpu_mhz : 2394 hw_caps : bfebfbff:2c100800:00000000:00003f40:029ee3ff:00000000:00000001:00000000 virt_caps : hvm hvm_directio total_memory : 32758 free_memory : 31516 free_cpus : 0 xen_major : 4 xen_minor : 1 xen_extra : .1 xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 xen_scheduler : credit xen_pagesize : 4096 platform_params : virt_start=0xfdc00000 xen_changeset : unavailable xen_commandline : dom0_mem=752M lowmem_emergency_pool=1M crashkernel=64M@32M console= vga=mode-0x0311 dom0_max_vcpus=8 cc_compiler : gcc version 4.1.2 20080704 (Red Hat 4.1.2-46) cc_compile_by : root cc_compile_domain : uk.xensource.com cc_compile_date : Mon Feb 6 19:01:42 EST 2012 xend_config_format : 4 xcp-xapi (debian) host : xen-blade11 release : 3.2.0-2-686-pae version : #1 SMP Sun Apr 15 17:56:31 UTC 2012 machine : i686 nr_cpus : 8 nr_nodes : 2 cores_per_socket : 4 threads_per_core : 1 cpu_mhz : 2394 hw_caps : bfebfbff:2c100800:00000000:00003f40:029ee3ff:00000000:00000001:00000000 virt_caps : hvm hvm_directio total_memory : 32758 free_memory : 30560 free_cpus : 0 xen_major : 4 xen_minor : 1 xen_extra : .2 xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 xen_scheduler : credit xen_pagesize : 4096 platform_params : virt_start=0xfdc00000 xen_changeset : unavailable xen_commandline : placeholder dom0_mem=1024M cc_compiler : gcc version 4.6.2 (Debian 4.6.2-6) cc_compile_by : waldi cc_compile_domain : debian.org cc_compile_date : Sun Dec 11 13:32:25 UTC 2011 xend_config_format : 4 I see not much difference except 4.1.1 for xcp 4.1.2 for xcp-xapi with same scheduler (credit) Also I found an old post (2008) of someone having the same symptoms back in the years: http://old-list-archives.xen.org/archives/html/xen-users/2008-12/msg00508.html but didn't find any follow ups ... Cheers, Sébastien On 30.04.2012 10:13, Uli Stärk wrote: > did you enable jumbo frames? > > ip link set eth2 mtu 9000 > > > ________________________________________ > Von: xen-api-bounces@xxxxxxxxxxxxx [xen-api-bounces@xxxxxxxxxxxxx]" im > Auftrag von"Sébastien Riccio [sr@xxxxxxxxxxxxxxx] > Gesendet: Sonntag, 29. April 2012 19:23 > Bis: xen-api@xxxxxxxxxxxxxxxxxxx > Betreff: Re: [Xen-API] XCP 1.5 BETA BUG: Slow performance with 10Gbit network > cards > > Another discovery: > > On XCP 1.5 > > iperf on dom0 to localhost (loopback) > > Client connecting to localhost, TCP port 5001 > TCP window size: 64.0 KByte (default) > ------------------------------------------------------------ > [ 3] local 127.0.0.1 port 38278 connected with 127.0.0.1 port 5001 > ^C[ 3] 0.0- 5.2 sec 1.79 GBytes 2.97 Gbits/sec > > Limited to 3gbit > > on xapi/debian > > root@kronos:~# iperf -c 10.50.50.111 > ------------------------------------------------------------ > Client connecting to 10.50.50.111, TCP port 5001 > TCP window size: 167 KByte (default) > ------------------------------------------------------------ > [ 3] local 127.0.0.1 port 41137 connected with 127.0.0.1 port 5001 > [ ID] Interval Transfer Bandwidth[ 3] 0.0- 5.6 sec 13.1 > GBytes 20.0 Gbits/sec > > clearly not a nic issue.... > > Cheers, > Sébastien > > > > > > > On 29.04.2012 17:28, Sébastien Riccio wrote: >> Me again (sorry), >> >> Still trying to sort things out to understand why it's slow under xcp, >> I've installed xcp on debian (project kronos), and did an iperf from dom0 >> >> [ 3] 0.0- 1.0 sec 623 MBytes 5.23 Gbits/sec >> [ 3] 1.0- 2.0 sec 817 MBytes 6.85 Gbits/sec >> [ 3] 2.0- 3.0 sec 818 MBytes 6.86 Gbits/sec >> [ 3] 3.0- 4.0 sec 816 MBytes 6.84 Gbits/sec >> [ 3] 4.0- 5.0 sec 818 MBytes 6.86 Gbits/sec >> [ 3] 5.0- 6.0 sec 818 MBytes 6.86 Gbits/sec >> [ 3] 6.0- 7.0 sec 816 MBytes 6.85 Gbits/sec >> >> It's not getting to the 10gibt/s so it looks that it's related to the >> xen hypervisor. >> But it still 2x better than with xcp/xenserver. >> >> Any ideas how to improve this? >> >> Cheers, >> Sébastien >> >> On 29.04.2012 14:44, Sébastien Riccio wrote: >>> Hi again, again :) >>> >>> Something strange : >>> >>> If i issue parallel threads with iperf (8 in this example, the >>> 10Gbit/s speed is reached) >>> >>> [ 5] 0.0- 1.0 sec 151 MBytes 1.27 Gbits/sec >>> [ 6] 0.0- 1.0 sec 178 MBytes 1.49 Gbits/sec >>> [ 7] 0.0- 1.0 sec 136 MBytes 1.14 Gbits/sec >>> [ 8] 0.0- 1.0 sec 189 MBytes 1.59 Gbits/sec >>> [ 10] 0.0- 1.0 sec 161 MBytes 1.35 Gbits/sec >>> [ 4] 0.0- 1.0 sec 124 MBytes 1.04 Gbits/sec >>> [ 3] 0.0- 1.0 sec 178 MBytes 1.49 Gbits/sec >>> [ 9] 0.0- 1.0 sec 74.8 MBytes 627 Mbits/sec >>> [SUM] 0.0- 1.0 sec 1.16 GBytes 10.0 Gbits/sec >>> >>> So it's surely not a NIC/driver issue but maybe a limitation in dom0 ? >>> It's really not efficient as we connect the iscsi storage to the >>> filer from dom0, so it will always be a single thread, so no >>> possibility to get the full 10Gbit bw for our storage. >>> >>> Is there a way to get rid of that (bug?) limitation ? >>> >>> Cheers, >>> Sébastien >>> >>> >>> >>> >>> On 29.04.2012 14:23, Sébastien Riccio wrote: >>>> Hi again, >>>> >>>> So I did my test with a debian 6 on the same hardware than I'm using >>>> to try the xcp 1.5. >>>> >>>> Speed to the filer via the 10gig nic: >>>> >>>> [ 3] 0.0- 8.0 sec 8.71 GBytes 9.41 Gbits/sec >>>> >>>> I would tend to conclude that there is something wrong with the >>>> XCP/XenServer kernel and the 10gigs interfaces (at least broadcom). >>>> >>>> (Note that our Dell hardware is certified as supported by XenServer, >>>> so it seems not so true :/) >>>> >>>> modinfo for bnx2x on debian 6 >>>> >>>> filename: /lib/modules/2.6.32-5-amd64/kernel/drivers/net/bnx2x.ko >>>> firmware: bnx2x-e1h-5.0.21.0.fw >>>> firmware: bnx2x-e1-5.0.21.0.fw >>>> version: 1.52.1 >>>> license: GPL >>>> description: Broadcom NetXtreme II BCM57710/57711/57711E Driver >>>> author: Eliezer Tamir >>>> srcversion: 050FF749F80E43CB5873BC9 >>>> alias: pci:v000014E4d00001650sv*sd*bc*sc*i* >>>> alias: pci:v000014E4d0000164Fsv*sd*bc*sc*i* >>>> alias: pci:v000014E4d0000164Esv*sd*bc*sc*i* >>>> depends: mdio,libcrc32c >>>> vermagic: 2.6.32-5-amd64 SMP mod_unload modversions >>>> parm: multi_mode: Multi queue mode (0 Disable; 1 Enable >>>> (default)) (int) >>>> parm: num_rx_queues: Number of Rx queues for multi_mode=1 >>>> (default is half number of CPUs) (int) >>>> parm: num_tx_queues: Number of Tx queues for multi_mode=1 >>>> (default is half number of CPUs) (int) >>>> parm: disable_tpa: Disable the TPA (LRO) feature (int) >>>> parm: int_mode: Force interrupt mode (1 INT#x; 2 MSI) (int) >>>> parm: dropless_fc: Pause on exhausted host ring (int) >>>> parm: poll: Use polling (for debug) (int) >>>> parm: mrrs: Force Max Read Req Size (0..3) (for debug) (int) >>>> parm: debug: Default debug msglevel (int) >>>> >>>> modinfo for bnx2x on xcp 1.5 >>>> >>>> [root@xen-blade13 ~]# modinfo bnx2x >>>> filename: >>>> /lib/modules/2.6.32.12-0.7.1.xs1.4.90.530.170661xen/kernel/drivers/net/bnx2x.ko >>>> version: 1.62.17 >>>> license: GPL >>>> description: Broadcom NetXtreme II >>>> BCM57710/57711/57711E/57712/57712E Driver >>>> author: Eliezer Tamir >>>> srcversion: 13EA885EBE061121BAEC28D >>>> alias: pci:v000014E4d00001663sv*sd*bc*sc*i* >>>> alias: pci:v000014E4d00001662sv*sd*bc*sc*i* >>>> alias: pci:v000014E4d00001650sv*sd*bc*sc*i* >>>> alias: pci:v000014E4d0000164Fsv*sd*bc*sc*i* >>>> alias: pci:v000014E4d0000164Esv*sd*bc*sc*i* >>>> depends: mdio >>>> vermagic: 2.6.32.12-0.7.1.xs1.4.90.530.170661xen SMP >>>> mod_unload modversions Xen 686 >>>> parm: multi_mode: Multi queue mode (0 Disable; 1 Enable >>>> (default); 2 VLAN PRI; 3 E1HOV PRI; 4 IP DSCP) (int) >>>> parm: pri_map: Priority to HW queue mapping (int) >>>> parm: qs_per_cos: Number of queues per HW queue (int) >>>> parm: cos_min_rate: Weight for RR between HW queues (int) >>>> parm: num_queues: Number of queues for multi_mode=1 >>>> (default is as a number of CPUs) (int) >>>> parm: disable_iscsi_ooo: Disable iSCSI OOO support (int) >>>> parm: disable_tpa: Disable the TPA (LRO) feature (int) >>>> parm: int_mode: Force interrupt mode other than MSI-X (1 >>>> INT#x; 2 MSI) (int) >>>> parm: dropless_fc: Pause on exhausted host ring (int) >>>> parm: poll: Use polling (for debug) (int) >>>> parm: mrrs: Force Max Read Req Size (0..3) (for debug) (int) >>>> parm: debug: Default debug msglevel (int) >>>> >>>> Seems that the xcp 1.5 got a more recent version of the module. So >>>> that might not be the problem. >>>> Maybe another problem in the kernel ? >>>> >>>> Any ideas how I could debug this ? >>>> >>>> Cheers, >>>> Sébastien >>>> >>>> >>>> On 29.04.2012 13:23, Sébastien Riccio wrote: >>>>> Hi, >>>>> >>>>> I'm currently playing with XCP 1.5 beta on a Dell Blade >>>>> infrastructure that is linked to redundent filers with 10gbits NICs. >>>>> >>>>> The redundent 10gigs interfaces are not managed by xapi (i've told >>>>> xapi to forget them) and are configured like on any normal linux box. >>>>> >>>>> They are connected to the same 10gig switch where the filers are >>>>> connected to. >>>>> >>>>> Interfaces on the filers: >>>>> 03:00.0 Ethernet controller: Broadcom Corporation NetXtreme II >>>>> BCM57711 10-Gigabit PCIe >>>>> 03:00.1 Ethernet controller: Broadcom Corporation NetXtreme II >>>>> BCM57711 10-Gigabit PCIe >>>>> 04:00.0 Ethernet controller: Broadcom Corporation NetXtreme II >>>>> BCM57711 10-Gigabit PCIe >>>>> 04:00.1 Ethernet controller: Broadcom Corporation NetXtreme II >>>>> BCM57711 10-Gigabit PCIe >>>>> >>>>> Interfaces on the blades: >>>>> 05:00.0 Ethernet controller: Broadcom Corporation NetXtreme II >>>>> BCM57711 10-Gigabit PCIe >>>>> 05:00.1 Ethernet controller: Broadcom Corporation NetXtreme II >>>>> BCM57711 10-Gigabit PCIe >>>>> >>>>> The problem is that after some testing with iperf i get to this >>>>> conclusion: >>>>> >>>>> 1) Between the filers the speed is 9.9Gbit/s. >>>>> 2) Between xcp 1.5 and both filer i get around 3Gbit/s which is >>>>> quite poor >>>>> 2) Between an xcp 1.1 and both filer i get around 3Gbit/s which is >>>>> quite poor (so that's not a new 1.5 bug) >>>>> 3) Between an xcp 1.5 and another xcp (1.1) i get too around 3Gbit/s. >>>>> >>>>> Do you have any ideas what could be the problem ? knowing that >>>>> between the filers the speed is good and it's passing through the >>>>> same switch. >>>>> >>>>> Is there a DDK VM available for XCP 1.5 beta? I would like to try >>>>> to build the latest Broadcom drivers for these nics (bnx2x) for the >>>>> kernel used on xcp 1.5... >>>>> >>>>> In the meantime I will try to install a debian on another blade to >>>>> check if the speed are correct with the filers. >>>>> >>>>> Thanks, >>>>> Sébastien >>>>> >>>>> >>>>> _______________________________________________ >>>>> Xen-api mailing list >>>>> Xen-api@xxxxxxxxxxxxx >>>>> http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api >>>>> >>>> >>>> _______________________________________________ >>>> Xen-api mailing list >>>> Xen-api@xxxxxxxxxxxxx >>>> http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api >>>> >>> >>> _______________________________________________ >>> Xen-api mailing list >>> Xen-api@xxxxxxxxxxxxx >>> http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api >>> >> >> _______________________________________________ >> Xen-api mailing list >> Xen-api@xxxxxxxxxxxxx >> http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api >> > > _______________________________________________ > Xen-api mailing list > Xen-api@xxxxxxxxxxxxx > http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api > _______________________________________________ Xen-api mailing list Xen-api@xxxxxxxxxxxxx http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api
|
 |
Lists.xenproject.org is hosted with RackSpace, monitoring our |