Re: [Xen-API] XCP 1.5 BETA BUG: Slow performance with 10Gbit network cards

[Sébastien Riccio <sr@xxxxxxxxxxxxxxx>]

Hi again, again :)

Something strange :

If i issue parallel threads with iperf (8 in this example, the 10Gbit/s 
speed is reached)
[  5]  0.0- 1.0 sec    151 MBytes  1.27 Gbits/sec
[  6]  0.0- 1.0 sec    178 MBytes  1.49 Gbits/sec
[  7]  0.0- 1.0 sec    136 MBytes  1.14 Gbits/sec
[  8]  0.0- 1.0 sec    189 MBytes  1.59 Gbits/sec
[ 10]  0.0- 1.0 sec    161 MBytes  1.35 Gbits/sec
[  4]  0.0- 1.0 sec    124 MBytes  1.04 Gbits/sec
[  3]  0.0- 1.0 sec    178 MBytes  1.49 Gbits/sec
[  9]  0.0- 1.0 sec  74.8 MBytes    627 Mbits/sec
[SUM]  0.0- 1.0 sec  1.16 GBytes  10.0 Gbits/sec

So it's surely not a NIC/driver issue but maybe a limitation in dom0 ?
It's really not efficient as we connect the iscsi storage to the filer 
from dom0, so it will always be a single thread, so no possibility to 
get the full 10Gbit bw for our storage.
Is there a way to get rid of that (bug?) limitation ?

Cheers,
Sébastien




On 29.04.2012 14:23, Sébastien Riccio wrote:
> Hi again,
>
> So I did my test with a debian 6 on the same hardware than I'm using 
> to try the xcp 1.5.
> Speed to the filer via the 10gig nic:
>
> [  3]  0.0- 8.0 sec  8.71 GBytes  9.41 Gbits/sec
>
> I would tend to conclude that there is something wrong with the 
> XCP/XenServer kernel and the 10gigs interfaces (at least broadcom).
> (Note that our Dell hardware is certified as supported by XenServer, 
> so it seems not so true :/)
> modinfo for bnx2x on debian 6
>
> filename:       /lib/modules/2.6.32-5-amd64/kernel/drivers/net/bnx2x.ko
> firmware:       bnx2x-e1h-5.0.21.0.fw
> firmware:       bnx2x-e1-5.0.21.0.fw
> version:        1.52.1
> license:        GPL
> description:    Broadcom NetXtreme II BCM57710/57711/57711E Driver
> author:         Eliezer Tamir
> srcversion:     050FF749F80E43CB5873BC9
> alias:          pci:v000014E4d00001650sv*sd*bc*sc*i*
> alias:          pci:v000014E4d0000164Fsv*sd*bc*sc*i*
> alias:          pci:v000014E4d0000164Esv*sd*bc*sc*i*
> depends:        mdio,libcrc32c
> vermagic:       2.6.32-5-amd64 SMP mod_unload modversions
> parm:           multi_mode: Multi queue mode (0 Disable; 1 Enable 
> (default)) (int)
> parm:           num_rx_queues: Number of Rx queues for multi_mode=1 
> (default is half number of CPUs) (int)
> parm:           num_tx_queues: Number of Tx queues for multi_mode=1 
> (default is half number of CPUs) (int)
> parm:           disable_tpa: Disable the TPA (LRO) feature (int)
> parm:           int_mode: Force interrupt mode (1 INT#x; 2 MSI) (int)
> parm:           dropless_fc: Pause on exhausted host ring (int)
> parm:           poll: Use polling (for debug) (int)
> parm:           mrrs: Force Max Read Req Size (0..3) (for debug) (int)
> parm:           debug: Default debug msglevel (int)
>
> modinfo for bnx2x on xcp 1.5
>
> [root@xen-blade13 ~]# modinfo bnx2x
> filename:       
> /lib/modules/2.6.32.12-0.7.1.xs1.4.90.530.170661xen/kernel/drivers/net/bnx2x.ko
> version:        1.62.17
> license:        GPL
> description:    Broadcom NetXtreme II 
> BCM57710/57711/57711E/57712/57712E Driver
> author:         Eliezer Tamir
> srcversion:     13EA885EBE061121BAEC28D
> alias:          pci:v000014E4d00001663sv*sd*bc*sc*i*
> alias:          pci:v000014E4d00001662sv*sd*bc*sc*i*
> alias:          pci:v000014E4d00001650sv*sd*bc*sc*i*
> alias:          pci:v000014E4d0000164Fsv*sd*bc*sc*i*
> alias:          pci:v000014E4d0000164Esv*sd*bc*sc*i*
> depends:        mdio
> vermagic:       2.6.32.12-0.7.1.xs1.4.90.530.170661xen SMP mod_unload 
> modversions Xen 686
> parm:           multi_mode: Multi queue mode (0 Disable; 1 Enable 
> (default); 2 VLAN PRI; 3 E1HOV PRI; 4 IP DSCP) (int)
> parm:           pri_map: Priority to HW queue mapping (int)
> parm:           qs_per_cos: Number of queues per HW queue (int)
> parm:           cos_min_rate: Weight for RR between HW queues (int)
> parm:           num_queues: Number of queues for multi_mode=1 (default 
> is as a number of CPUs) (int)
> parm:           disable_iscsi_ooo: Disable iSCSI OOO support (int)
> parm:           disable_tpa: Disable the TPA (LRO) feature (int)
> parm:           int_mode: Force interrupt mode other than MSI-X (1 
> INT#x; 2 MSI) (int)
> parm:           dropless_fc: Pause on exhausted host ring (int)
> parm:           poll: Use polling (for debug) (int)
> parm:           mrrs: Force Max Read Req Size (0..3) (for debug) (int)
> parm:           debug: Default debug msglevel (int)
>
> Seems that the xcp 1.5 got a more recent version of the module. So 
> that might not be the problem.
> Maybe another problem in the kernel ?
>
> Any ideas how I could debug this ?
>
> Cheers,
> Sébastien
>
>
> On 29.04.2012 13:23, Sébastien Riccio wrote:
> > Hi,
> >
> > I'm currently playing with XCP 1.5 beta on a Dell Blade 
> > infrastructure that is linked to redundent filers with 10gbits NICs.
> > The redundent 10gigs interfaces are not managed by xapi (i've told 
> > xapi to forget them) and are configured like on any normal linux box.
> > They are connected to the same 10gig switch where the filers are 
> > connected to.
> > Interfaces on the filers:
> > 03:00.0 Ethernet controller: Broadcom Corporation NetXtreme II 
> > BCM57711 10-Gigabit PCIe
> > 03:00.1 Ethernet controller: Broadcom Corporation NetXtreme II 
> > BCM57711 10-Gigabit PCIe
> > 04:00.0 Ethernet controller: Broadcom Corporation NetXtreme II 
> > BCM57711 10-Gigabit PCIe
> > 04:00.1 Ethernet controller: Broadcom Corporation NetXtreme II 
> > BCM57711 10-Gigabit PCIe
> > Interfaces on the blades:
> > 05:00.0 Ethernet controller: Broadcom Corporation NetXtreme II 
> > BCM57711 10-Gigabit PCIe
> > 05:00.1 Ethernet controller: Broadcom Corporation NetXtreme II 
> > BCM57711 10-Gigabit PCIe
> > The problem is that after some testing with iperf i get to this 
> > conclusion:
> > 1) Between the filers the speed is 9.9Gbit/s.
> > 2) Between xcp 1.5 and both filer i get around 3Gbit/s which is quite 
> > poor
> > 2) Between an xcp 1.1 and both filer i get around 3Gbit/s which is 
> > quite poor (so that's not a new 1.5 bug)
> > 3) Between an xcp 1.5 and another xcp (1.1) i get too around 3Gbit/s.
> >
> > Do you have any ideas what could be the problem ? knowing that 
> > between the filers the speed is good and it's passing through the 
> > same switch.
> > Is there a DDK VM available for XCP 1.5 beta? I would like to try to 
> > build the latest Broadcom drivers for these nics (bnx2x) for the 
> > kernel used on xcp 1.5...
> > In the meantime I will try to install a debian on another blade to 
> > check if the speed are correct with the filers.
> > Thanks,
> > Sébastien
> >
> >
> > _______________________________________________
> > Xen-api mailing list
> > Xen-api@xxxxxxxxxxxxx
> > http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api
>
> _______________________________________________
> Xen-api mailing list
> Xen-api@xxxxxxxxxxxxx
> http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api

_______________________________________________
Xen-api mailing list
Xen-api@xxxxxxxxxxxxx
http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api