Re: [Xen-devel] Significant network performance difference when different NIC MQ handles the packet

["Zhangleiqiang (Trump)" <zhangleiqiang@xxxxxxxxxx>]

> On 31/12/14 09:06, Zhangleiqiang (Trump) wrote:
> > Hi, all:
> >
> >     I have used pkt-gen to send udp packet (1400 bytes) from one Dom0 A to
> another Dom0 B each of which is connected by 10GE network. On the receive
> side (B)，different "ksoftirqd/x" processes will handle the packet during each
> testing because the nic has multiple queues. I find that if "ksoftirqd/0" 
> handles
> the packet, the throughout can reach to about 7Gbps. However, when
> "ksoftirqd/2" handles the packet, the throughout can only reach to about
> 2Gbps, and for "ksoftirqd/3", the throughout can even as low as 1Gbps.
> >
> >     I am wondering what the reason it is. Could anyone give me some hint
> about it ?
> NUMA topology? Your card might use memory "close" to CPU core 0, but not
> to the others?

Thanks for your hint. 

I am not sure if the NUMA topology is the reason for it. Because my server has 
two nodes, each of which has 8 cpus. The result of "numctl --hardware" is as 
follows:

available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7
node 0 size: 32090 MB
node 0 free: 31640 MB
node 1 cpus: 8 9 10 11 12 13 14 15
node 1 size: 32311 MB
node 1 free: 31643 MB
node distances:
node   0   1 
  0:  10  20 
  1:  20  10

I booted dom0 with "dom0_max_vcpus=6 dom0_vcpu_pin" command line. So I think 
"ksoftirqd/x" will running on cpu "x", e.g. ksoftirqd/0 and ksortirqd/2 are 
running on cpu0 and cpu2 respectively. If it is so, ksoftirqd/0 and ksortirqd/2 
should have the "same distance" to the memory which NIC card uses.

> Zoltan
> 
> >
> > ----------
> > zhangleiqiang (Trump)
> >
> > Best Regards
> >
> >
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@xxxxxxxxxxxxx
> > http://lists.xen.org/xen-devel
> >
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel