[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] Re: [Xen-devel] Poor network performance between DomU with multiqueue support
Thanks for your reply, Wei. I do the following testing just now and found the results as follows: There are three DomUs (4U4G) are running on Host A (6U6G) and one DomU (4U4G) is running on Host B (6U6G), I send packets from three DomUs to the DomU on Host B simultaneously. 1. The "top" output of Host B as follows: top - 09:42:11 up 1:07, 2 users, load average: 2.46, 1.90, 1.47 Tasks: 173 total, 4 running, 169 sleeping, 0 stopped, 0 zombie %Cpu0 : 0.0 us, 0.0 sy, 0.0 ni, 97.3 id, 0.0 wa, 0.0 hi, 0.8 si, 1.9 st %Cpu1 : 0.0 us, 27.0 sy, 0.0 ni, 63.1 id, 0.0 wa, 0.0 hi, 9.5 si, 0.4 st %Cpu2 : 0.0 us, 90.0 sy, 0.0 ni, 8.3 id, 0.0 wa, 0.0 hi, 1.7 si, 0.0 st %Cpu3 : 0.4 us, 1.4 sy, 0.0 ni, 95.4 id, 0.0 wa, 0.0 hi, 1.4 si, 1.4 st %Cpu4 : 0.0 us, 60.2 sy, 0.0 ni, 39.5 id, 0.0 wa, 0.0 hi, 0.3 si, 0.0 st %Cpu5 : 0.0 us, 2.8 sy, 0.0 ni, 89.4 id, 0.0 wa, 0.0 hi, 6.9 si, 0.9 st KiB Mem: 4517144 total, 3116480 used, 1400664 free, 876 buffers KiB Swap: 2103292 total, 0 used, 2103292 free. 2374656 cached Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 7440 root 20 0 0 0 0 R 71.10 0.000 8:15.38 vif4.0-q3-guest 7434 root 20 0 0 0 0 R 59.14 0.000 9:00.58 vif4.0-q0-guest 18 root 20 0 0 0 0 R 33.89 0.000 2:35.06 ksoftirqd/2 28 root 20 0 0 0 0 S 20.93 0.000 3:01.81 ksoftirqd/4 As shown above, only two netback related processes (vif4.0-*) are running with high cpu usage, and the other 2 netback processes are idle. The "ps" result of vif4.0-* processes as follows: root 7434 50.5 0.0 0 0 ? R 09:23 11:29 [vif4.0-q0-guest] root 7435 0.0 0.0 0 0 ? S 09:23 0:00 [vif4.0-q0-deall] root 7436 0.0 0.0 0 0 ? S 09:23 0:00 [vif4.0-q1-guest] root 7437 0.0 0.0 0 0 ? S 09:23 0:00 [vif4.0-q1-deall] root 7438 0.0 0.0 0 0 ? S 09:23 0:00 [vif4.0-q2-guest] root 7439 0.0 0.0 0 0 ? S 09:23 0:00 [vif4.0-q2-deall] root 7440 48.1 0.0 0 0 ? R 09:23 10:55 [vif4.0-q3-guest] root 7441 0.0 0.0 0 0 ? S 09:23 0:00 [vif4.0-q3-deall] root 9724 0.0 0.0 9244 1520 pts/0 S+ 09:46 0:00 grep --color=auto 2. The "rx" related content in /proc/interupts in receiver DomU (on Host B): 73: 2 0 2925405 0 xen-dyn-event eth0-q0-rx 75: 43 93 0 118 xen-dyn-event eth0-q1-rx 77: 2 3376 14 1983 xen-dyn-event eth0-q2-rx 79: 2414666 0 9 0 xen-dyn-event eth0-q3-rx As shown above, it seems like that only q0 and q3 handles the interrupt triggered by packet receving. Any advise? Thanks. ---------- zhangleiqiang (Trump) Best Regards > -----Original Message----- > From: Wei Liu [mailto:wei.liu2@xxxxxxxxxx] > Sent: Tuesday, December 02, 2014 8:12 PM > To: Zhangleiqiang (Trump) > Cc: Wei Liu; zhangleiqiang; xen-devel@xxxxxxxxxxxxx; Luohao (brian); Xiaoding > (B); Yuzhou (C); Zhuangyuxin > Subject: Re: [Xen-devel] Poor network performance between DomU with > multiqueue support > > On Tue, Dec 02, 2014 at 11:50:59AM +0000, Zhangleiqiang (Trump) wrote: > > > -----Original Message----- > > > From: xen-devel-bounces@xxxxxxxxxxxxx > > > [mailto:xen-devel-bounces@xxxxxxxxxxxxx] On Behalf Of Wei Liu > > > Sent: Tuesday, December 02, 2014 7:02 PM > > > To: zhangleiqiang > > > Cc: wei.liu2@xxxxxxxxxx; xen-devel@xxxxxxxxxxxxx > > > Subject: Re: [Xen-devel] Poor network performance between DomU with > > > multiqueue support > > > > > > On Tue, Dec 02, 2014 at 04:30:49PM +0800, zhangleiqiang wrote: > > > > Hi, all > > > > I am testing the performance of xen netfront-netback driver > > > > that with > > > multi-queues support. The throughput from domU to remote dom0 is > > > 9.2Gb/s, but the throughput from domU to remote domU is only > > > 3.6Gb/s, I think the bottleneck is the throughput from dom0 to local > > > domU. However, we have done some testing and found the throughput > > > from dom0 to local domU is 5.8Gb/s. > > > > And if we send packets from one DomU to other 3 DomUs on > > > > different > > > host simultaneously, the sum of throughout can reach 9Gbps. It seems > > > like the bottleneck is the receiver? > > > > After some analysis, I found that even the max_queue of > > > > netfront/back > > > is set to 4, there are some strange results as follows: > > > > 1. In domU, only one rx queue deal with softirq > > > > > > Try to bind irq to different vcpus? > > > > Do you mean we try to bind irq to different vcpus in DomU? I will try it > > now. > > > > Yes. Given the fact that you have two backend threads running while only one > DomU vcpu is busy, it smells like misconfiguration in DomU. > > If this phenomenon persists after correctly binding irqs, you might want to > check traffic is steering correctly to different queues. > > > > > > > > 2. In dom0, only two netback queues process are scheduled, > > > > other two > > > process aren't scheduled. > > > > > > How many Dom0 vcpu do you have? If it only has two then there will > > > only be two processes running at a time. > > > > Dom0 has 6 vcpus, and 6G memory. There are only one DomU running in > Dom0 and so four netback processes are running in Dom0 (because the > max_queue param of netback kernel module is set to 4). > > The phenomenon is that only 2 of these four netback process were running > with about 70% cpu usage, and another two use little CPU. > > Is there a hash algorithm to determine which netback process to handle the > input packet? > > > > I think that's whatever default algorithm Linux kernel is using. > > We don't currently support other algorithms. > > Wei. _______________________________________________ Xen-devel mailing list Xen-devel@xxxxxxxxxxxxx http://lists.xen.org/xen-devel
|
Lists.xenproject.org is hosted with RackSpace, monitoring our |