|
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] Re: [Xen-users] 99% iowait on one core in 8 core processor
Here is my "cat /proc/interrupts" output
CPU0 CPU1 CPU2 CPU3
CPU4 CPU5 CPU6 CPU7
1: 8 0 0 0 0 0
0 0 Phys-irq i8042
8: 0 0 0 0 0 0
0 0 Phys-irq rtc
9: 0 0 0 0 0 0
0 0 Phys-irq acpi
12: 105 0 0 0 0 0
0 0 Phys-irq i8042
17: 0 0 0 0 0 0
0 0 Phys-irq uhci_hcd:usb3
18: 0 0 0 0 0 0
0 0 Phys-irq ehci_hcd:usb1, uhci_hcd:usb8
19: 89 0 0 0 0 0
0 0 Phys-irq ehci_hcd:usb2, uhci_hcd:usb6
20: 0 0 0 0 0 0
0 0 Phys-irq uhci_hcd:usb4
21: 0 0 0 0 0 0
0 0 Phys-irq uhci_hcd:usb5, uhci_hcd:usb7
22: 80057205 0 0 0 0 0
40990 149827 Phys-irq aacraid
248: 45804593 0 0 0 0 0
0 223 Phys-irq peth0
249: 0 0 0 0 0 0
0 0 Phys-irq ahci
256: 34539669 0 0 0 0 0
0 0 Dynamic-irq timer0
257: 41413620 0 0 0 0 0
0 0 Dynamic-irq resched0
258: 85 0 0 0 0 0
0 0 Dynamic-irq callfunc0
259: 0 41166287 0 0 0 0
0 0 Dynamic-irq resched1
260: 0 933 0 0 0 0
0 0 Dynamic-irq callfunc1
261: 0 19504489 0 0 0 0
0 0 Dynamic-irq timer1
262: 0 0 31251343 0 0 0
0 0 Dynamic-irq resched2
263: 0 0 932 0 0 0
0 0 Dynamic-irq callfunc2
264: 0 0 11054510 0 0 0
0 0 Dynamic-irq timer2
265: 0 0 0 27386916 0 0
0 0 Dynamic-irq resched3
266: 0 0 0 932 0 0
0 0 Dynamic-irq callfunc3
267: 0 0 0 11444950 0 0
0 0 Dynamic-irq timer3
268: 0 0 0 0 58130277 0
0 0 Dynamic-irq resched4
269: 0 0 0 0 910 0
0 0 Dynamic-irq callfunc4
270: 0 0 0 0 28261822 0
0 0 Dynamic-irq timer4
271: 0 0 0 0 0 43997038
0 0 Dynamic-irq resched5
272: 0 0 0 0 0 923
0 0 Dynamic-irq callfunc5
273: 0 0 0 0 0 15338193
0 0 Dynamic-irq timer5
274: 0 0 0 0 0 0
48900209 0 Dynamic-irq resched6
275: 0 0 0 0 0 0
923 0 Dynamic-irq callfunc6
276: 0 0 0 0 0 0
15407236 0 Dynamic-irq timer6
277: 0 0 0 0 0 0
0 50136483 Dynamic-irq resched7
278: 0 0 0 0 0 0
0 900 Dynamic-irq callfunc7
279: 0 0 0 0 0 0
0 16601995 Dynamic-irq timer7
280: 15653 3318 8 0 0 0
0 0 Dynamic-irq xenbus
281: 0 0 0 0 0 0
0 0 Dynamic-irq console
282: 148849 101244 11648 8306 5326 2336
220 93 Dynamic-irq blkif-backend
283: 6 0 0 0 0 0
0 0 Dynamic-irq blkif-backend
284: 65895 47506 23079 29645 23784 11350
246 147 Dynamic-irq vif1.0
285: 1863994 691202 63261 36091 15258 4937
437 193 Dynamic-irq blkif-backend
286: 213220 119178 7103 5451 2557 517
19 0 Dynamic-irq blkif-backend
287: 930007 387710 35444 16389 11486 4540
198 229 Dynamic-irq vif19.0
288: 635545 409092 69409 40913 24484 8576
741 136 Dynamic-irq blkif-backend
289: 7137631 1115254 173115 71776 29668 12043
1008 2289 Dynamic-irq blkif-backend
290: 31508 30372 21679 16886 8057 2840
74 252 Dynamic-irq vif3.0
291: 5246 16885 29753 36947 28554 12073
406 1 Dynamic-irq vif4.0
292: 47906 36257 8923 10614 7908 2954
3 6 Dynamic-irq blkif-backend
293: 1 0 0 0 0 0
0 0 Dynamic-irq blkif-backend
294: 17554 15779 10298 10228 7353 4006
114 58 Dynamic-irq blkif-backend
295: 23 0 0 0 0 0
0 0 Dynamic-irq blkif-backend
296: 21802 13356 19719 24348 19953 8304
113 0 Dynamic-irq vif5.0
297: 273660 179974 15464 8548 4237 1913
21 34 Dynamic-irq blkif-backend
298: 3 0 0 0 0 0
0 0 Dynamic-irq blkif-backend
299: 7652040 1568396 290845 170934 76196 35670
3622 4062 Dynamic-irq vif6.0
300: 329614 141967 17636 4723 2181 696
43 48 Dynamic-irq blkif-backend
301: 183112 77217 7104 4301 2413 257
0 2 Dynamic-irq blkif-backend
302: 328166 181810 18838 8180 4534 1045
256 108 Dynamic-irq vif7.0
303: 166185 117000 16767 12453 10239 4296
96 16 Dynamic-irq blkif-backend
304: 18 0 0 0 0 0
0 0 Dynamic-irq blkif-backend
305: 74460 48748 4299 2838 1670 592
47 18 Dynamic-irq blkif-backend
306: 17 0 0 0 0 0
0 0 Dynamic-irq blkif-backend
307: 849646 410362 42829 20108 18286 2718
147 73 Dynamic-irq blkif-backend
308: 3031 4122 413 118 280 1
0 0 Dynamic-irq blkif-backend
309: 454672 211984 11278 13235 4483 1949
106 307 Dynamic-irq blkif-backend
310: 54 2 0 0 1 0
0 0 Dynamic-irq blkif-backend
311: 151460 100790 10438 8542 3677 1836
130 91 Dynamic-irq blkif-backend
312: 17 0 0 0 0 0
0 0 Dynamic-irq blkif-backend
313: 289222 147029 13023 6293 3359 997
32 38 Dynamic-irq blkif-backend
314: 15919 13241 297 219 45 67
1 234 Dynamic-irq blkif-backend
315: 171629 139277 11409 8646 6430 2920
82 170 Dynamic-irq vif17.0
316: 1684637 884290 70555 43332 22190 6997
1874 2528 Dynamic-irq vif20.0
317: 5997001 1669698 253862 124739 53491 14573
3931 2018 Dynamic-irq vif13.0
318: 196460 112218 7607 4688 3110 959
203 283 Dynamic-irq vif18.0
319: 70632 50590 19755 22438 18394 7733
336 156 Dynamic-irq vif14.0
320: 430659 270256 20899 13806 7980 2998
260 146 Dynamic-irq vif15.0
NMI: 0 0 0 0 0 0
0 0
LOC: 0 0 0 0 0 0
0 0
ERR: 0
Also here are few smp_affinity values
root@hiox-vps ~]# cat /proc/irq/316/smp_affinity
00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000004
[root@hiox-vps ~]# cat /proc/irq/316/smp_affinity
00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000004
[root@hiox-vps ~]# cat /proc/irq/312/smp_affinity
00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000001
[root@hiox-vps ~]# cat /proc/irq/317/smp_affinity
00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000002
[root@hiox-vps ~]# cat /proc/irq/318/smp_affinity
00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000002
[root@hiox-vps ~]# cat /proc/irq/300/smp_affinity
00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000002
[root@hiox-vps ~]# cat /proc/irq/260/smp_affinity
00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000002
On 13-Jul-2012, at 1:44 PM, Matthias wrote:
> Some updates:
>
> - Deactivating CONFIg_HOTPlUG_CPU and suspend to ram in kernel didn'T
> change anything.
> - I tried to change the smp_affinity to just a single other cpu and
> this worked. So basically I was wrong that there is something changing
> the settings back constantly but linux is simply discarding every
> assignment to more then one cpu (1,2,4, etc works, 3,5,etc is
> discarded)
>
> so the problem is that my linux can not assign irqs to multiple cpus.
> I think this is also why irqbalance does not make a difference because
> it tries the same.
>
> I will check if i can reproduce this behaviour with a stock kernel
> without xen or if this is really xen related in the evening. But if
> you have any other idea what could cause this, I'm open for
> suggestions
>
> @Rajesh can you check with your setup if you have the same case or if
> this is a different problem? simply do a 'cat /proc/interrupts' to
> check the cpu affinity and if everything is done on cpu0 try a 'echo
> "3" > /proc/irq/<some irq number from the other command>/smp_affinity'
> and afterwards check if the smp_affinity now really has '3' as
> content.
>
> 2012/7/13 Matthias <matthias.kannenberg@xxxxxxxxxxxxxx>:
>> Found a hint somewhere that my problem might be related to the
>> CONFIG_HOTPLUG_CPU kernel option.. recompiling the kernel right now and will
>> update tomorrow if this resolved the issue..
>>
>> Am 13.07.2012 00:27 schrieb "Matthias" <matthias.kannenberg@xxxxxxxxxxxxxx>:
>>
>>> Hi Ian,
>>>
>>> sorry, but it's still not working..
>>>
>>> I rebooted the server, then manually started irqbalanced just to be
>>> sure, then started xen and the domUs.. still the same as above: every
>>> I/O related is done by cpu0 (even after hours of running 5 domUs). i
>>> see some spikes on the other cpu's, but i think this is due to domUs
>>> using their assigned vcpus and xentop shows a distribution of cputime,
>>> too..
>>>
>>> I started irqbalance in debug modus once and it complained that my
>>> hardware is not numa compatible. Might this be an issue? From what I
>>> read, xen should support proper scheduling on non-numa hardware and
>>> only the numa-support is new and might be a bit quirky..
>>>
>>> checked my smp_affinity stuff then: currently, it shows the following for
>>> a domU
>>> smp_affinity: 01
>>> smp_affinity_list: 0
>>>
>>> tried to change smp_affinity to 3f (=111111 for my 6vcpu) and it was
>>> changed immediately back to 01.. thought that was irqbalance going
>>> rogue but after stopd the deamon, this still happens..
>>>
>>> so my take is: something is setting the irq to only use cpu0 and
>>> changes i do manually or which are done by irqbalance are overwritten
>>> constantly making irqbalance useless..
>>>
>>> Any idea what this can be? supposently something within xen?
>>>
>>>
>>>
>>> 2012/7/12 Ian Campbell <ian.campbell@xxxxxxxxxx>:
>>>> On Thu, 2012-07-12 at 13:04 -0400, Matthias wrote:
>>>>>
>>>>> Any other idea how we can make xen utilize the other (v)cpus for it's
>>>>> I/O stuff?
>>>>
>>>> Are you sure irqbalanced is running? Some versions had a bug and would
>>>> crash on a Xen system (they crash if there is no irq 0 or something like
>>>> that). Even if it is running it can take some time for irqbalanced to
>>>> realise that things are unbalanced and start moving stuff around.
>>>>
>>>> There are ways in Linux to manually balance IRQs. You have to much
>>>> around with /proc/irq/*/smp_affinity*
>>>>
>>>> Really irqbalanced should be doing this for you though.
>>>>
>>>> Ian.
>>>>
>>>>
_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users
|
 |
Lists.xenproject.org is hosted with RackSpace, monitoring our |