|
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] Re: [Xen-users] 99% iowait on one core in 8 core processor
After installing and starting irqbalance (from debian package), the
situation is exactly the same, even after restarting the domUs:
[snippet from /pro/interrupts]
# cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3 CPU4 CPU5
83: 5824430 0 0 0 0
0 Phys-fasteoi ahci
84: 1 0 0 0 0
0 Phys-fasteoi xhci_hcd
85: 0 0 0 0 0
0 Phys-fasteoi xhci_hcd
86: 0 0 0 0 0
0 Phys-fasteoi xhci_hcd
87: 0 0 0 0 0
0 Phys-fasteoi xhci_hcd
88: 0 0 0 0 0
0 Phys-fasteoi xhci_hcd
89: 0 0 0 0 0
0 Phys-fasteoi xhci_hcd
90: 0 0 0 0 0
0 Phys-fasteoi xhci_hcd
91: 1559973 0 0 0 0
0 Phys-fasteoi eth0
92: 29 0 0 0 0
0 Phys-fasteoi snd_hda_intel
93: 8202 0 0 0 0
0 Dynamic-fasteoi evtchn:xenstored[2827]
94: 14362 0 0 0 0
0 Dynamic-fasteoi xenbus
95: 12 0 0 0 0
0 Dynamic-fasteoi evtchn:xenstored[2827]
96: 729 0 0 0 0
0 Dynamic-fasteoi evtchn:xenstored[2827]
97: 345357 0 0 0 0
0 Dynamic-fasteoi evtchn:qemu-dm[3725]
98: 15239731 0 0 0 0
0 Dynamic-fasteoi evtchn:qemu-dm[3725]
99: 602773 0 0 0 0
0 Dynamic-fasteoi fw11
100: 25609 0 0 0 0
0 Dynamic-fasteoi blkif-backend
101: 41651 0 0 0 0
0 Dynamic-fasteoi fw12
102: 1 0 0 0 0
0 Dynamic-fasteoi fw14
103: 1 0 0 0 0
0 Dynamic-fasteoi fw15
104: 1 0 0 0 0
0 Dynamic-fasteoi fw16
105: 960520 0 0 0 0
0 Dynamic-fasteoi fw17
106: 408 0 0 0 0
0 Dynamic-fasteoi evtchn:xenstored[2827]
107: 226434 0 0 0 0
0 Dynamic-fasteoi evtchn:qemu-dm[4535]
108: 15237929 0 0 0 0
0 Dynamic-fasteoi evtchn:qemu-dm[4535]
109: 77709 0 0 0 0
0 Dynamic-fasteoi blkif-backend
110: 592222 0 0 0 0
0 Dynamic-fasteoi fw21
111: 820394 0 0 0 0
0 Dynamic-fasteoi fw23
112: 205 0 0 0 0
0 Dynamic-fasteoi evtchn:xenstored[2827]
113: 14548772 0 0 0 0
0 Dynamic-fasteoi evtchn:qemu-dm[24468]
114: 14105231 0 0 0 0
0 Dynamic-fasteoi evtchn:qemu-dm[24468]
115: 14016654 0 0 0 0
0 Dynamic-fasteoi evtchn:qemu-dm[24468]
116: 13903196 0 0 0 0
0 Dynamic-fasteoi evtchn:qemu-dm[24468]
117: 298 0 0 0 0
0 Dynamic-fasteoi evtchn:xenstored[2827]
118: 3070868 0 0 0 0
0 Dynamic-fasteoi evtchn:qemu-dm[5658]
119: 15303878 0 0 0 0
0 Dynamic-fasteoi evtchn:qemu-dm[5658]
120: 51825 0 0 0 0
0 Dynamic-fasteoi evtchn:qemu-dm[5658]
121: 562778 0 0 0 0
0 Dynamic-fasteoi blkif-backend
122: 1412 0 0 0 0
0 Dynamic-fasteoi blkif-backend
123: 1 0 0 0 0
0 Dynamic-fasteoi blkif-backend
124: 102196 0 0 0 0
0 Dynamic-fasteoi work
125: 1 0 0 0 0
0 Dynamic-fasteoi usbif-backend
126: 73329 0 0 0 0
0 Dynamic-fasteoi blkif-backend
127: 28622 0 0 0 0
0 Dynamic-fasteoi web
128: 272 0 0 0 0
0 Dynamic-fasteoi evtchn:xenstored[2827]
129: 272913 0 0 0 0
0 Dynamic-fasteoi evtchn:qemu-dm[13834]
130: 3040 0 0 0 0
0 Dynamic-fasteoi evtchn:qemu-dm[13834]
131: 4550 0 0 0 0
0 Dynamic-fasteoi evtchn:qemu-dm[13834]
132: 106866 0 0 0 0
0 Dynamic-fasteoi blkif-backend
133: 25 0 0 0 0
0 Dynamic-fasteoi dev
dev (the last one) was the one i tested restarting.. i ran a bonnie++
for testing
The only things which are devided uppon irqs are:
72: 1714878151 61618469 345335242 35614003 27627578
29681845 Dynamic-percpu timer
73: 2150718 1139885 4941560 618991 490155
491626 Dynamic-percpu ipi
RES: 2189003 1105537 4803667 577681 466929
444842 Rescheduling interrupts
CAL: 1846 34636 139105 41611 23420
46919 Function call interrupts
LCK: 133 89 299 70 60
61 Spinlock wakeups
MCP: 1 1 1 1 1
1 Machine check polls
Any other idea how we can make xen utilize the other (v)cpus for it's I/O stuff?
2012/7/12 Matthias <matthias.kannenberg@xxxxxxxxxxxxxx>:
> Just checked my /proc/interrupts:
>
> EVERY xen related thing (blkif-backend,the domUs, xhci_hcd, all
> xen-interfaces, evtchn:xenstored, evtchn:qemu-dm is all bound to cpu0,
> only zeros for the other cpu-cores..
>
> Sys is debian wheezy, xen is current testing from hg, kernel is
> openSuse3.4.2 with the xen patches they ship with it..
>
> will try your irqbalanced-suggestion next..
>
> 2012/7/12 Ian Campbell <ian.campbell@xxxxxxxxxx>:
>> On Wed, 2012-07-11 at 09:56 -0400, Matthias wrote:
>>> Hi,
>>>
>>> I experianced the same.
>>>
>>> I figured the reason behind this has nothing to do with what (v)cpus
>>> you assign to the domUs. In fact, I think that the real reason is that
>>> all I/O emulation from the domUs (like HDD, Network and Stuff) is all
>>> handled by CPU0 of dom0, even in a multi-core-architecture.
>>
>> This should not be the case. I/O emulation is done by qemu running in
>> domain 0 and should be scheduled on any dom0 vcpu. If it is not then
>> this is something to investigate. Normally this would require explicit
>> admin action to pin the affinity of the process though.
>>
>> One easy thing to look at would be /proc/interrupts to check that the
>> irq associated with ioreq upcalls for each guest are being properly
>> balanced (if not the installing irqbalanced might help)
>>
>> Ian.
>>
>>>
>>> You can test this with trying something I/O intensive like a hdd
>>> benchmark in the domU and you will see the dom0-cpu0 utilization
>>> rising.
>>>
>>> Using PVHVM or GPLPV-Driver only reduces the amount of utilization but
>>> it is still handled by cpu0 only. Due to that fact i leave cpu0 to my
>>> dom0 alone and only assign the other cpu-cores to the domUs.
>>>
>>>
>>> Note: This is only an assumption I made after seeing my machines
>>> behaviour. If the above isn't true, please correct me.
>>>
>>>
>>> 2012/7/11 Rajesh Kumar <rajesh@xxxxxxxx>:
>>> > Hi All,
>>> >
>>> > We have a xen server and using 8 core processor.
>>> >
>>> > I can see that there is 99% iowait on only core 0.
>>> >
>>> >
>>> > 02:28:49 AM CPU %user %nice %sys %iowait %irq %soft %steal
>>> > %idle intr/s
>>> > 02:28:54 AM all 0.00 0.00 0.00 12.65 0.00 0.02 2.24
>>> > 85.08 1359.88
>>> > 02:28:54 AM 0 0.00 0.00 0.00 96.21 0.00 0.20 3.19
>>> > 0.40 847.11
>>> > 02:28:54 AM 1 0.00 0.00 0.00 6.41 0.00 0.00 9.42
>>> > 84.17 219.56
>>> > 02:28:54 AM 2 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>>> > 100.00 2.59
>>> > 02:28:54 AM 3 0.00 0.00 0.00 0.00 0.00 0.00 2.12
>>> > 97.88 76.25
>>> > 02:28:54 AM 4 0.00 0.00 0.00 0.00 0.00 0.00 1.20
>>> > 98.80 118.56
>>> > 02:28:54 AM 5 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>>> > 100.00 3.59
>>> > 02:28:54 AM 6 0.00 0.00 0.00 0.00 0.00 0.00 2.02
>>> > 97.98 89.62
>>> > 02:28:54 AM 7 0.00 0.00 0.00 0.00 0.00 0.00 0.20
>>> > 99.80 2.59
>>> >
>>> > 02:28:54 AM CPU %user %nice %sys %iowait %irq %soft %steal
>>> > %idle intr/s
>>> > 02:28:59 AM all 0.00 0.00 0.00 12.48 0.00 0.00 2.78
>>> > 84.74 1317.43
>>> > 02:28:59 AM 0 0.00 0.00 0.00 98.80 0.00 0.00 0.80
>>> > 0.40 885.17
>>> > 02:28:59 AM 1 0.00 0.00 0.00 0.00 0.00 0.00 11.38
>>> > 88.62 151.30
>>> > 02:28:59 AM 2 0.00 0.00 0.00 0.00 0.00 0.00 0.20
>>> > 99.80 2.81
>>> > 02:28:59 AM 3 0.00 0.00 0.00 0.00 0.00 0.00 7.21
>>> > 92.79 94.79
>>> > 02:28:59 AM 4 0.00 0.00 0.00 0.00 0.00 0.00 2.20
>>> > 97.80 170.34
>>> > 02:28:59 AM 5 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>>> > 100.00 4.41
>>> > 02:28:59 AM 6 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>>> > 100.00 5.81
>>> > 02:28:59 AM 7 0.00 0.00 0.00 0.00 0.00 0.00 0.00
>>> > 100.00 2.81
>>> >
>>> >
>>> > I have even tries changing the cpus mapped to the DomU with out no effect.
>>> >
>>> > Name CPU CPU Affinity
>>> > 4pulse 1 1
>>> > 2music 3 3
>>> > Domain-0 0 0
>>> > Domain-0 1 1
>>> > Domain-0 2 2
>>> > Domain-0 3 3
>>> > Domain-0 4 4
>>> > Domain-0 5 5
>>> > Domain-0 6 6
>>> > Domain-0 7 7
>>> > analshah 6 any cpu
>>> > arunvelayudhan 7 any cpu
>>> > backup 7 any cpu
>>> > crickruns 3 1-3
>>> > crickruns 2 1-3
>>> > crickruns 1 1-3
>>> > crickruns 2 1-3
>>> > crickruns 1 1-3
>>> > crickruns 1 1-3
>>> > dedicatedjv 7 any cpu
>>> > yeluthu 4 3-5
>>> > yeluthu 3 3-5
>>> > yeluthu 3 3-5
>>> > yeluthu 3 3-5
>>> > yeluthu 3 3-5
>>> > yeluthu 3 3-5
>>> > freshnfresh 3 any cpu
>>> > monitoring 7 any cpu
>>> > reporter 6 5-7
>>> > reporter 7 5-7
>>> > reporter 6 5-7
>>> > reporter 6 5-7
>>> > reporter 7 5-7
>>> > reporter 7 5-7
>>> > reporter 5 5-7
>>> > reporter 7 5-7
>>> > radio03 7 any cpu
>>> > saampeter 2 1-2
>>> > saampeter 2 1-2
>>> >
>>> >
>>> > Thanks,
>>> > Rajesh
>>> > _______________________________________________
>>> > Xen-users mailing list
>>> > Xen-users@xxxxxxxxxxxxx
>>> > http://lists.xen.org/xen-users
>>>
>>> _______________________________________________
>>> Xen-users mailing list
>>> Xen-users@xxxxxxxxxxxxx
>>> http://lists.xen.org/xen-users
>>
>>
_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users
|
 |
Lists.xenproject.org is hosted with RackSpace, monitoring our |