Xen project Mailing List

Re: [Xen-devel] High CPU temp, suspend problem - xen 4.1.5-pre, linux 3.7.x

To: Andrew Cooper <andrew.cooper3@xxxxxxxxxx>

From: Marek Marczykowski <marmarek@xxxxxxxxxxxxxxxxxxxxxx>

Date: Tue, 26 Mar 2013 18:42:52 +0100

Cc: "xen-devel@xxxxxxxxxxxxx" <xen-devel@xxxxxxxxxxxxx>, Jan Beulich <JBeulich@xxxxxxxx>, Konrad Rzeszutek Wilk <konrad.wilk@xxxxxxxxxx>

Delivery-date: Tue, 26 Mar 2013 17:43:32 +0000

List-id: Xen developer discussion <xen-devel.lists.xen.org>

On 26.03.2013 18:02, Andrew Cooper wrote: > On 26/03/2013 16:45, Marek Marczykowski wrote: >> On 26.03.2013 17:03, Jan Beulich wrote: >>>>>> On 26.03.13 at 14:50, Marek Marczykowski >>>>>> <marmarek@xxxxxxxxxxxxxxxxxxxxxx> >>> wrote: >>>> On 26.03.2013 14:11, Jan Beulich wrote: >>>>>>>> On 26.03.13 at 13:17, Marek Marczykowski >>>>>>>> <marmarek@xxxxxxxxxxxxxxxxxxxxxx> >>>> wrote: >>>>>> Finally got serial console :) >>>>>> The debug=y problem is (actually at resume): >>>>>> (XEN) Assertion 'test_bit(vector, cfg->used_vectors)' failed at >>>>>> io_apic.c:542 >>>>>> (XEN) ----[ Xen-4.1.5-rc1 x86_64 debug=y Tainted: C ]---- >>>>>> (XEN) CPU: 0 >>>>>> (XEN) RIP: e008:[<ffff82c48015e288>] >>>>>> smp_irq_move_cleanup_interrupt+0x1c3/0x23d >>>>>> (XEN) RFLAGS: 0000000000010046 CONTEXT: hypervisor >>>>>> (XEN) rax: 0000000000000000 rbx: 00000000000000e9 rcx: >>>>>> ffff82c48029ff18 >>>>>> (XEN) rdx: 00000000000000e9 rsi: 000000000000002a rdi: >>>>>> ffff830421060538 >>>>>> (XEN) rbp: ffff82c48029ff08 rsp: ffff82c48029feb8 r8: >>>>>> ffff88041820eb60 >>>>>> (XEN) r9: 0000000000000000 r10: 0000000000007ff0 r11: >>>>>> 0000000000000000 >>>>>> (XEN) r12: ffff830421080250 r13: ffff830421060534 r14: >>>>>> ffff82c48029ff18 >>>>>> (XEN) r15: ffff82c4802dd9e0 cr0: 000000008005003b cr4: >>>>>> 00000000000026f0 >>>>>> (XEN) cr3: 0000000300b81000 cr2: ffff880402070198 >>>>>> (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: 0000 cs: e008 >>>>>> (XEN) Xen stack trace from rsp=ffff82c48029feb8: >>>>>> (XEN) 0000000000000000 000000000000e030 ffff82c48029ff18 >>>>>> ffff82c4802dd9e0 >>>>>> (XEN) ffff8802cac3c7c0 00000000ffff3729 00000000ffff3729 >>>>>> 000000013fff3728 >>>>>> (XEN) ffffffff81b907c0 00000000ffff3729 00007d3b7fd600c7 >>>>>> ffff82c48014de60 >>>>>> (XEN) 00000000ffff3729 ffffffff81b907c0 000000013fff3728 >>>>>> 00000000ffff3729 >>>>>> (XEN) ffffffff81a01e18 00000000ffff3729 0000000000000000 >>>>>> 0000000000007ff0 >>>>>> (XEN) 0000000000000000 ffff88041820eb60 ffff8803fd1820a8 >>>>>> ffffffff81b90a88 >>>>>> (XEN) 000000000000002a 000000000000002a 00000000ffff372a >>>>>> 0000002000000000 >>>>>> (XEN) ffffffff8105dd5a 000000000000e033 0000000000000246 >>>>>> ffffffff81a01db8 >>>>>> (XEN) 000000000000e02b 0000000000000000 0000000000000000 >>>>>> 0000000000000000 >>>>>> (XEN) 0000000000000000 0000000000000000 ffff8300ca9a0000 >>>>>> 0000000000000000 >>>>>> (XEN) 0000000000000000 >>>>>> (XEN) Xen call trace: >>>>>> (XEN) [<ffff82c48015e288>] smp_irq_move_cleanup_interrupt+0x1c3/0x23d >>>>>> (XEN) >>>>>> (XEN) >>>>>> (XEN) **************************************** >>>>>> (XEN) Panic on CPU 0: >>>>>> (XEN) Assertion 'test_bit(vector, cfg->used_vectors)' failed at >>>>>> io_apic.c:542 >>>>>> (XEN) **************************************** >>>>> To make sense of this, we need to know the register (and maybe >>>>> stack) allocation at this point, to know which vector it was that >>>>> triggered the assertion. You can either do this analysis for us, or >>>>> point us at the xen-syms binary matching the xen.gz you used. >>>> "info scope smp_irq_move_cleanup_interrupt" said vector is in %rbx, so >>>> 0xe9. >>> And that system isn't using a strange mixed mode IO-APIC/legacy >>> PIC model, where particularly IRQ 9 (usually ACPI SCI) gets >>> channeled through the legacy PIC? >> I don't know... >> >>> Could you attach the complete log, ideally with 'i' output logged >>> right before suspending? >> Sure, attached. >> >>> Is this reproducible with 4.2.x or 4.3-unstable? If not, but if readily >>> reproducible with 4.1.5-rc1, could you try changing the containing >>> loop's upper bound from "< NR_VECTORS" to >>> "<= LAST_DYNAMIC_VECTOR"? >> I've tried 4.2.x some time ago and bug also exists there (but I had not >> console, so not sure if exactly the same). 4.3 seems to be not affected. Checked 4.2 and indeed also assert() in similar place. If anyone interested, log here: http://duch.mimuw.edu.pl/~marmarek/qubes/console-4.2-failed-resume.log >> > > Can you replace the ASSERT() with code similar to that in > > http://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=xen/arch/x86/irq.c;h=5e0f463c381750090373dabd8967635bc297d457;hb=refs/heads/staging#l668 > > Which should call dump_irqs() in before dying because of the ASSERT. > You might need to also take the latest version of dump_irqs() from > unstable, as I seem to remember there was another assertion failure due > to xfree()'ing in IRQ context. Full log here: http://duch.mimuw.edu.pl/~marmarek/qubes/xen-4.1-failed-resume-dump-irqs.log Interesting part: (XEN) *** IRQ BUG found *** (XEN) CPU0 -Testing vector 233 from bitmap 39,47,63-65,72,80,88,96,98,112,120,125,144,152,160,168,174,182-183,190,192,198,200,208,214,222 (XEN) Guest interrupt information: (XEN) IRQ: 0 affinity:00000000,00000000,00000000,00000001 vec:f0 type=IO-APIC-edge status=00000000 mapped, unbound (XEN) IRQ: 1 affinity:00000000,00000000,00000000,00000002 vec:c6 type=IO-APIC-edge status=00000050 in-flight=0 domain-list=0: 1(-S--), (XEN) IRQ: 2 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:e2 type=XT-PIC status=00000000 mapped, unbound (XEN) IRQ: 3 affinity:00000000,00000000,00000000,00000001 vec:40 type=IO-APIC-edge status=00000002 mapped, unbound (XEN) IRQ: 4 affinity:00000000,00000000,00000000,00000001 vec:f1 type=IO-APIC-edge status=00000000 mapped, unbound (XEN) IRQ: 5 affinity:00000000,00000000,00000000,00000001 vec:48 type=IO-APIC-edge status=00000002 mapped, unbound (XEN) IRQ: 6 affinity:00000000,00000000,00000000,00000001 vec:50 type=IO-APIC-edge status=00000002 mapped, unbound (XEN) IRQ: 7 affinity:00000000,00000000,00000000,00000001 vec:58 type=IO-APIC-edge status=00000050 in-flight=0 domain-list=0: 7(-S--), (XEN) IRQ: 8 affinity:00000000,00000000,00000000,00000001 vec:60 type=IO-APIC-edge status=00000050 in-flight=0 domain-list=0: 8(-S--), (XEN) IRQ: 9 affinity:00000000,00000000,00000000,00000001 vec:de type=IO-APIC-level status=00000010 in-flight=0 domain-list=0: 9(-S--), (XEN) IRQ: 10 affinity:00000000,00000000,00000000,00000001 vec:70 type=IO-APIC-edge status=00000002 mapped, unbound (XEN) IRQ: 11 affinity:00000000,00000000,00000000,00000001 vec:78 type=IO-APIC-edge status=00000002 mapped, unbound (XEN) IRQ: 12 affinity:00000000,00000000,00000000,00000001 vec:27 type=IO-APIC-edge status=00000050 in-flight=0 domain-list=0: 12(-S--), (XEN) IRQ: 13 affinity:00000000,00000000,00000000,0000000f vec:90 type=IO-APIC-edge status=00000002 mapped, unbound (XEN) IRQ: 14 affinity:00000000,00000000,00000000,00000001 vec:98 type=IO-APIC-edge status=00000002 mapped, unbound (XEN) IRQ: 15 affinity:00000000,00000000,00000000,00000001 vec:a0 type=IO-APIC-edge status=00000002 mapped, unbound (XEN) IRQ: 16 affinity:00000000,00000000,00000000,00000001 vec:2f type=IO-APIC-level status=00000050 in-flight=0 domain-list=0: 16(-S--), (XEN) IRQ: 17 affinity:00000000,00000000,00000000,00000001 vec:3f type=IO-APIC-level status=00000050 in-flight=0 domain-list=0: 17(-S--), (XEN) IRQ: 18 affinity:00000000,00000000,00000000,00000008 vec:41 type=IO-APIC-level status=00000002 mapped, unbound (XEN) IRQ: 19 affinity:00000000,00000000,00000000,0000000f vec:c8 type=IO-APIC-level status=00000002 mapped, unbound (XEN) IRQ: 20 affinity:00000000,00000000,00000000,00000002 vec:b7 type=IO-APIC-level status=00000050 in-flight=0 domain-list=0: 20(-S--), (XEN) IRQ: 22 affinity:00000000,00000000,00000000,0000000f vec:62 type=IO-APIC-level status=00000002 mapped, unbound (XEN) IRQ: 23 affinity:00000000,00000000,00000000,0000000f vec:a8 type=IO-APIC-edge status=00000002 mapped, unbound (XEN) IRQ: 24 affinity:00000000,00000000,00000000,00000001 vec:28 type=DMA_MSI status=00000000 mapped, unbound (XEN) IRQ: 25 affinity:00000000,00000000,00000000,00000001 vec:30 type=DMA_MSI status=00000000 mapped, unbound (XEN) IRQ: 26 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:6f type=PCI-MSI status=00000042 mapped, unbound (XEN) IRQ: 27 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:77 type=PCI-MSI status=00000042 mapped, unbound (XEN) IRQ: 28 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:7f type=PCI-MSI status=00000042 mapped, unbound (XEN) IRQ: 29 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:87 type=PCI-MSI status=00000042 mapped, unbound (XEN) IRQ: 31 affinity:00000000,00000000,00000000,00000002 vec:a6 type=PCI-MSI status=00000002 mapped, unbound (XEN) IRQ: 32 affinity:00000000,00000000,00000000,00000001 vec:47 type=PCI-MSI status=00000050 in-flight=0 domain-list=0:273(-S--), (XEN) IRQ: 33 affinity:00000000,00000000,00000000,00000002 vec:5f type=PCI-MSI status=00000010 in-flight=0 domain-list=0:272(PS--), (XEN) IRQ: 34 affinity:00000000,00000000,00000000,00000001 vec:67 type=PCI-MSI status=00000010 in-flight=0 domain-list=0:271(-S--), (XEN) IRQ: 35 affinity:00000000,00000000,00000000,00000001 vec:4f type=PCI-MSI status=00000050 in-flight=0 domain-list=1: 55(-S--), (XEN) IO-APIC interrupt information: (XEN) IRQ 0 Vec240: (XEN) Apic 0x00, Pin 2: vec=f0 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:0 (XEN) IRQ 1 Vec198: (XEN) Apic 0x00, Pin 1: vec=c6 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:0 (XEN) IRQ 3 Vec 64: (XEN) Apic 0x00, Pin 3: vec=40 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:0 (XEN) IRQ 4 Vec241: (XEN) Apic 0x00, Pin 4: vec=f1 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:0 (XEN) IRQ 5 Vec 72: (XEN) Apic 0x00, Pin 5: vec=48 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:0 (XEN) IRQ 6 Vec 80: (XEN) Apic 0x00, Pin 6: vec=50 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:0 (XEN) IRQ 7 Vec 88: (XEN) Apic 0x00, Pin 7: vec=58 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:0 (XEN) IRQ 8 Vec 96: (XEN) Apic 0x00, Pin 8: vec=60 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:0 (XEN) IRQ 9 Vec222: (XEN) Apic 0x00, Pin 9: vec=de delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=L mask=0 dest_id:0 (XEN) IRQ 10 Vec112: (XEN) Apic 0x00, Pin 10: vec=70 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:0 (XEN) IRQ 11 Vec120: (XEN) Apic 0x00, Pin 11: vec=78 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:0 (XEN) IRQ 12 Vec 39: (XEN) Apic 0x00, Pin 12: vec=27 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:0 (XEN) IRQ 13 Vec144: (XEN) Apic 0x00, Pin 13: vec=90 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=1 dest_id:0 (XEN) IRQ 14 Vec152: (XEN) Apic 0x00, Pin 14: vec=98 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:0 (XEN) IRQ 15 Vec160: (XEN) Apic 0x00, Pin 15: vec=a0 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=0 dest_id:0 (XEN) IRQ 16 Vec 47: (XEN) Apic 0x00, Pin 16: vec=2f delivery=LoPri dest=L status=0 polarity=1 irr=0 trig=L mask=0 dest_id:0 (XEN) IRQ 17 Vec 63: (XEN) Apic 0x00, Pin 17: vec=3f delivery=LoPri dest=L status=0 polarity=1 irr=0 trig=L mask=0 dest_id:0 (XEN) IRQ 18 Vec 65: (XEN) Apic 0x00, Pin 18: vec=41 delivery=LoPri dest=L status=0 polarity=1 irr=0 trig=L mask=1 dest_id:0 (XEN) IRQ 19 Vec200: (XEN) Apic 0x00, Pin 19: vec=c8 delivery=LoPri dest=L status=0 polarity=1 irr=0 trig=L mask=1 dest_id:0 (XEN) IRQ 20 Vec183: (XEN) Apic 0x00, Pin 20: vec=b7 delivery=LoPri dest=L status=0 polarity=1 irr=0 trig=L mask=0 dest_id:0 (XEN) IRQ 22 Vec 98: (XEN) Apic 0x00, Pin 22: vec=62 delivery=LoPri dest=L status=0 polarity=1 irr=0 trig=L mask=1 dest_id:0 (XEN) IRQ 23 Vec168: (XEN) Apic 0x00, Pin 23: vec=a8 delivery=LoPri dest=L status=0 polarity=0 irr=0 trig=E mask=1 dest_id:0 (XEN) Xen BUG at io_apic.c:554 (XEN) ----[ Xen-4.1.5-rc1 x86_64 debug=y Tainted: C ]---- (XEN) CPU: 0 (XEN) RIP: e008:[<ffff82c48015e2d6>] smp_irq_move_cleanup_interrupt+0x211/0x289 (XEN) RFLAGS: 0000000000010092 CONTEXT: hypervisor (XEN) rax: 0000000000000000 rbx: 00000000000000e9 rcx: 0000000000000000 (XEN) rdx: 0000000000000016 rsi: 000000000000000a rdi: ffff82c4802592e0 (XEN) rbp: ffff82c48029fd08 rsp: ffff82c48029fcb8 r8: 0000000000000018 (XEN) r9: 0000000000000000 r10: 0000000000000000 r11: 0000000000000001 (XEN) r12: ffff830421080250 r13: ffff830421060534 r14: ffff82c48029ff18 (XEN) r15: ffff82c4802dd9e0 cr0: 000000008005003b cr4: 00000000000026f0 (XEN) cr3: 0000000119a96000 cr2: ffff880402070198 (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) Xen stack trace from rsp=ffff82c48029fcb8: (XEN) 0000000000000000 ffff82c48029ff18 ffff82c4802dd9e0 000000e900000000 (XEN) ffff83042109ba04 ffff830421008000 0000000000000114 000000000000001d (XEN) 0000000000000114 0000000000000000 00007d3b7fd602c7 ffff82c48014de60 (XEN) 0000000000000000 0000000000000114 000000000000001d 0000000000000114 (XEN) ffff82c48029fdc8 ffff830421008000 0000000000000246 ffff82c48025c1f0 (XEN) 0000000000000003 0000001944602466 0000000000000000 0000000000000001 (XEN) 0000000000000000 0000000000000286 ffff830421060f34 0000002000000000 (XEN) ffff82c4801226c0 000000000000e008 0000000000000286 ffff82c48029fdc8 (XEN) 000000000000e010 0000000000000286 ffff82c48029fe48 ffff82c480164446 (XEN) ffff82c4802dd9e0 0000000000000286 ffff830421060f00 ffff830421060f34 (XEN) ffff830421050ac0 000000000000001d 0000000000000246 ffff8301108fd140 (XEN) ffff82c4801226d3 ffff82c48029fe78 000000000000001d ffff8803fa889af0 (XEN) 0000000000000114 ffff8804023be000 ffff82c48029fef8 ffff82c48017655b (XEN) ffff830114c7f300 ffffffff81381646 ffff82f600000008 ffff830421008000 (XEN) 0000000000000003 000000030000001d 00000000e2200000 0000000100a0fb00 (XEN) 0000000000007ff0 ffffffffffffffff 0000000000000003 0000000000000003 (XEN) 00000000e2200000 c390ed90d1ffffff 0000000000000202 ffff8300ca666000 (XEN) ffff8803fc880240 0000000000000011 ffff8804023be858 ffff8804023be000 (XEN) 00007d3b7fd600c7 ffff82c480209f38 ffffffff8100142a 0000000000000021 (XEN) ffff8804023be000 ffff8804023be858 0000000000000011 ffff8803fc880240 (XEN) Xen call trace: (XEN) [<ffff82c48015e2d6>] smp_irq_move_cleanup_interrupt+0x211/0x289 (XEN) [<ffff82c48014de60>] irq_move_cleanup_interrupt+0x30/0x40 (XEN) [<ffff82c4801226c0>] _spin_unlock_irqrestore+0x22/0x24 (XEN) [<ffff82c480164446>] map_domain_pirq+0x37a/0x3df (XEN) [<ffff82c48017655b>] do_physdev_op+0xa2b/0x1508 (XEN) [<ffff82c480209f38>] syscall_enter+0xc8/0x122 > > ~Andrew > -- Best Regards / Pozdrawiam, Marek Marczykowski Invisible Things Lab

Attachment: signature.asc
Description: OpenPGP digital signature

_______________________________________________ Xen-devel mailing list Xen-devel@xxxxxxxxxxxxx http://lists.xen.org/xen-devel

©2013 Xen Project, A Linux Foundation Collaborative Project. All Rights Reserved.
Linux Foundation is a registered trademark of The Linux Foundation.
Xen Project is a trademark of The Linux Foundation.