Xen project Mailing List

Re: [Xen-devel] High CPU temp, suspend problem - xen 4.1.5-pre, linux 3.7.x

To: Marek Marczykowski <marmarek@xxxxxxxxxxxxxxxxxxxxxx>

From: Andrew Cooper <andrew.cooper3@xxxxxxxxxx>

Date: Tue, 26 Mar 2013 18:50:51 +0000

Cc: "xen-devel@xxxxxxxxxxxxx" <xen-devel@xxxxxxxxxxxxx>, Jan Beulich <JBeulich@xxxxxxxx>, Konrad Rzeszutek Wilk <konrad.wilk@xxxxxxxxxx>

Delivery-date: Tue, 26 Mar 2013 18:51:16 +0000

List-id: Xen developer discussion <xen-devel.lists.xen.org>

On 26/03/2013 18:21, Marek Marczykowski wrote: > On 26.03.2013 18:54, Andrew Cooper wrote: >>>> Can you replace the ASSERT() with code similar to that in >>>> >>>> http://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=xen/arch/x86/irq.c;h=5e0f463c381750090373dabd8967635bc297d457;hb=refs/heads/staging#l668 >>>> >>>> Which should call dump_irqs() in before dying because of the ASSERT. >>>> You might need to also take the latest version of dump_irqs() from >>>> unstable, as I seem to remember there was another assertion failure due >>>> to xfree()'ing in IRQ context. >>> Full log here: >>> http://duch.mimuw.edu.pl/~marmarek/qubes/xen-4.1-failed-resume-dump-irqs.log >>> Interesting part: > (...) >> Even more curious. vector e9 does not appear to be programmed in. Can >> you extend the debugging to also call __print_IO_APIC(). >> >> The i debug key and z debug key list IO-APIC entries from different >> sources of information. > As you wish, full log: > http://duch.mimuw.edu.pl/~marmarek/qubes/xen-4.1-failed-resume-dump-irqs2.log > > Final part: > (XEN) *** IRQ BUG found *** > (XEN) CPU0 -Testing vector 233 from bitmap > 43,49,64,72,80,87-88,95-96,103,112,119-121,127,135,143-144,151-152,159-160,168,192,197,200,211,216,218 > (XEN) Guest interrupt information: > (XEN) IRQ: 0 affinity:00000000,00000000,00000000,00000001 vec:f0 > type=IO-APIC-edge status=00000000 mapped, unbound > (XEN) IRQ: 1 affinity:00000000,00000000,00000000,00000001 vec:7f > type=IO-APIC-edge status=00000050 in-flight=0 domain-list=0: 1(-S--), > (XEN) IRQ: 2 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:e2 > type=XT-PIC status=00000000 mapped, unbound > (XEN) IRQ: 3 affinity:00000000,00000000,00000000,00000001 vec:40 > type=IO-APIC-edge status=00000002 mapped, unbound > (XEN) IRQ: 4 affinity:00000000,00000000,00000000,00000001 vec:f1 > type=IO-APIC-edge status=00000000 mapped, unbound > (XEN) IRQ: 5 affinity:00000000,00000000,00000000,00000001 vec:48 > type=IO-APIC-edge status=00000002 mapped, unbound > (XEN) IRQ: 6 affinity:00000000,00000000,00000000,00000001 vec:50 > type=IO-APIC-edge status=00000002 mapped, unbound > (XEN) IRQ: 7 affinity:00000000,00000000,00000000,00000008 vec:da > type=IO-APIC-edge status=00000050 in-flight=0 domain-list=0: 7(-S--), > (XEN) IRQ: 8 affinity:00000000,00000000,00000000,00000004 vec:d8 > type=IO-APIC-edge status=00000050 in-flight=0 domain-list=0: 8(-S--), > (XEN) IRQ: 9 affinity:00000000,00000000,00000000,00000001 vec:87 > type=IO-APIC-level status=00000010 in-flight=0 domain-list=0: 9(-S--), > (XEN) IRQ: 10 affinity:00000000,00000000,00000000,00000001 vec:70 > type=IO-APIC-edge status=00000002 mapped, unbound > (XEN) IRQ: 11 affinity:00000000,00000000,00000000,00000001 vec:78 > type=IO-APIC-edge status=00000002 mapped, unbound > (XEN) IRQ: 12 affinity:00000000,00000000,00000000,00000001 vec:8f > type=IO-APIC-edge status=00000050 in-flight=0 domain-list=0: 12(-S--), > (XEN) IRQ: 13 affinity:00000000,00000000,00000000,0000000f vec:90 > type=IO-APIC-edge status=00000002 mapped, unbound > (XEN) IRQ: 14 affinity:00000000,00000000,00000000,00000001 vec:98 > type=IO-APIC-edge status=00000002 mapped, unbound > (XEN) IRQ: 15 affinity:00000000,00000000,00000000,00000001 vec:a0 > type=IO-APIC-edge status=00000002 mapped, unbound > (XEN) IRQ: 16 affinity:00000000,00000000,00000000,00000001 vec:97 > type=IO-APIC-level status=00000050 in-flight=0 domain-list=0: 16(-S--), > (XEN) IRQ: 17 affinity:00000000,00000000,00000000,00000001 vec:9f > type=IO-APIC-level status=00000050 in-flight=0 domain-list=0: 17(-S--), > (XEN) IRQ: 18 affinity:00000000,00000000,00000000,00000004 vec:79 > type=IO-APIC-level status=00000002 mapped, unbound > (XEN) IRQ: 19 affinity:00000000,00000000,00000000,0000000f vec:c8 > type=IO-APIC-level status=00000002 mapped, unbound > (XEN) IRQ: 20 affinity:00000000,00000000,00000000,00000002 vec:d3 > type=IO-APIC-level status=00000050 in-flight=0 domain-list=0: 20(-S--), > (XEN) IRQ: 22 affinity:00000000,00000000,00000000,0000000f vec:2b > type=IO-APIC-level status=00000002 mapped, unbound > (XEN) IRQ: 23 affinity:00000000,00000000,00000000,0000000f vec:a8 > type=IO-APIC-edge status=00000002 mapped, unbound > (XEN) IRQ: 24 affinity:00000000,00000000,00000000,00000001 vec:28 > type=DMA_MSI status=00000000 mapped, unbound > (XEN) IRQ: 25 affinity:00000000,00000000,00000000,00000001 vec:30 > type=DMA_MSI status=00000000 mapped, unbound > (XEN) IRQ: 26 affinity:00000000,00000000,00000000,00000001 vec:c7 > type=PCI-MSI status=00000010 in-flight=0 domain-list=0:279(-S--), > (XEN) IRQ: 27 affinity:00000000,00000000,00000000,00000001 vec:cf > type=PCI-MSI status=00000050 in-flight=0 domain-list=0:278(-S--), > (XEN) IRQ: 28 affinity:00000000,00000000,00000000,00000001 vec:d7 > type=PCI-MSI status=00000050 in-flight=0 domain-list=0:277(-S--), > (XEN) IRQ: 29 affinity:00000000,00000000,00000000,00000001 vec:df > type=PCI-MSI status=00000050 in-flight=0 domain-list=0:276(-S--), > (XEN) IRQ: 30 affinity:00000000,00000000,00000000,00000001 vec:38 > type=PCI-MSI status=00000050 in-flight=0 domain-list=0:275(-S--), > (XEN) IRQ: 31 affinity:00000000,00000000,00000000,00000004 vec:47 > type=PCI-MSI status=00000002 mapped, unbound > (XEN) IRQ: 32 affinity:00000000,00000000,00000000,00000001 vec:a7 > type=PCI-MSI status=00000050 in-flight=0 domain-list=0:273(-S--), > (XEN) IRQ: 33 affinity:00000000,00000000,00000000,00000001 vec:b7 > type=PCI-MSI status=00000010 in-flight=0 domain-list=0:272(-S--), > (XEN) IRQ: 34 affinity:00000000,00000000,00000000,00000004 vec:40 > type=PCI-MSI status=00000010 in-flight=0 domain-list=0:271(PS--), > (XEN) IRQ: 35 affinity:00000000,00000000,00000000,00000001 vec:af > type=PCI-MSI status=00000050 in-flight=0 domain-list=1: 55(-S--), > (XEN) IO-APIC interrupt information: > (XEN) IRQ 0 Vec240: > (XEN) Apic 0x00, Pin 2: vec=f0 delivery=LoPri dest=L status=0 > polarity=0 irr=0 trig=E mask=0 dest_id:0 > (XEN) IRQ 1 Vec127: > (XEN) Apic 0x00, Pin 1: vec=7f delivery=LoPri dest=L status=0 > polarity=0 irr=0 trig=E mask=0 dest_id:0 > (XEN) IRQ 3 Vec 64: > (XEN) Apic 0x00, Pin 3: vec=40 delivery=LoPri dest=L status=0 > polarity=0 irr=0 trig=E mask=0 dest_id:0 > (XEN) IRQ 4 Vec241: > (XEN) Apic 0x00, Pin 4: vec=f1 delivery=LoPri dest=L status=0 > polarity=0 irr=0 trig=E mask=0 dest_id:0 > (XEN) IRQ 5 Vec 72: > (XEN) Apic 0x00, Pin 5: vec=48 delivery=LoPri dest=L status=0 > polarity=0 irr=0 trig=E mask=0 dest_id:0 > (XEN) IRQ 6 Vec 80: > (XEN) Apic 0x00, Pin 6: vec=50 delivery=LoPri dest=L status=0 > polarity=0 irr=0 trig=E mask=0 dest_id:0 > (XEN) IRQ 7 Vec218: > (XEN) Apic 0x00, Pin 7: vec=da delivery=LoPri dest=L status=0 > polarity=0 irr=0 trig=E mask=0 dest_id:0 > (XEN) IRQ 8 Vec216: > (XEN) Apic 0x00, Pin 8: vec=d8 delivery=LoPri dest=L status=0 > polarity=0 irr=0 trig=E mask=0 dest_id:0 > (XEN) IRQ 9 Vec135: > (XEN) Apic 0x00, Pin 9: vec=87 delivery=LoPri dest=L status=0 > polarity=0 irr=0 trig=L mask=0 dest_id:0 > (XEN) IRQ 10 Vec112: > (XEN) Apic 0x00, Pin 10: vec=70 delivery=LoPri dest=L status=0 > polarity=0 irr=0 trig=E mask=0 dest_id:0 > (XEN) IRQ 11 Vec120: > (XEN) Apic 0x00, Pin 11: vec=78 delivery=LoPri dest=L status=0 > polarity=0 irr=0 trig=E mask=0 dest_id:0 > (XEN) IRQ 12 Vec143: > (XEN) Apic 0x00, Pin 12: vec=8f delivery=LoPri dest=L status=0 > polarity=0 irr=0 trig=E mask=0 dest_id:0 > (XEN) IRQ 13 Vec144: > (XEN) Apic 0x00, Pin 13: vec=90 delivery=LoPri dest=L status=0 > polarity=0 irr=0 trig=E mask=1 dest_id:0 > (XEN) IRQ 14 Vec152: > (XEN) Apic 0x00, Pin 14: vec=98 delivery=LoPri dest=L status=0 > polarity=0 irr=0 trig=E mask=0 dest_id:0 > (XEN) IRQ 15 Vec160: > (XEN) Apic 0x00, Pin 15: vec=a0 delivery=LoPri dest=L status=0 > polarity=0 irr=0 trig=E mask=0 dest_id:0 > (XEN) IRQ 16 Vec151: > (XEN) Apic 0x00, Pin 16: vec=97 delivery=LoPri dest=L status=0 > polarity=1 irr=0 trig=L mask=0 dest_id:0 > (XEN) IRQ 17 Vec159: > (XEN) Apic 0x00, Pin 17: vec=9f delivery=LoPri dest=L status=0 > polarity=1 irr=0 trig=L mask=0 dest_id:0 > (XEN) IRQ 18 Vec121: > (XEN) Apic 0x00, Pin 18: vec=79 delivery=LoPri dest=L status=0 > polarity=1 irr=0 trig=L mask=1 dest_id:0 > (XEN) IRQ 19 Vec200: > (XEN) Apic 0x00, Pin 19: vec=c8 delivery=LoPri dest=L status=0 > polarity=1 irr=0 trig=L mask=1 dest_id:0 > (XEN) IRQ 20 Vec211: > (XEN) Apic 0x00, Pin 20: vec=d3 delivery=LoPri dest=L status=0 > polarity=1 irr=0 trig=L mask=0 dest_id:0 > (XEN) IRQ 22 Vec 43: > (XEN) Apic 0x00, Pin 22: vec=2b delivery=LoPri dest=L status=0 > polarity=1 irr=0 trig=L mask=1 dest_id:0 > (XEN) IRQ 23 Vec168: > (XEN) Apic 0x00, Pin 23: vec=a8 delivery=LoPri dest=L status=0 > polarity=0 irr=0 trig=E mask=1 dest_id:0 > (XEN) number of MP IRQ sources: 15. > (XEN) number of IO-APIC #2 registers: 24. > (XEN) testing the IO APIC....................... > (XEN) IO APIC #2...... > (XEN) .... register #00: 02000000 > (XEN) ....... : physical APIC id: 02 > (XEN) ....... : Delivery Type: 0 > (XEN) ....... : LTS : 0 > (XEN) .... register #01: 00170020 > (XEN) ....... : max redirection entries: 0017 > (XEN) ....... : PRQ implemented: 0 > (XEN) ....... : IO APIC version: 0020 > (XEN) .... IRQ redirection table: > (XEN) NR Log Phy Mask Trig IRR Pol Stat Dest Deli Vect: > (XEN) 00 0DC 0C 1 0 0 0 0 1 2 87 > (XEN) 01 000 00 0 0 0 0 0 1 1 7F > (XEN) 02 000 00 0 0 0 0 0 1 1 F0 > (XEN) 03 000 00 0 0 0 0 0 1 1 40 > (XEN) 04 000 00 0 0 0 0 0 1 1 F1 > (XEN) 05 000 00 0 0 0 0 0 1 1 48 > (XEN) 06 000 00 0 0 0 0 0 1 1 50 > (XEN) 07 000 00 0 0 0 0 0 1 1 DA > (XEN) 08 000 00 0 0 0 0 0 1 1 D8 > (XEN) 09 000 00 0 1 0 0 0 1 1 87 > (XEN) 0a 000 00 0 0 0 0 0 1 1 70 > (XEN) 0b 000 00 0 0 0 0 0 1 1 78 > (XEN) 0c 000 00 0 0 0 0 0 1 1 8F > (XEN) 0d 000 00 1 0 0 0 0 1 1 90 > (XEN) 0e 000 00 0 0 0 0 0 1 1 98 > (XEN) 0f 000 00 0 0 0 0 0 1 1 A0 > (XEN) 10 000 00 0 1 0 1 0 1 1 97 > (XEN) 11 000 00 0 1 0 1 0 1 1 9F > (XEN) 12 000 00 1 1 0 1 0 1 1 79 > (XEN) 13 000 00 1 1 0 1 0 1 1 C8 > (XEN) 14 000 00 0 1 0 1 0 1 1 D3 > (XEN) 15 000 00 1 0 0 0 0 0 0 00 > (XEN) 16 000 00 1 1 0 1 0 1 1 2B > (XEN) 17 000 00 1 0 0 0 0 1 1 A8 > (XEN) Using vector-based indexing > (XEN) IRQ to pin mappings: > (XEN) IRQ240 -> 0:2 > (XEN) IRQ127 -> 0:1 > (XEN) IRQ64 -> 0:3 > (XEN) IRQ241 -> 0:4 > (XEN) IRQ72 -> 0:5 > (XEN) IRQ80 -> 0:6 > (XEN) IRQ218 -> 0:7 > (XEN) IRQ216 -> 0:8 > (XEN) IRQ135 -> 0:9 > (XEN) IRQ112 -> 0:10 > (XEN) IRQ120 -> 0:11 > (XEN) IRQ143 -> 0:12 > (XEN) IRQ144 -> 0:13 > (XEN) IRQ152 -> 0:14 > (XEN) IRQ160 -> 0:15 > (XEN) IRQ151 -> 0:16 > (XEN) IRQ159 -> 0:17 > (XEN) IRQ121 -> 0:18 > (XEN) IRQ200 -> 0:19 > (XEN) IRQ211 -> 0:20 > (XEN) IRQ43 -> 0:22 > (XEN) IRQ168 -> 0:23 > (XEN) .................................... done. > (XEN) Xen BUG at io_apic.c:556 > (XEN) ----[ Xen-4.1.5-rc1 x86_64 debug=y Tainted: C ]---- > (XEN) CPU: 0 > (XEN) RIP: e008:[<ffff82c48015e2db>] > smp_irq_move_cleanup_interrupt+0x216/0x28e > (XEN) RFLAGS: 0000000000010092 CONTEXT: hypervisor > (XEN) rax: 0000000000000000 rbx: 00000000000000e9 rcx: 0000000000000000 > (XEN) rdx: 0000000000000000 rsi: 000000000000000a rdi: ffff82c4802592e0 > (XEN) rbp: ffff82c48029ff08 rsp: ffff82c48029feb8 r8: 0000000000000004 > (XEN) r9: 0000000000000004 r10: 0000000000000004 r11: 0000000000000002 > (XEN) r12: ffff830421080250 r13: ffff830421060534 r14: ffff82c48029ff18 > (XEN) r15: ffff82c4802dd9e0 cr0: 000000008005003b cr4: 00000000000026f0 > (XEN) cr3: 000000026582c000 cr2: ffff8804020701d8 > (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: 0000 cs: e008 > (XEN) Xen stack trace from rsp=ffff82c48029feb8: > (XEN) 0000000000000000 ffff82c48029ff18 ffff82c4802dd9e0 000000e900000000 > (XEN) 000000000000e02b 0000000000000000 000000004bf51982 00000000000060a9 > (XEN) 0000000000000000 0000000000000000 00007d3b7fd600c7 ffff82c48014de60 > (XEN) 0000000000000000 0000000000000000 00000000000060a9 000000004bf51982 > (XEN) ffff8802d2665b28 0000000000000000 0000000000000000 0000000000007ff0 > (XEN) 0000000000000022 0000000000000000 000000024bf57322 0000000001307da0 > (XEN) 00000000000059a0 0000000000000000 00000000000060a9 0000002000000000 > (XEN) ffffffff8123c51a 000000000000e033 0000000000000293 ffff8802d2665b08 > (XEN) 000000000000e02b 0000000000000000 0000000000000000 0000000000000000 > (XEN) 0000000000000000 0000000000000000 ffff8300ca9a0000 0000000000000000 > (XEN) 0000000000000000 > (XEN) Xen call trace: > (XEN) [<ffff82c48015e2db>] smp_irq_move_cleanup_interrupt+0x216/0x28e > > > So vector e9 doesn't appear to be programmed in anywhere. I am starting to get more into the realm of guessing here but, can you use apic_verbosity=debug on the command line and copy this extra debugging logic to send_cleanup_vector() You should be able to conditionally trigger it on "desc->arch.vector == 0xe9". You will probably also want to change the BUG() to a WARN(), so we get the interrupt and ioapic information on both sides of the cleanup vector, as well as getting the stack trace of the codepath through Xen as a result of vector 0xe9. ~Andrew _______________________________________________ Xen-devel mailing list Xen-devel@xxxxxxxxxxxxx http://lists.xen.org/xen-devel

©2013 Xen Project, A Linux Foundation Collaborative Project. All Rights Reserved.
Linux Foundation is a registered trademark of The Linux Foundation.
Xen Project is a trademark of The Linux Foundation.