Xen project Mailing List

Re: [Xen-devel] High CPU temp, suspend problem - xen 4.1.5-pre, linux 3.7.x

To: Andrew Cooper <andrew.cooper3@xxxxxxxxxx>

From: Marek Marczykowski <marmarek@xxxxxxxxxxxxxxxxxxxxxx>

Date: Wed, 27 Mar 2013 09:50:07 +0100

Cc: "xen-devel@xxxxxxxxxxxxx" <xen-devel@xxxxxxxxxxxxx>, Jan Beulich <JBeulich@xxxxxxxx>, Konrad Rzeszutek Wilk <konrad.wilk@xxxxxxxxxx>

Delivery-date: Wed, 27 Mar 2013 08:50:40 +0000

List-id: Xen developer discussion <xen-devel.lists.xen.org>

On 26.03.2013 19:50, Andrew Cooper wrote: > On 26/03/2013 18:21, Marek Marczykowski wrote: >> On 26.03.2013 18:54, Andrew Cooper wrote: >>>>> Can you replace the ASSERT() with code similar to that in >>>>> >>>>> http://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=xen/arch/x86/irq.c;h=5e0f463c381750090373dabd8967635bc297d457;hb=refs/heads/staging#l668 >>>>> >>>>> Which should call dump_irqs() in before dying because of the ASSERT. >>>>> You might need to also take the latest version of dump_irqs() from >>>>> unstable, as I seem to remember there was another assertion failure due >>>>> to xfree()'ing in IRQ context. >>>> Full log here: >>>> http://duch.mimuw.edu.pl/~marmarek/qubes/xen-4.1-failed-resume-dump-irqs.log >>>> Interesting part: >> (...) >>> Even more curious. vector e9 does not appear to be programmed in. Can >>> you extend the debugging to also call __print_IO_APIC(). >>> >>> The i debug key and z debug key list IO-APIC entries from different >>> sources of information. >> As you wish, full log: >> http://duch.mimuw.edu.pl/~marmarek/qubes/xen-4.1-failed-resume-dump-irqs2.log >> >> Final part: >> (XEN) *** IRQ BUG found *** >> (XEN) CPU0 -Testing vector 233 from bitmap >> 43,49,64,72,80,87-88,95-96,103,112,119-121,127,135,143-144,151-152,159-160,168,192,197,200,211,216,218 >> (XEN) Guest interrupt information: >> (XEN) IRQ: 0 affinity:00000000,00000000,00000000,00000001 vec:f0 >> type=IO-APIC-edge status=00000000 mapped, unbound >> (XEN) IRQ: 1 affinity:00000000,00000000,00000000,00000001 vec:7f >> type=IO-APIC-edge status=00000050 in-flight=0 domain-list=0: 1(-S--), >> (XEN) IRQ: 2 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:e2 >> type=XT-PIC status=00000000 mapped, unbound >> (XEN) IRQ: 3 affinity:00000000,00000000,00000000,00000001 vec:40 >> type=IO-APIC-edge status=00000002 mapped, unbound >> (XEN) IRQ: 4 affinity:00000000,00000000,00000000,00000001 vec:f1 >> type=IO-APIC-edge status=00000000 mapped, unbound >> (XEN) IRQ: 5 affinity:00000000,00000000,00000000,00000001 vec:48 >> type=IO-APIC-edge status=00000002 mapped, unbound >> (XEN) IRQ: 6 affinity:00000000,00000000,00000000,00000001 vec:50 >> type=IO-APIC-edge status=00000002 mapped, unbound >> (XEN) IRQ: 7 affinity:00000000,00000000,00000000,00000008 vec:da >> type=IO-APIC-edge status=00000050 in-flight=0 domain-list=0: 7(-S--), >> (XEN) IRQ: 8 affinity:00000000,00000000,00000000,00000004 vec:d8 >> type=IO-APIC-edge status=00000050 in-flight=0 domain-list=0: 8(-S--), >> (XEN) IRQ: 9 affinity:00000000,00000000,00000000,00000001 vec:87 >> type=IO-APIC-level status=00000010 in-flight=0 domain-list=0: 9(-S--), >> (XEN) IRQ: 10 affinity:00000000,00000000,00000000,00000001 vec:70 >> type=IO-APIC-edge status=00000002 mapped, unbound >> (XEN) IRQ: 11 affinity:00000000,00000000,00000000,00000001 vec:78 >> type=IO-APIC-edge status=00000002 mapped, unbound >> (XEN) IRQ: 12 affinity:00000000,00000000,00000000,00000001 vec:8f >> type=IO-APIC-edge status=00000050 in-flight=0 domain-list=0: 12(-S--), >> (XEN) IRQ: 13 affinity:00000000,00000000,00000000,0000000f vec:90 >> type=IO-APIC-edge status=00000002 mapped, unbound >> (XEN) IRQ: 14 affinity:00000000,00000000,00000000,00000001 vec:98 >> type=IO-APIC-edge status=00000002 mapped, unbound >> (XEN) IRQ: 15 affinity:00000000,00000000,00000000,00000001 vec:a0 >> type=IO-APIC-edge status=00000002 mapped, unbound >> (XEN) IRQ: 16 affinity:00000000,00000000,00000000,00000001 vec:97 >> type=IO-APIC-level status=00000050 in-flight=0 domain-list=0: 16(-S--), >> (XEN) IRQ: 17 affinity:00000000,00000000,00000000,00000001 vec:9f >> type=IO-APIC-level status=00000050 in-flight=0 domain-list=0: 17(-S--), >> (XEN) IRQ: 18 affinity:00000000,00000000,00000000,00000004 vec:79 >> type=IO-APIC-level status=00000002 mapped, unbound >> (XEN) IRQ: 19 affinity:00000000,00000000,00000000,0000000f vec:c8 >> type=IO-APIC-level status=00000002 mapped, unbound >> (XEN) IRQ: 20 affinity:00000000,00000000,00000000,00000002 vec:d3 >> type=IO-APIC-level status=00000050 in-flight=0 domain-list=0: 20(-S--), >> (XEN) IRQ: 22 affinity:00000000,00000000,00000000,0000000f vec:2b >> type=IO-APIC-level status=00000002 mapped, unbound >> (XEN) IRQ: 23 affinity:00000000,00000000,00000000,0000000f vec:a8 >> type=IO-APIC-edge status=00000002 mapped, unbound >> (XEN) IRQ: 24 affinity:00000000,00000000,00000000,00000001 vec:28 >> type=DMA_MSI status=00000000 mapped, unbound >> (XEN) IRQ: 25 affinity:00000000,00000000,00000000,00000001 vec:30 >> type=DMA_MSI status=00000000 mapped, unbound >> (XEN) IRQ: 26 affinity:00000000,00000000,00000000,00000001 vec:c7 >> type=PCI-MSI status=00000010 in-flight=0 domain-list=0:279(-S--), >> (XEN) IRQ: 27 affinity:00000000,00000000,00000000,00000001 vec:cf >> type=PCI-MSI status=00000050 in-flight=0 domain-list=0:278(-S--), >> (XEN) IRQ: 28 affinity:00000000,00000000,00000000,00000001 vec:d7 >> type=PCI-MSI status=00000050 in-flight=0 domain-list=0:277(-S--), >> (XEN) IRQ: 29 affinity:00000000,00000000,00000000,00000001 vec:df >> type=PCI-MSI status=00000050 in-flight=0 domain-list=0:276(-S--), >> (XEN) IRQ: 30 affinity:00000000,00000000,00000000,00000001 vec:38 >> type=PCI-MSI status=00000050 in-flight=0 domain-list=0:275(-S--), >> (XEN) IRQ: 31 affinity:00000000,00000000,00000000,00000004 vec:47 >> type=PCI-MSI status=00000002 mapped, unbound >> (XEN) IRQ: 32 affinity:00000000,00000000,00000000,00000001 vec:a7 >> type=PCI-MSI status=00000050 in-flight=0 domain-list=0:273(-S--), >> (XEN) IRQ: 33 affinity:00000000,00000000,00000000,00000001 vec:b7 >> type=PCI-MSI status=00000010 in-flight=0 domain-list=0:272(-S--), >> (XEN) IRQ: 34 affinity:00000000,00000000,00000000,00000004 vec:40 >> type=PCI-MSI status=00000010 in-flight=0 domain-list=0:271(PS--), >> (XEN) IRQ: 35 affinity:00000000,00000000,00000000,00000001 vec:af >> type=PCI-MSI status=00000050 in-flight=0 domain-list=1: 55(-S--), >> (XEN) IO-APIC interrupt information: >> (XEN) IRQ 0 Vec240: >> (XEN) Apic 0x00, Pin 2: vec=f0 delivery=LoPri dest=L status=0 >> polarity=0 irr=0 trig=E mask=0 dest_id:0 >> (XEN) IRQ 1 Vec127: >> (XEN) Apic 0x00, Pin 1: vec=7f delivery=LoPri dest=L status=0 >> polarity=0 irr=0 trig=E mask=0 dest_id:0 >> (XEN) IRQ 3 Vec 64: >> (XEN) Apic 0x00, Pin 3: vec=40 delivery=LoPri dest=L status=0 >> polarity=0 irr=0 trig=E mask=0 dest_id:0 >> (XEN) IRQ 4 Vec241: >> (XEN) Apic 0x00, Pin 4: vec=f1 delivery=LoPri dest=L status=0 >> polarity=0 irr=0 trig=E mask=0 dest_id:0 >> (XEN) IRQ 5 Vec 72: >> (XEN) Apic 0x00, Pin 5: vec=48 delivery=LoPri dest=L status=0 >> polarity=0 irr=0 trig=E mask=0 dest_id:0 >> (XEN) IRQ 6 Vec 80: >> (XEN) Apic 0x00, Pin 6: vec=50 delivery=LoPri dest=L status=0 >> polarity=0 irr=0 trig=E mask=0 dest_id:0 >> (XEN) IRQ 7 Vec218: >> (XEN) Apic 0x00, Pin 7: vec=da delivery=LoPri dest=L status=0 >> polarity=0 irr=0 trig=E mask=0 dest_id:0 >> (XEN) IRQ 8 Vec216: >> (XEN) Apic 0x00, Pin 8: vec=d8 delivery=LoPri dest=L status=0 >> polarity=0 irr=0 trig=E mask=0 dest_id:0 >> (XEN) IRQ 9 Vec135: >> (XEN) Apic 0x00, Pin 9: vec=87 delivery=LoPri dest=L status=0 >> polarity=0 irr=0 trig=L mask=0 dest_id:0 >> (XEN) IRQ 10 Vec112: >> (XEN) Apic 0x00, Pin 10: vec=70 delivery=LoPri dest=L status=0 >> polarity=0 irr=0 trig=E mask=0 dest_id:0 >> (XEN) IRQ 11 Vec120: >> (XEN) Apic 0x00, Pin 11: vec=78 delivery=LoPri dest=L status=0 >> polarity=0 irr=0 trig=E mask=0 dest_id:0 >> (XEN) IRQ 12 Vec143: >> (XEN) Apic 0x00, Pin 12: vec=8f delivery=LoPri dest=L status=0 >> polarity=0 irr=0 trig=E mask=0 dest_id:0 >> (XEN) IRQ 13 Vec144: >> (XEN) Apic 0x00, Pin 13: vec=90 delivery=LoPri dest=L status=0 >> polarity=0 irr=0 trig=E mask=1 dest_id:0 >> (XEN) IRQ 14 Vec152: >> (XEN) Apic 0x00, Pin 14: vec=98 delivery=LoPri dest=L status=0 >> polarity=0 irr=0 trig=E mask=0 dest_id:0 >> (XEN) IRQ 15 Vec160: >> (XEN) Apic 0x00, Pin 15: vec=a0 delivery=LoPri dest=L status=0 >> polarity=0 irr=0 trig=E mask=0 dest_id:0 >> (XEN) IRQ 16 Vec151: >> (XEN) Apic 0x00, Pin 16: vec=97 delivery=LoPri dest=L status=0 >> polarity=1 irr=0 trig=L mask=0 dest_id:0 >> (XEN) IRQ 17 Vec159: >> (XEN) Apic 0x00, Pin 17: vec=9f delivery=LoPri dest=L status=0 >> polarity=1 irr=0 trig=L mask=0 dest_id:0 >> (XEN) IRQ 18 Vec121: >> (XEN) Apic 0x00, Pin 18: vec=79 delivery=LoPri dest=L status=0 >> polarity=1 irr=0 trig=L mask=1 dest_id:0 >> (XEN) IRQ 19 Vec200: >> (XEN) Apic 0x00, Pin 19: vec=c8 delivery=LoPri dest=L status=0 >> polarity=1 irr=0 trig=L mask=1 dest_id:0 >> (XEN) IRQ 20 Vec211: >> (XEN) Apic 0x00, Pin 20: vec=d3 delivery=LoPri dest=L status=0 >> polarity=1 irr=0 trig=L mask=0 dest_id:0 >> (XEN) IRQ 22 Vec 43: >> (XEN) Apic 0x00, Pin 22: vec=2b delivery=LoPri dest=L status=0 >> polarity=1 irr=0 trig=L mask=1 dest_id:0 >> (XEN) IRQ 23 Vec168: >> (XEN) Apic 0x00, Pin 23: vec=a8 delivery=LoPri dest=L status=0 >> polarity=0 irr=0 trig=E mask=1 dest_id:0 >> (XEN) number of MP IRQ sources: 15. >> (XEN) number of IO-APIC #2 registers: 24. >> (XEN) testing the IO APIC....................... >> (XEN) IO APIC #2...... >> (XEN) .... register #00: 02000000 >> (XEN) ....... : physical APIC id: 02 >> (XEN) ....... : Delivery Type: 0 >> (XEN) ....... : LTS : 0 >> (XEN) .... register #01: 00170020 >> (XEN) ....... : max redirection entries: 0017 >> (XEN) ....... : PRQ implemented: 0 >> (XEN) ....... : IO APIC version: 0020 >> (XEN) .... IRQ redirection table: >> (XEN) NR Log Phy Mask Trig IRR Pol Stat Dest Deli Vect: >> (XEN) 00 0DC 0C 1 0 0 0 0 1 2 87 >> (XEN) 01 000 00 0 0 0 0 0 1 1 7F >> (XEN) 02 000 00 0 0 0 0 0 1 1 F0 >> (XEN) 03 000 00 0 0 0 0 0 1 1 40 >> (XEN) 04 000 00 0 0 0 0 0 1 1 F1 >> (XEN) 05 000 00 0 0 0 0 0 1 1 48 >> (XEN) 06 000 00 0 0 0 0 0 1 1 50 >> (XEN) 07 000 00 0 0 0 0 0 1 1 DA >> (XEN) 08 000 00 0 0 0 0 0 1 1 D8 >> (XEN) 09 000 00 0 1 0 0 0 1 1 87 >> (XEN) 0a 000 00 0 0 0 0 0 1 1 70 >> (XEN) 0b 000 00 0 0 0 0 0 1 1 78 >> (XEN) 0c 000 00 0 0 0 0 0 1 1 8F >> (XEN) 0d 000 00 1 0 0 0 0 1 1 90 >> (XEN) 0e 000 00 0 0 0 0 0 1 1 98 >> (XEN) 0f 000 00 0 0 0 0 0 1 1 A0 >> (XEN) 10 000 00 0 1 0 1 0 1 1 97 >> (XEN) 11 000 00 0 1 0 1 0 1 1 9F >> (XEN) 12 000 00 1 1 0 1 0 1 1 79 >> (XEN) 13 000 00 1 1 0 1 0 1 1 C8 >> (XEN) 14 000 00 0 1 0 1 0 1 1 D3 >> (XEN) 15 000 00 1 0 0 0 0 0 0 00 >> (XEN) 16 000 00 1 1 0 1 0 1 1 2B >> (XEN) 17 000 00 1 0 0 0 0 1 1 A8 >> (XEN) Using vector-based indexing >> (XEN) IRQ to pin mappings: >> (XEN) IRQ240 -> 0:2 >> (XEN) IRQ127 -> 0:1 >> (XEN) IRQ64 -> 0:3 >> (XEN) IRQ241 -> 0:4 >> (XEN) IRQ72 -> 0:5 >> (XEN) IRQ80 -> 0:6 >> (XEN) IRQ218 -> 0:7 >> (XEN) IRQ216 -> 0:8 >> (XEN) IRQ135 -> 0:9 >> (XEN) IRQ112 -> 0:10 >> (XEN) IRQ120 -> 0:11 >> (XEN) IRQ143 -> 0:12 >> (XEN) IRQ144 -> 0:13 >> (XEN) IRQ152 -> 0:14 >> (XEN) IRQ160 -> 0:15 >> (XEN) IRQ151 -> 0:16 >> (XEN) IRQ159 -> 0:17 >> (XEN) IRQ121 -> 0:18 >> (XEN) IRQ200 -> 0:19 >> (XEN) IRQ211 -> 0:20 >> (XEN) IRQ43 -> 0:22 >> (XEN) IRQ168 -> 0:23 >> (XEN) .................................... done. >> (XEN) Xen BUG at io_apic.c:556 >> (XEN) ----[ Xen-4.1.5-rc1 x86_64 debug=y Tainted: C ]---- >> (XEN) CPU: 0 >> (XEN) RIP: e008:[<ffff82c48015e2db>] >> smp_irq_move_cleanup_interrupt+0x216/0x28e >> (XEN) RFLAGS: 0000000000010092 CONTEXT: hypervisor >> (XEN) rax: 0000000000000000 rbx: 00000000000000e9 rcx: 0000000000000000 >> (XEN) rdx: 0000000000000000 rsi: 000000000000000a rdi: ffff82c4802592e0 >> (XEN) rbp: ffff82c48029ff08 rsp: ffff82c48029feb8 r8: 0000000000000004 >> (XEN) r9: 0000000000000004 r10: 0000000000000004 r11: 0000000000000002 >> (XEN) r12: ffff830421080250 r13: ffff830421060534 r14: ffff82c48029ff18 >> (XEN) r15: ffff82c4802dd9e0 cr0: 000000008005003b cr4: 00000000000026f0 >> (XEN) cr3: 000000026582c000 cr2: ffff8804020701d8 >> (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: 0000 cs: e008 >> (XEN) Xen stack trace from rsp=ffff82c48029feb8: >> (XEN) 0000000000000000 ffff82c48029ff18 ffff82c4802dd9e0 000000e900000000 >> (XEN) 000000000000e02b 0000000000000000 000000004bf51982 00000000000060a9 >> (XEN) 0000000000000000 0000000000000000 00007d3b7fd600c7 ffff82c48014de60 >> (XEN) 0000000000000000 0000000000000000 00000000000060a9 000000004bf51982 >> (XEN) ffff8802d2665b28 0000000000000000 0000000000000000 0000000000007ff0 >> (XEN) 0000000000000022 0000000000000000 000000024bf57322 0000000001307da0 >> (XEN) 00000000000059a0 0000000000000000 00000000000060a9 0000002000000000 >> (XEN) ffffffff8123c51a 000000000000e033 0000000000000293 ffff8802d2665b08 >> (XEN) 000000000000e02b 0000000000000000 0000000000000000 0000000000000000 >> (XEN) 0000000000000000 0000000000000000 ffff8300ca9a0000 0000000000000000 >> (XEN) 0000000000000000 >> (XEN) Xen call trace: >> (XEN) [<ffff82c48015e2db>] smp_irq_move_cleanup_interrupt+0x216/0x28e >> >> >> > > So vector e9 doesn't appear to be programmed in anywhere. > > I am starting to get more into the realm of guessing here but, can you > use apic_verbosity=debug on the command line and copy this extra > debugging logic to send_cleanup_vector() > > You should be able to conditionally trigger it on "desc->arch.vector == > 0xe9". You will probably also want to change the BUG() to a WARN(), so > we get the interrupt and ioapic information on both sides of the cleanup > vector, as well as getting the stack trace of the codepath through Xen > as a result of vector 0xe9. send_cleanup_vector() doesn't seem to be called with cfg->vector == 0xe9... Can dom0 mess something here around? -- Best Regards / Pozdrawiam, Marek Marczykowski Invisible Things Lab

Attachment: signature.asc
Description: OpenPGP digital signature

_______________________________________________ Xen-devel mailing list Xen-devel@xxxxxxxxxxxxx http://lists.xen.org/xen-devel

©2013 Xen Project, A Linux Foundation Collaborative Project. All Rights Reserved.
Linux Foundation is a registered trademark of The Linux Foundation.
Xen Project is a trademark of The Linux Foundation.