Re: [Xen-devel] High CPU temp, suspend problem - xen 4.1.5-pre, linux 3.7.x

[Marek Marczykowski <marmarek@xxxxxxxxxxxxxxxxxxxxxx>]

On 27.03.2013 19:56, Andrew Cooper wrote:
> On 27/03/2013 18:16, Marek Marczykowski wrote:
>> On 27.03.2013 17:27, Andrew Cooper wrote:
>>> On 27/03/2013 15:51, Marek Marczykowski wrote:
>>>> On 27.03.2013 15:49, Marek Marczykowski wrote:
>>>>> On 27.03.2013 15:46, Andrew Cooper wrote:
>>>>>> As for locating the cause of the legacy vectors, it might be a good idea
>>>>>> to stick a printk at the top of do_IRQ() which indicates an interrupt
>>>>>> with vector between 0xe0 and 0xef.  This might at least indicate whether
>>>>>> legacy vectors are genuinely being delivered, or whether we have some
>>>>>> memory corruption causing these effects.
>>>>> Ok, will try something like this.
>>>> Nothing interesting here...
>>>> Only vector 0xf1 for irq 4 and 0xf0 for irq 0 (which match irq dump 
>>>> information).
>>>>
>>> Even in the case where we hit the original assertion?
>> Yes, even then.
>>
>>> If so, then all I can thing is that the move_pending flag for that
>>> specific GSI has been corrupted in memory somehow.
>> I guest this isn't the case, see below.
>>
>>> I wonder if hexdumping irq_desc[9] after setup, before sleep, on resume
>>> and in the case of the assertion failure might give some hints.
>> I've tried something like this. Detailed log here:
>> http://duch.mimuw.edu.pl/~marmarek/qubes/xen-4.1-suspend-irq9-dump.log
> 
> This is concerning, unless I am getting utterly confused.  Jan: Do you
> mind double checking my reasoning?
> 
> irq 0 through 15 should be the PIC irqs, set up in init_IRQ() in
> arch/x86/i8259.c
> 
> irq9 should be the irq for the PIC vector which is set up as 0xe9, and
> its vector should never change.
> 
> Could you put in extra checks for the sanity of per_cpu(vector_irq,
> cpu)[0xe0 thru 0xef] ?

Ok, got something here:
http://duch.mimuw.edu.pl/~marmarek/qubes/xen-4.1-suspend-irq9-dump2.log

Now bug triggered after some time after resume (about 15s). But only CPU0 by
scheduler immediately after resume. Interesting part - note vector_irq(e1):
(XEN) irq_cfg of IRQ 9:
(XEN)   vector: 188
(XEN)   cpu_mask: 00000000,00000000,00000000,00000001
(XEN)   old_cpu_mask: 00000000,00000000,00000000,00000002
(XEN)   move_cleanup_count: 0x0
(XEN)   used_vectors:
49,64,72,74,80-81,88,98,112,120,144,148,152,156,160,164,168,172,178,188,192,196,200,207-208
(XEN)   move_in_progress: 0x0
(XEN) irq_desc of IRQ 9:
(XEN)   status: 16
(XEN)   handler: ffff82c480252660
(XEN)   msi_desc: 0000000000000000
(XEN)   action: ffff83041d9f1ed0
(XEN)   depth: 0
(XEN)   chip_data: ffff830421080250
(XEN)   irq: 9
(XEN)   affinity: 00000000,00000000,00000000,00000001
(XEN)   pending_mask: 00000000,00000000,00000000,00000000
(XEN)   (...)
(XEN) vector_irq(e0): 0
(XEN) vector_irq(e1): -1
(XEN) vector_irq(e2): 2
(XEN) vector_irq(e3): 3
(XEN) vector_irq(e4): 4
(XEN) vector_irq(e5): 5
(XEN) vector_irq(e6): 6
(XEN) vector_irq(e7): 7
(XEN) vector_irq(e8): 8
(XEN) vector_irq(e9): 9
(XEN) vector_irq(ea): 10
(XEN) vector_irq(eb): 11
(XEN) vector_irq(ec): 12
(XEN) vector_irq(ed): 13
(XEN) vector_irq(ee): 14
(XEN) vector_irq(ef): 15
(XEN) Xen WARN at io_apic.c:639
(XEN) ----[ Xen-4.1.5-rc1  x86_64  debug=y  Not tainted ]----
(XEN) CPU:    0
(XEN) RIP:    e008:[<ffff82c48015e5fb>] 
smp_irq_move_cleanup_interrupt+0x246/0x2c6
(XEN) RFLAGS: 0000000000010046   CONTEXT: hypervisor
(XEN) rax: 0000000000000000   rbx: 00000000000000e1   rcx: 0000000000000000
(XEN) rdx: 0000000000000000   rsi: 000000000000000a   rdi: ffff82c4802592e0
(XEN) rbp: ffff82c48029fda8   rsp: ffff82c48029fd58   r8:  0000000000000004
(XEN) r9:  0000000000000001   r10: 000000000000000f   r11: 0000000000000002
(XEN) r12: ffff830421080050   r13: ffff830421060134   r14: ffff82c48029ff18
(XEN) r15: ffff82c4802dd9e0   cr0: 000000008005003b   cr4: 00000000000026f0
(XEN) cr3: 0000000273d3c000   cr2: ffff88000c360318
(XEN) ds: 002b   es: 002b   fs: 0000   gs: 0000   ss: e010   cs: e008
(XEN) Xen stack trace from rsp=ffff82c48029fd58:
(XEN)    0000000000000000 000000008029fd70 ffff82c48029ff18 ffff82c4802dd9e0
(XEN)    ffff82c480153f55 ffff830421043260 ffff830421043320 0000006f207ab134
(XEN)    0000006f207c3b14 ffff82c4802dd600 00007d3b7fd60227 ffff82c48014de60
(XEN)    ffff82c4802dd600 0000006f207c3b14 0000006f207ab134 ffff830421043320
(XEN)    ffff82c48029fef0 ffff830421043260 0000ffff0000ffff 0000006f416dab2e
(XEN)    ffff830007ef4060 0000006f1fad2570 0000000000003f40 0000000000000001
(XEN)    0000000000000000 ffff82c4802de200 0000000002048cac 0000002000000000
(XEN)    ffff82c480197940 000000000000e008 0000000000000246 ffff82c48029fe68
(XEN)    000000000000e010 ffff82c48029fef0 ffff82c4801987b7 ffff880402105d30
(XEN)    00000000ca9a4000 ffffffffffffffff aaaaaaaaaaaaaa00 aaaaaaaaaaaaaaaa
(XEN)    0000006f21136437 0000000000000000 0000000000000000 ffffffffffffffff
(XEN)    000004c200000542 0000000000000000 ffff82c48029ff18 ffff82c48029ff18
(XEN)    00000000ffffffff 0000000000000002 ffff82c4802dd600 ffff82c48029ff10
(XEN)    ffff82c4801549ce ffff8300ca9a4000 ffff8300ca666000 ffff82c48029fdc8
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000001
(XEN)    ffff880402105f00 ffff880402105fd8 0000000000000246 0000000000000001
(XEN)    0000000000000000 0000000000000000 0000000000000000 ffffffff810013aa
(XEN)    ffffffff81a2a858 00000000deadbeef 00000000deadbeef 0000010000000000
(XEN)    ffffffff810013aa 000000000000e033 0000000000000246 ffff880402105ee8
(XEN)    000000000000e02b 0000000000000000 0000000000000000 0000000000000000
(XEN) Xen call trace:
(XEN)    [<ffff82c48015e5fb>] smp_irq_move_cleanup_interrupt+0x246/0x2c6
(XEN)    [<ffff82c48014de60>] irq_move_cleanup_interrupt+0x30/0x40
(XEN)    [<ffff82c480197940>] lapic_timer_nop+0x0/0x6
(XEN)    [<ffff82c4801549ce>] idle_loop+0x4b/0x59



Ignore rest of comments from my previous mail - I clearly don't understand IRQ
handling code.

-- 
Best Regards / Pozdrawiam,
Marek Marczykowski
Invisible Things Lab

Attachment: signature.asc
Description: OpenPGP digital signature

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel