[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] High CPU temp, suspend problem - xen 4.1.5-pre, linux 3.7.x



On 26.03.2013 18:02, Andrew Cooper wrote:
> On 26/03/2013 16:45, Marek Marczykowski wrote:
>> On 26.03.2013 17:03, Jan Beulich wrote:
>>>>>> On 26.03.13 at 14:50, Marek Marczykowski 
>>>>>> <marmarek@xxxxxxxxxxxxxxxxxxxxxx>
>>> wrote:
>>>> On 26.03.2013 14:11, Jan Beulich wrote:
>>>>>>>> On 26.03.13 at 13:17, Marek Marczykowski 
>>>>>>>> <marmarek@xxxxxxxxxxxxxxxxxxxxxx> 
>>>> wrote:
>>>>>> Finally got serial console :)
>>>>>> The debug=y problem is (actually at resume):
>>>>>> (XEN) Assertion 'test_bit(vector, cfg->used_vectors)' failed at 
>>>>>> io_apic.c:542
>>>>>> (XEN) ----[ Xen-4.1.5-rc1  x86_64  debug=y  Tainted:    C ]----
>>>>>> (XEN) CPU:    0
>>>>>> (XEN) RIP:    e008:[<ffff82c48015e288>] 
>>>>>> smp_irq_move_cleanup_interrupt+0x1c3/0x23d
>>>>>> (XEN) RFLAGS: 0000000000010046   CONTEXT: hypervisor
>>>>>> (XEN) rax: 0000000000000000   rbx: 00000000000000e9   rcx: 
>>>>>> ffff82c48029ff18
>>>>>> (XEN) rdx: 00000000000000e9   rsi: 000000000000002a   rdi: 
>>>>>> ffff830421060538
>>>>>> (XEN) rbp: ffff82c48029ff08   rsp: ffff82c48029feb8   r8:  
>>>>>> ffff88041820eb60
>>>>>> (XEN) r9:  0000000000000000   r10: 0000000000007ff0   r11: 
>>>>>> 0000000000000000
>>>>>> (XEN) r12: ffff830421080250   r13: ffff830421060534   r14: 
>>>>>> ffff82c48029ff18
>>>>>> (XEN) r15: ffff82c4802dd9e0   cr0: 000000008005003b   cr4: 
>>>>>> 00000000000026f0
>>>>>> (XEN) cr3: 0000000300b81000   cr2: ffff880402070198
>>>>>> (XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: e008
>>>>>> (XEN) Xen stack trace from rsp=ffff82c48029feb8:
>>>>>> (XEN)    0000000000000000 000000000000e030 ffff82c48029ff18 
>>>>>> ffff82c4802dd9e0
>>>>>> (XEN)    ffff8802cac3c7c0 00000000ffff3729 00000000ffff3729 
>>>>>> 000000013fff3728
>>>>>> (XEN)    ffffffff81b907c0 00000000ffff3729 00007d3b7fd600c7 
>>>>>> ffff82c48014de60
>>>>>> (XEN)    00000000ffff3729 ffffffff81b907c0 000000013fff3728 
>>>>>> 00000000ffff3729
>>>>>> (XEN)    ffffffff81a01e18 00000000ffff3729 0000000000000000 
>>>>>> 0000000000007ff0
>>>>>> (XEN)    0000000000000000 ffff88041820eb60 ffff8803fd1820a8 
>>>>>> ffffffff81b90a88
>>>>>> (XEN)    000000000000002a 000000000000002a 00000000ffff372a 
>>>>>> 0000002000000000
>>>>>> (XEN)    ffffffff8105dd5a 000000000000e033 0000000000000246 
>>>>>> ffffffff81a01db8
>>>>>> (XEN)    000000000000e02b 0000000000000000 0000000000000000 
>>>>>> 0000000000000000
>>>>>> (XEN)    0000000000000000 0000000000000000 ffff8300ca9a0000 
>>>>>> 0000000000000000
>>>>>> (XEN)    0000000000000000
>>>>>> (XEN) Xen call trace:
>>>>>> (XEN)    [<ffff82c48015e288>] smp_irq_move_cleanup_interrupt+0x1c3/0x23d
>>>>>> (XEN)
>>>>>> (XEN)
>>>>>> (XEN) ****************************************
>>>>>> (XEN) Panic on CPU 0:
>>>>>> (XEN) Assertion 'test_bit(vector, cfg->used_vectors)' failed at 
>>>>>> io_apic.c:542
>>>>>> (XEN) ****************************************
>>>>> To make sense of this, we need to know the register (and maybe
>>>>> stack) allocation at this point, to know which vector it was that
>>>>> triggered the assertion. You can either do this analysis for us, or
>>>>> point us at the xen-syms binary matching the xen.gz you used.
>>>> "info scope smp_irq_move_cleanup_interrupt" said vector is in %rbx, so 
>>>> 0xe9.
>>> And that system isn't using a strange mixed mode IO-APIC/legacy
>>> PIC model, where particularly IRQ 9 (usually ACPI SCI) gets
>>> channeled through the legacy PIC?
>> I don't know...
>>
>>> Could you attach the complete log, ideally with 'i' output logged
>>> right before suspending?
>> Sure, attached.
>>
>>> Is this reproducible with 4.2.x or 4.3-unstable? If not, but if readily
>>> reproducible with 4.1.5-rc1, could you try changing the containing
>>> loop's upper bound from "< NR_VECTORS" to
>>> "<= LAST_DYNAMIC_VECTOR"?
>> I've tried 4.2.x some time ago and bug also exists there (but I had not
>> console, so not sure if exactly the same). 4.3 seems to be not affected.

Checked 4.2 and indeed also assert() in similar place. If anyone interested,
log here:
http://duch.mimuw.edu.pl/~marmarek/qubes/console-4.2-failed-resume.log

>>
> 
> Can you replace the ASSERT() with code similar to that in
> 
> http://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=xen/arch/x86/irq.c;h=5e0f463c381750090373dabd8967635bc297d457;hb=refs/heads/staging#l668
> 
> Which should call dump_irqs() in before dying because of the ASSERT. 
> You might need to also take the latest version of dump_irqs() from
> unstable, as I seem to remember there was another assertion failure due
> to xfree()'ing in IRQ context.

Full log here:
http://duch.mimuw.edu.pl/~marmarek/qubes/xen-4.1-failed-resume-dump-irqs.log
Interesting part:
(XEN) *** IRQ BUG found ***
(XEN) CPU0 -Testing vector 233 from bitmap
39,47,63-65,72,80,88,96,98,112,120,125,144,152,160,168,174,182-183,190,192,198,200,208,214,222
(XEN) Guest interrupt information:
(XEN)    IRQ:   0 affinity:00000000,00000000,00000000,00000001 vec:f0
type=IO-APIC-edge    status=00000000 mapped, unbound
(XEN)    IRQ:   1 affinity:00000000,00000000,00000000,00000002 vec:c6
type=IO-APIC-edge    status=00000050 in-flight=0 domain-list=0:  1(-S--),
(XEN)    IRQ:   2 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:e2
type=XT-PIC          status=00000000 mapped, unbound
(XEN)    IRQ:   3 affinity:00000000,00000000,00000000,00000001 vec:40
type=IO-APIC-edge    status=00000002 mapped, unbound
(XEN)    IRQ:   4 affinity:00000000,00000000,00000000,00000001 vec:f1
type=IO-APIC-edge    status=00000000 mapped, unbound
(XEN)    IRQ:   5 affinity:00000000,00000000,00000000,00000001 vec:48
type=IO-APIC-edge    status=00000002 mapped, unbound
(XEN)    IRQ:   6 affinity:00000000,00000000,00000000,00000001 vec:50
type=IO-APIC-edge    status=00000002 mapped, unbound
(XEN)    IRQ:   7 affinity:00000000,00000000,00000000,00000001 vec:58
type=IO-APIC-edge    status=00000050 in-flight=0 domain-list=0:  7(-S--),
(XEN)    IRQ:   8 affinity:00000000,00000000,00000000,00000001 vec:60
type=IO-APIC-edge    status=00000050 in-flight=0 domain-list=0:  8(-S--),
(XEN)    IRQ:   9 affinity:00000000,00000000,00000000,00000001 vec:de
type=IO-APIC-level   status=00000010 in-flight=0 domain-list=0:  9(-S--),
(XEN)    IRQ:  10 affinity:00000000,00000000,00000000,00000001 vec:70
type=IO-APIC-edge    status=00000002 mapped, unbound
(XEN)    IRQ:  11 affinity:00000000,00000000,00000000,00000001 vec:78
type=IO-APIC-edge    status=00000002 mapped, unbound
(XEN)    IRQ:  12 affinity:00000000,00000000,00000000,00000001 vec:27
type=IO-APIC-edge    status=00000050 in-flight=0 domain-list=0: 12(-S--),
(XEN)    IRQ:  13 affinity:00000000,00000000,00000000,0000000f vec:90
type=IO-APIC-edge    status=00000002 mapped, unbound
(XEN)    IRQ:  14 affinity:00000000,00000000,00000000,00000001 vec:98
type=IO-APIC-edge    status=00000002 mapped, unbound
(XEN)    IRQ:  15 affinity:00000000,00000000,00000000,00000001 vec:a0
type=IO-APIC-edge    status=00000002 mapped, unbound
(XEN)    IRQ:  16 affinity:00000000,00000000,00000000,00000001 vec:2f
type=IO-APIC-level   status=00000050 in-flight=0 domain-list=0: 16(-S--),
(XEN)    IRQ:  17 affinity:00000000,00000000,00000000,00000001 vec:3f
type=IO-APIC-level   status=00000050 in-flight=0 domain-list=0: 17(-S--),
(XEN)    IRQ:  18 affinity:00000000,00000000,00000000,00000008 vec:41
type=IO-APIC-level   status=00000002 mapped, unbound
(XEN)    IRQ:  19 affinity:00000000,00000000,00000000,0000000f vec:c8
type=IO-APIC-level   status=00000002 mapped, unbound
(XEN)    IRQ:  20 affinity:00000000,00000000,00000000,00000002 vec:b7
type=IO-APIC-level   status=00000050 in-flight=0 domain-list=0: 20(-S--),
(XEN)    IRQ:  22 affinity:00000000,00000000,00000000,0000000f vec:62
type=IO-APIC-level   status=00000002 mapped, unbound
(XEN)    IRQ:  23 affinity:00000000,00000000,00000000,0000000f vec:a8
type=IO-APIC-edge    status=00000002 mapped, unbound
(XEN)    IRQ:  24 affinity:00000000,00000000,00000000,00000001 vec:28
type=DMA_MSI         status=00000000 mapped, unbound
(XEN)    IRQ:  25 affinity:00000000,00000000,00000000,00000001 vec:30
type=DMA_MSI         status=00000000 mapped, unbound
(XEN)    IRQ:  26 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:6f
type=PCI-MSI         status=00000042 mapped, unbound
(XEN)    IRQ:  27 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:77
type=PCI-MSI         status=00000042 mapped, unbound
(XEN)    IRQ:  28 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:7f
type=PCI-MSI         status=00000042 mapped, unbound
(XEN)    IRQ:  29 affinity:ffffffff,ffffffff,ffffffff,ffffffff vec:87
type=PCI-MSI         status=00000042 mapped, unbound
(XEN)    IRQ:  31 affinity:00000000,00000000,00000000,00000002 vec:a6
type=PCI-MSI         status=00000002 mapped, unbound
(XEN)    IRQ:  32 affinity:00000000,00000000,00000000,00000001 vec:47
type=PCI-MSI         status=00000050 in-flight=0 domain-list=0:273(-S--),
(XEN)    IRQ:  33 affinity:00000000,00000000,00000000,00000002 vec:5f
type=PCI-MSI         status=00000010 in-flight=0 domain-list=0:272(PS--),
(XEN)    IRQ:  34 affinity:00000000,00000000,00000000,00000001 vec:67
type=PCI-MSI         status=00000010 in-flight=0 domain-list=0:271(-S--),
(XEN)    IRQ:  35 affinity:00000000,00000000,00000000,00000001 vec:4f
type=PCI-MSI         status=00000050 in-flight=0 domain-list=1: 55(-S--),
(XEN) IO-APIC interrupt information:
(XEN)     IRQ  0 Vec240:
(XEN)       Apic 0x00, Pin  2: vec=f0 delivery=LoPri dest=L status=0
polarity=0 irr=0 trig=E mask=0 dest_id:0
(XEN)     IRQ  1 Vec198:
(XEN)       Apic 0x00, Pin  1: vec=c6 delivery=LoPri dest=L status=0
polarity=0 irr=0 trig=E mask=0 dest_id:0
(XEN)     IRQ  3 Vec 64:
(XEN)       Apic 0x00, Pin  3: vec=40 delivery=LoPri dest=L status=0
polarity=0 irr=0 trig=E mask=0 dest_id:0
(XEN)     IRQ  4 Vec241:
(XEN)       Apic 0x00, Pin  4: vec=f1 delivery=LoPri dest=L status=0
polarity=0 irr=0 trig=E mask=0 dest_id:0
(XEN)     IRQ  5 Vec 72:
(XEN)       Apic 0x00, Pin  5: vec=48 delivery=LoPri dest=L status=0
polarity=0 irr=0 trig=E mask=0 dest_id:0
(XEN)     IRQ  6 Vec 80:
(XEN)       Apic 0x00, Pin  6: vec=50 delivery=LoPri dest=L status=0
polarity=0 irr=0 trig=E mask=0 dest_id:0
(XEN)     IRQ  7 Vec 88:
(XEN)       Apic 0x00, Pin  7: vec=58 delivery=LoPri dest=L status=0
polarity=0 irr=0 trig=E mask=0 dest_id:0
(XEN)     IRQ  8 Vec 96:
(XEN)       Apic 0x00, Pin  8: vec=60 delivery=LoPri dest=L status=0
polarity=0 irr=0 trig=E mask=0 dest_id:0
(XEN)     IRQ  9 Vec222:
(XEN)       Apic 0x00, Pin  9: vec=de delivery=LoPri dest=L status=0
polarity=0 irr=0 trig=L mask=0 dest_id:0
(XEN)     IRQ 10 Vec112:
(XEN)       Apic 0x00, Pin 10: vec=70 delivery=LoPri dest=L status=0
polarity=0 irr=0 trig=E mask=0 dest_id:0
(XEN)     IRQ 11 Vec120:
(XEN)       Apic 0x00, Pin 11: vec=78 delivery=LoPri dest=L status=0
polarity=0 irr=0 trig=E mask=0 dest_id:0
(XEN)     IRQ 12 Vec 39:
(XEN)       Apic 0x00, Pin 12: vec=27 delivery=LoPri dest=L status=0
polarity=0 irr=0 trig=E mask=0 dest_id:0
(XEN)     IRQ 13 Vec144:
(XEN)       Apic 0x00, Pin 13: vec=90 delivery=LoPri dest=L status=0
polarity=0 irr=0 trig=E mask=1 dest_id:0
(XEN)     IRQ 14 Vec152:
(XEN)       Apic 0x00, Pin 14: vec=98 delivery=LoPri dest=L status=0
polarity=0 irr=0 trig=E mask=0 dest_id:0
(XEN)     IRQ 15 Vec160:
(XEN)       Apic 0x00, Pin 15: vec=a0 delivery=LoPri dest=L status=0
polarity=0 irr=0 trig=E mask=0 dest_id:0
(XEN)     IRQ 16 Vec 47:
(XEN)       Apic 0x00, Pin 16: vec=2f delivery=LoPri dest=L status=0
polarity=1 irr=0 trig=L mask=0 dest_id:0
(XEN)     IRQ 17 Vec 63:
(XEN)       Apic 0x00, Pin 17: vec=3f delivery=LoPri dest=L status=0
polarity=1 irr=0 trig=L mask=0 dest_id:0
(XEN)     IRQ 18 Vec 65:
(XEN)       Apic 0x00, Pin 18: vec=41 delivery=LoPri dest=L status=0
polarity=1 irr=0 trig=L mask=1 dest_id:0
(XEN)     IRQ 19 Vec200:
(XEN)       Apic 0x00, Pin 19: vec=c8 delivery=LoPri dest=L status=0
polarity=1 irr=0 trig=L mask=1 dest_id:0
(XEN)     IRQ 20 Vec183:
(XEN)       Apic 0x00, Pin 20: vec=b7 delivery=LoPri dest=L status=0
polarity=1 irr=0 trig=L mask=0 dest_id:0
(XEN)     IRQ 22 Vec 98:
(XEN)       Apic 0x00, Pin 22: vec=62 delivery=LoPri dest=L status=0
polarity=1 irr=0 trig=L mask=1 dest_id:0
(XEN)     IRQ 23 Vec168:
(XEN)       Apic 0x00, Pin 23: vec=a8 delivery=LoPri dest=L status=0
polarity=0 irr=0 trig=E mask=1 dest_id:0
(XEN) Xen BUG at io_apic.c:554
(XEN) ----[ Xen-4.1.5-rc1  x86_64  debug=y  Tainted:    C ]----
(XEN) CPU:    0
(XEN) RIP:    e008:[<ffff82c48015e2d6>] 
smp_irq_move_cleanup_interrupt+0x211/0x289
(XEN) RFLAGS: 0000000000010092   CONTEXT: hypervisor
(XEN) rax: 0000000000000000   rbx: 00000000000000e9   rcx: 0000000000000000
(XEN) rdx: 0000000000000016   rsi: 000000000000000a   rdi: ffff82c4802592e0
(XEN) rbp: ffff82c48029fd08   rsp: ffff82c48029fcb8   r8:  0000000000000018
(XEN) r9:  0000000000000000   r10: 0000000000000000   r11: 0000000000000001
(XEN) r12: ffff830421080250   r13: ffff830421060534   r14: ffff82c48029ff18
(XEN) r15: ffff82c4802dd9e0   cr0: 000000008005003b   cr4: 00000000000026f0
(XEN) cr3: 0000000119a96000   cr2: ffff880402070198
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e010   cs: e008
(XEN) Xen stack trace from rsp=ffff82c48029fcb8:
(XEN)    0000000000000000 ffff82c48029ff18 ffff82c4802dd9e0 000000e900000000
(XEN)    ffff83042109ba04 ffff830421008000 0000000000000114 000000000000001d
(XEN)    0000000000000114 0000000000000000 00007d3b7fd602c7 ffff82c48014de60
(XEN)    0000000000000000 0000000000000114 000000000000001d 0000000000000114
(XEN)    ffff82c48029fdc8 ffff830421008000 0000000000000246 ffff82c48025c1f0
(XEN)    0000000000000003 0000001944602466 0000000000000000 0000000000000001
(XEN)    0000000000000000 0000000000000286 ffff830421060f34 0000002000000000
(XEN)    ffff82c4801226c0 000000000000e008 0000000000000286 ffff82c48029fdc8
(XEN)    000000000000e010 0000000000000286 ffff82c48029fe48 ffff82c480164446
(XEN)    ffff82c4802dd9e0 0000000000000286 ffff830421060f00 ffff830421060f34
(XEN)    ffff830421050ac0 000000000000001d 0000000000000246 ffff8301108fd140
(XEN)    ffff82c4801226d3 ffff82c48029fe78 000000000000001d ffff8803fa889af0
(XEN)    0000000000000114 ffff8804023be000 ffff82c48029fef8 ffff82c48017655b
(XEN)    ffff830114c7f300 ffffffff81381646 ffff82f600000008 ffff830421008000
(XEN)    0000000000000003 000000030000001d 00000000e2200000 0000000100a0fb00
(XEN)    0000000000007ff0 ffffffffffffffff 0000000000000003 0000000000000003
(XEN)    00000000e2200000 c390ed90d1ffffff 0000000000000202 ffff8300ca666000
(XEN)    ffff8803fc880240 0000000000000011 ffff8804023be858 ffff8804023be000
(XEN)    00007d3b7fd600c7 ffff82c480209f38 ffffffff8100142a 0000000000000021
(XEN)    ffff8804023be000 ffff8804023be858 0000000000000011 ffff8803fc880240
(XEN) Xen call trace:
(XEN)    [<ffff82c48015e2d6>] smp_irq_move_cleanup_interrupt+0x211/0x289
(XEN)    [<ffff82c48014de60>] irq_move_cleanup_interrupt+0x30/0x40
(XEN)    [<ffff82c4801226c0>] _spin_unlock_irqrestore+0x22/0x24
(XEN)    [<ffff82c480164446>] map_domain_pirq+0x37a/0x3df
(XEN)    [<ffff82c48017655b>] do_physdev_op+0xa2b/0x1508
(XEN)    [<ffff82c480209f38>] syscall_enter+0xc8/0x122


> 
> ~Andrew
> 


-- 
Best Regards / Pozdrawiam,
Marek Marczykowski
Invisible Things Lab

Attachment: signature.asc
Description: OpenPGP digital signature

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel

 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.