[Top] [All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Xen-devel] Need help in debugging partially blocked hypervisor

  • To: xen-devel@xxxxxxxxxxxxxxxxxxx
  • From: Dietmar Hahn <dietmar.hahn@xxxxxxxxxxxxxx>
  • Date: Wed, 21 Oct 2009 15:07:05 +0200
  • Delivery-date: Wed, 21 Oct 2009 06:07:39 -0700
  • Domainkey-signature: s=s1536a; d=ts.fujitsu.com; c=nofws; q=dns; h=X-SBRSScore:X-IronPort-AV:Received:X-IronPort-AV: Received:Received:From:To:Subject:Date:User-Agent: MIME-Version:Content-Type:Content-Transfer-Encoding: Message-Id; b=PidfquaiArdkGCY4zR/9HEdLawfB3Te3eg8LRarsOnSyvqMpoopvKVQu KdgdVfOoH91RIgj+S4YXAT28UiQsOhCO6jfDcwOF7xbUOBq5oYh7sPy5P CKX7zgTjW1b47ExRLmrrobB/s5z8rCddrj1sYU2nUSQD48XkNusMFcUAF 9MAuZnmq+aYDEnpNrYhijqBZtUo42GYSVyTtLQEey9T0KTEc9ENf0ML4J uApjegQQ9Xh7jCcXAiSs0MDIbLQa9;
  • List-id: Xen developer discussion <xen-devel.lists.xensource.com>
Hi,

I need some help in debugging a strange hypervisor behavior together
with using fully virtualized performance counters.

For info I use SLES11, means xen-3.3.1 and linux-2.6.27.19-5... on a
Intel nehalem machine.
I tried the hypervisor from xen-unstable but the machine didn't boot.

dom0 1 cpu
domU 2 cpu's
3 cpu's paused.

I start performance counter in domU and after some time the domU cpus
are running forever  (seeing with xm vcpu-list) and the domU is not accessible.
dom0 is still working like expected.
Serial console doesn't react on 3xCTRL-A, but xm debug-keys prints it's output
on the serial console.
When I try to pause the domU (xm pause ...), using xenctx or some debug keys 
where
the domU must get paused, the dom0 freezes and only a hard reset helps, what
seems to come from the call of vcpu_sleep_sync().

I tried xentrace while in the strange state and saw only loggings from the CPU0
(dom0 cpu), what means for me that the domU CPU's are somewhere in the
hypervisor.

Attached is the output of "xm debug-keys d". I hope someone has an idea about 
the
direction where I have to look deeper.

Many thanks in advance!
Dietmar.


(XEN) 'd' pressed -> dumping registers
(XEN) *** Dumping CPU0 guest state (d0:v0): ***
(XEN) ----[ Xen-3.3.1  x86_64  debug=n  Tainted:    C ]----
(XEN) CPU:    0
(XEN) RIP:    e033:[<ffffffff8020746a>]
(XEN) RFLAGS: 0000000000000216   EM: 0   CONTEXT: pv guest
(XEN) rax: 0000000000000023   rbx: ffffffff803c7505   rcx: ffffffff8020746a
(XEN) rdx: 00007fd955ef2f8a   rsi: 00007fd95635dc00   rdi: 00007fd946ff9170
(XEN) rbp: ffffffffffffffda   rsp: ffff8800da541dc0   r8:  00007fd956324390
(XEN) r9:  0000000000000002   r10: 0000000000000000   r11: 0000000000000216
(XEN) r12: ffff8800dbd42080   r13: ffff8800db4d5500   r14: 0000000000000000
(XEN) r15: 00007fd946ff9200   cr0: 0000000080050033   cr4: 00000000000026b0
(XEN) cr3: 000000025c880000   cr2: 00007fef4f880ad0
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: e02b   cs: e033
(XEN) Guest stack trace from rsp=ffff8800da541dc0:
(XEN)    ffffffff80307263 ffffffff80207460 ffffffff803c7593 ffff8800ce4d7720
(XEN)    00000000da4fc067 ffff8800050a7180 ffffffff8028484c ffff8800ce1399c0
(XEN)    0000000000000003 0000000000000000 ffff8800ce1399c0 00007fd946ff9000
(XEN)    0000000000000000 0000000000000023 00007fd946ff9170 00007fd95635dc00
(XEN)    00007fd955ef2f8a 0000000000000000 00007fd956324390 ffff8800d9bc4780
(XEN)    0000000000000001 00007fd946ff9000 ffffffff803c7505 ffff8800dbd42100
(XEN)    ffff8800dbd42080 ffff8800db4d5500 0000000000000000 00007fd946ff9200
(XEN)    ffffffff802e0ae3 0030500046ff9000 ffff8800db4d5500 00007fd946ff9200
(XEN)    0000000000305000 0000000000000006 0000000000000006 00007fd956208608
(XEN)    ffffffff802aa8b5 ffff8800db4d5500 ffff8800db4d5500 00007fd946ff9200
(XEN)    ffffffff802aab22 0000000000001000 ffff8800dbde7520 00007fd946ff9000
(XEN)    0000000000000000 ffff8800db4d5500 00007fd946ff9200 0000000000305000
(XEN)    ffffffff802aab82 0000000000000006 0000000100000001 0000000000000000
(XEN)    0000000001ce0b34 0000000001c8eed0 0000000000000006 0000000000000001
(XEN)    ffffffff8020b3b8 0000000000000246 0000000000000000 0000000000000200
(XEN)    fffffffffffffffd 0000000000000010 ffffffff8020b350 00007fd946ff9200
(XEN)    0000000000305000 0000000000000006 0000000000000010 00007fd95536fb77
(XEN)    000000000000e033 0000000000000246 00007fd946ff9168 000000000000e02b
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) 
(XEN) *** Dumping CPU1 host state: ***
(XEN) ----[ Xen-3.3.1  x86_64  debug=n  Tainted:    C ]----
(XEN) CPU:    1
(XEN) RIP:    e008:[<ffff828c8013a24b>] default_idle+0x2b/0x40
(XEN) RFLAGS: 0000000000000246   CONTEXT: hypervisor
(XEN) rax: 0000000000000080   rbx: ffff8300bf5f7f28   rcx: 0000000000000001
(XEN) rdx: ffff828c80276980   rsi: ffff828c8021ad40   rdi: 0000000000002000
(XEN) rbp: ffff8300bf5f7f28   rsp: ffff8300bf5f7f08   r8:  0000000000000002
(XEN) r9:  ffff8300be601e00   r10: 0000000000000000   r11: ffff8300be601e10
(XEN) r12: ffff828c80276980   r13: 00000014ef213474   r14: ffff828c8021a160
(XEN) r15: ffff828c8021a100   cr0: 000000008005003b   cr4: 00000000000026b0
(XEN) cr3: 00000000be864000   cr2: 00007fd946ff3ed0
(XEN) ds: 002b   es: 002b   fs: 0000   gs: 0000   ss: e010   cs: e008
(XEN) Xen stack trace from rsp=ffff8300bf5f7f08:
(XEN)    ffff828c8013e126 0000000000002000 ffff8300be6fc080 ffff8300be61c080
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000246 0000000000007ff0
(XEN)    ffff880080ad1000 ffff8800dd488000 0000000000000000 ffffffff8020730a
(XEN)    0000000000000000 0000000000000001 0000000000000002 0000010000000000
(XEN)    ffffffff8020730a 000000000000e033 0000000000000246 ffff8800dd489f28
(XEN)    000000000000e02b 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000001 ffff8300be6fc080
(XEN) Xen call trace:
(XEN)    [<ffff828c8013a24b>] default_idle+0x2b/0x40
(XEN)    [<ffff828c8013e126>] idle_loop+0xa6/0xd0
(XEN)    
(XEN) No guest context (CPU1 is idle).
(XEN) 
(XEN) *** Dumping CPU2 host state: ***
(XEN) ----[ Xen-3.3.1  x86_64  debug=n  Tainted:    C ]----
(XEN) CPU:    2
(XEN) RIP:    e008:[<ffff828c8019a45c>] vmx_vmexit_handler+0x2ec/0x1b20
(XEN) RFLAGS: 0000000000000246   CONTEXT: hypervisor
(XEN) rax: 0000000000000020   rbx: ffff8300be6e0080   rcx: 0000000000000000
(XEN) rdx: ffff828c8021c3a0   rsi: 00000000000003de   rdi: ffff8300be6f7f28
(XEN) rbp: ffff9700ffb80990   rsp: ffff8300be6f7e38   r8:  ffff97600036379c
(XEN) r9:  ffff9700ff428b5b   r10: ffff976000363794   r11: 0000000000000000
(XEN) r12: 0000000000000000   r13: ffff8300be6e0080   r14: ffff8300be6f7f28
(XEN) r15: ffff976000363958   cr0: 000000008005003b   cr4: 00000000000026b0
(XEN) cr3: 000000033fc01000   cr2: 0000000000000000
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: e008
(XEN) Xen stack trace from rsp=ffff8300be6f7e38:
(XEN)    0000000000000000 ffff8300be6e0080 ffff8300be6e1858 ffff8300be6e0080
(XEN)    ffff9700ffb80990 ffff828c80187141 000000000000e102 000000000000e102
(XEN)    00000000000000e1 ffff828c8019483d ffff8300be6ee102 ffff828c80137d6e
(XEN)    ffff8300be6f7f28 ffff8300be6e0080 0000000000000000 ffff8300be601f08
(XEN)    00000078be6edeea 0000000000000002 ffff8300be6f7f28 ffff828c8011b87a
(XEN)    ffff828c80276980 0000000000000002 ffff828c80277980 ffff8300be6e0080
(XEN)    ffff9700ffb80990 0000000000000000 ffff976000363958 ffffffffffffffff
(XEN)    ffff976000363958 ffff828c801944c3 ffff976000363958 ffffffffffffffff
(XEN)    ffff976000363958 0000000000000000 ffff9700ffb80990 0000000000000050
(XEN)    0000000000000000 ffff976000363794 ffff9700ff428b5b ffff97600036379c
(XEN)    0000000000000730 ffffb000000b8000 00000000000003de 00000000000003de
(XEN)    ffff9700ffb80990 000000000000000b ffff9700ff025250 0000000000000000
(XEN)    0000000000010097 ffff976000363938 0000000000000000 5555555555555555
(XEN)    5555555555555555 5555555555555555 5555555555555555 5555555500000002
(XEN)    ffff8300be6e0080
(XEN) Xen call trace:
(XEN)    [<ffff828c8019a45c>] vmx_vmexit_handler+0x2ec/0x1b20
(XEN)    [<ffff828c80187141>] hvm_vcpu_has_pending_irq+0x41/0x60
(XEN)    [<ffff828c8019483d>] vmx_intr_assist+0x2bd/0x490
(XEN)    [<ffff828c80137d6e>] reprogram_timer+0x1e/0x90
(XEN)    [<ffff828c8011b87a>] _spin_unlock_irq+0x1a/0x40
(XEN)    [<ffff828c801944c3>] vmx_asm_do_vmentry+0x0/0xbd
(XEN)    
(XEN) *** Dumping CPU2 guest state (d1:v1): ***
(XEN) ----[ Xen-3.3.1  x86_64  debug=n  Tainted:    C ]----
(XEN) CPU:    2
(XEN) RIP:    0020:[<ffff9700ff025250>]
(XEN) RFLAGS: 0000000000010097   CONTEXT: hvm guest
(XEN) rax: 0000000000000730   rbx: 0000000000000050   rcx: ffffb000000b8000
(XEN) rdx: 00000000000003de   rsi: 00000000000003de   rdi: ffff9700ffb80990
(XEN) rbp: ffff9700ffb80990   rsp: ffff976000363938   r8:  ffff97600036379c
(XEN) r9:  ffff9700ff428b5b   r10: ffff976000363794   r11: 0000000000000000
(XEN) r12: 0000000000000000   r13: ffff976000363958   r14: ffffffffffffffff
(XEN) r15: ffff976000363958   cr0: 0000000080050033   cr4: 00000000000006b0
(XEN) cr3: 0000000001822000   cr2: 0000000000000000
(XEN) ds: 0028   es: 0028   fs: 0028   gs: 0028   ss: 0028   cs: 0020
(XEN) 
(XEN) *** Dumping CPU3 host state: ***
(XEN) ----[ Xen-3.3.1  x86_64  debug=n  Tainted:    C ]----
(XEN) CPU:    3
(XEN) RIP:    e008:[<ffff828c8019a45c>] vmx_vmexit_handler+0x2ec/0x1b20
(XEN) RFLAGS: 0000000000000202   CONTEXT: hypervisor
(XEN) rax: 0000000000000027   rbx: ffff8300be6e4080   rcx: 0000000000000007
(XEN) rdx: ffff828c8021e3a0   rsi: ffff9700fe1a9b70   rdi: ffff8300be91ff28
(XEN) rbp: ffff9700ffb80998   rsp: ffff8300be91fe38   r8:  0000000000000000
(XEN) r9:  ffff9700ff41e074   r10: 0000000000000000   r11: 0000000000000000
(XEN) r12: 0000000000000007   r13: ffff8300be6e4080   r14: ffff8300be91ff28
(XEN) r15: ffff9700ff01f9c0   cr0: 000000008005003b   cr4: 00000000000026b0
(XEN) cr3: 000000033fc26000   cr2: 0000000000000000
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: e008
(XEN) Xen stack trace from rsp=ffff8300be91fe38:
(XEN)    ffff828c8021e100 ffff8300be6e4080 ffff8300be6e5858 ffff8300be6e4080
(XEN)    ffff9700ffb80998 ffff828c80187141 000000000000e102 000000000000e102
(XEN)    00000000000000e1 ffff828c8019483d ffff8300be6ee102 ffff828c80137d6e
(XEN)    ffff8300be91ff28 ffff8300be6e4080 0000000000000000 ffff8300be852088
(XEN)    000001f9889c1558 0000000000000003 ffff8300be91ff28 ffff828c8011b87a
(XEN)    ffff828c80276980 0000000000000003 ffff828c80277980 ffff8300be6e4080
(XEN)    ffff9700ffb80998 ffff9700ff0476fc ffff9700ff047700 ffff9700fe000000
(XEN)    ffff9700ff01f9c0 ffff828c801944c3 ffff9700ff01f9c0 ffff9700fe000000
(XEN)    ffff9700ff047700 ffff9700ff0476fc ffff9700ffb80998 00000000c0010001
(XEN)    0000000000000000 0000000000000000 ffff9700ff41e074 0000000000000000
(XEN)    ffff9700ff02e59a 0000000000000043 0000000000000043 ffff9700fe1a9b70
(XEN)    ffff9700ffb80998 000000f100000001 ffff9700ff02e5c9 0000000000000000
(XEN)    0000000000000282 ffff9700fe1a9b60 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000003
(XEN)    ffff8300be6e4080
(XEN) Xen call trace:
(XEN)    [<ffff828c8019a45c>] vmx_vmexit_handler+0x2ec/0x1b20
(XEN)    [<ffff828c80187141>] hvm_vcpu_has_pending_irq+0x41/0x60
(XEN)    [<ffff828c8019483d>] vmx_intr_assist+0x2bd/0x490
(XEN)    [<ffff828c80137d6e>] reprogram_timer+0x1e/0x90
(XEN)    [<ffff828c8011b87a>] _spin_unlock_irq+0x1a/0x40
(XEN)    [<ffff828c801944c3>] vmx_asm_do_vmentry+0x0/0xbd
(XEN)    
(XEN) *** Dumping CPU3 guest state (d1:v0): ***
(XEN) ----[ Xen-3.3.1  x86_64  debug=n  Tainted:    C ]----
(XEN) CPU:    3
(XEN) RIP:    0020:[<ffff9700ff02e5c9>]
(XEN) RFLAGS: 0000000000000282   CONTEXT: hvm guest
(XEN) rax: ffff9700ff02e59a   rbx: 00000000c0010001   rcx: 0000000000000043
(XEN) rdx: 0000000000000043   rsi: ffff9700fe1a9b70   rdi: ffff9700ffb80998
(XEN) rbp: ffff9700ffb80998   rsp: ffff9700fe1a9b60   r8:  0000000000000000
(XEN) r9:  ffff9700ff41e074   r10: 0000000000000000   r11: 0000000000000000
(XEN) r12: ffff9700ff0476fc   r13: ffff9700ff047700   r14: ffff9700fe000000
(XEN) r15: ffff9700ff01f9c0   cr0: 0000000080050033   cr4: 00000000000006b0
(XEN) cr3: 0000000001423000   cr2: 0000000000000000
(XEN) ds: 0028   es: 0028   fs: 0028   gs: 0028   ss: 0028   cs: 0020
(XEN) 



-- 
Company details: http://ts.fujitsu.com/imprint.html

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel

 

©2013 Xen Project, A Linux Foundation Collaborative Project. All Rights Reserved.
Linux Foundation is a registered trademark of The Linux Foundation.
Xen Project is a trademark of The Linux Foundation.

Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.