[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] Xen host crash

On Thu, Aug 29, 2013 at 10:32 PM, Andrew Cooper <andrew.cooper3@xxxxxxxxxx> wrote:
On 29/08/13 17:38, Rushikesh Jadhav wrote:
This is a spinlock deadlock, resulting in the NMI watchdog timing out and killing the host.  Do you have Stack and register dump for PCPU6 ?


Hi Andrew, here is the stack trace and register dump for PCPU6 & PCPU7

PCPU6 host state:
RIP:    e008:[<ffff828c8010e310>]
RFLAGS: 0000000000000002
rax: 0000000000000004   rbx: 0000000000000001   rcx: ffff828c803629cc
rdx: ffff828c8036286c   rsi: ffff828c803628dc   rdi: 00000000ffffffff
rbp: 0000000000000082   rsp: ffff83247fd88e10   r8:  0000000000000001
r9:  0000000000000001   r10: 00000000fffffffc   r11: 0000000000000001
r12: 0000000000000001   r13: ffff832270af39a0   r14: 0000000000000002
r15: 0000000000000009
cr0: 0000000080050033   cr4: 00000000000026f0
cr3: 000000205a12e000   cr2: fffff880005c5000
ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: e008

current: DOM185 VCPU6 (ffff830047ee8000)
stack context: DOM185 VCPU6 (ffff830047ee8000)
idle VCPU: ffff83007ea5e000

Stack at 0xffff83247fd88e10: 
 ffff83247fd88e00:                                     8010eeb7 ffff828c 801fb1d0 ffff828c
 ffff83247fd88e20: 80127b36 ffff828c 00000028 00000030 7fd88f18 ffff8324 7fd88e48 ffff8324
 ffff83247fd88e40: 00000001 00000000 00000002 00000000 00000002 00000000 801f423d ffff828c
 ffff83247fd88e60: 00000000 00000000 801f45b3 ffff828c 00000000 00000000 00000096 00000000
 ffff83247fd88e80: 8011b7da ffff828c 000000e5 00000000 00000000 00000000 8019b4ab ffff828c
 ffff83247fd88ea0: 7fd8ff20 ffff8324 80100000 ffff828c 8015eb75 ffff828c 7fd88f58 ffff8324
 ffff83247fd88ec0: 7fd8ff28 ffff8324 7fd88f58 ffff8324 00000002 00000000 7fd8ff28 ffff8324
 ffff83247fd88ee0: 7fd88f58 ffff8324 00000002 00000000 8015eeba ffff828c 7fd8ff28 ffff8324
 ffff83247fd88f00: 7fd8ff28 ffff8324 7fd88f58 ffff8324 801567a1 ffff828c 00000000 00000000
 ffff83247fd88f20: 00000006 00000000 7fd88f58 ffff8324 8015f37f ffff828c 00000000 00000000
 ffff83247fd88f40: 339e8000 ffff8322 7fd8ff28 ffff8324 801d6877 ffff828c 00000009 00000000
 ffff83247fd88f60: 00000002 00000000 70af39a0 ffff8322 00000001 00000000 7fd8ff28 ffff8324
 ffff83247fd88f80: 339e8000 ffff8322 00ff00ff 00ff00ff 0000ffff 0000ffff 339e8018 ffff8322
 ffff83247fd88fa0: 00000002 00000000 00000000 00000000 0000000f 00000000 47ee8000 ffff8300
 ffff83247fd88fc0: 00366807 00000000 60e72e90 ffff8319 00000000 00000002 8011ab02 ffff828c
 ffff83247fd88fe0: 0000e008 00000000 00000246 00000000 7fd8fd70 ffff8324 00000000 00000000

 da e8 df 91 01 00 e9 19 fd ff ff 90 90 90 90 90 90 90 90 90 90 <4c> 89 3f 4c 89 77 08 4c 89 6f 10 

Call Trace:
 [ffff828c8010e310] dump_domains+0x4d0
  ffff828c8010eeb7  kexec_crash+0x57
  ffff828c80127b36  panic+0x136
  ffff828c8011b7da  __print_symbol+0x8a
  ffff828c8019b4ab  vmx_asm_vmexit_handler+0x6b
  ffff828c80100000  __per_cpu_shift+0x800ffff4
  ffff828c8015eb75  show_stack+0x155
  ffff828c8015eeba  fatal_trap+0x6a
  ffff828c801567a1  nmi_watchdog_tick+0x131
  ffff828c8015f37f  do_nmi+0xbf
  ffff828c801d6877  handle_ist_exception+0x52
  ffff828c8011ab02  _spin_lock+0x12

  PCPU6 guest state:
RIP:    0000:[<fffff800016caee0>]
RFLAGS: 0000000000010206
rax: 0000000000000000   rbx: ffffffffffffffff   rcx: fffffa6002998000
rdx: 0000000000000100   rsi: fffff6fd30014d00   rdi: 0000000000000010
rbp: 0000000000000010   rsp: fffffa6001bc6c48   r8:  0000000000000000
r9:  0000000000000000   r10: 0000000000000000   r11: 0000000000000000
r12: 0000000000000000   r13: 0000000000000000   r14: 0000000000000000
r15: 0000000000000000
cr0: 0000000080050033   cr4: 00000000000026f0
cr3: 000000205a12e000   cr2: fffff880005c5000
ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: 0000

VCPU pause flags: 0 arch flags 0x1

current on PCPU6
struct vcpu at ffff830047ee8000

Stack unavailable.


The problematic spinlock is at address 0xffff831960e72e90, which is sadly a dynamically allocated one so cant be traced back to a symbol using the symbol table.

Thanks. How easy or hard it is to trace such thing ? I tried google for xen crash analyze and came up with https://github.com/xenserver/xen-crashdump-analyserhttp://xenbits.xen.org/people/andrewcoop/.

Having said that, you are using Xen 3.4 which is ages out of date, and in fact, probably using XenServer 5.6SP2 (so shouldn't be using xen-devel anyway).

It is quite an old yet stable host with trusted guests. I wanted more information about Xen stack traces and whats the best way to read them hence sent on the list.

Current crash did not generate a core dump, hence I tried crashdump analyzer from xenbits.xen.org on other core dumps but that fails with 

INFO  Elf CORE crash file: /tmp/core.kdump.1405
ERROR Unexpected class 1
ERROR Failed to parse the crash file

in xen-crashdump-analyser.log.

Thanks for your help.

I suggest you upgrade to something less ancient.

Host upgrades are in process for XenServer 6.2, I hope this one does not get ancient again by the time further XS7 comes in.


Xen-devel mailing list



Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.