[Xen-devel] Re: Unable to single step HVM guest

[Daniel J Mathew <danieljmathew@xxxxxxxxx>]

Some more info on this. These kdb commands were executed after another crash at the same breakpoint, and shows the kdb stack.

[1]xkdb> go
cmd not available in fatal/crashed state....
[1]xkdb> kdbdbg
kdbdbg set to:1
[1]xkdb> kdbf
trapimm:ccpu:1 reas:3
ccpu:1 trapdbg reas:3
(XEN) ----[ Xen-4.1-unstable  x86_64  debug=n  Not tainted ]----
(XEN) CPU:    1
(XEN) RIP:    e008:[<ffff82c4801fe9bf>] kdb_trap_immed+0x3f/0x80
(XEN) RFLAGS: 0000000000000202   CONTEXT: hypervisor
(XEN) rax: 0000000000000001   rbx: 0000000000000003   rcx: 0000000000000004
(XEN) rdx: 0000000000000000   rsi: 0000000000000082   rdi: ffff82c480249f4c
(XEN) rbp: 0000000000000092   rsp: ffff83007c4cfcc8   r8:  0000000000000000
(XEN) r9:  0000000000000001   r10: ffff83007c4cfbc8   r11: ffff82c4801371d0
(XEN) r12: ffff82c4802d65c0   r13: 0000000000000001   r14: ffff83007c4cfe28
(XEN) r15: ffff83007c4cfcf8   cr0: 000000008005003b   cr4: 00000000000426f0
(XEN) cr3: 000000005805c000   cr2: ffff82c49f7e7170
(XEN) ds: 002b   es: 002b   fs: 0000   gs: 0000   ss: e010   cs: e008
(XEN) Xen stack trace from rsp=ffff83007c4cfcc8:
(XEN)    0000000000000001 ffff82c4802d65c4 0000000000000001 ffff82c48020114c
(XEN)    ffff83007c4cfcf8 ffff82c48020640d ffff82c4802d65c0 00000000000000c6
(XEN)    ffff82c4801371d0 ffff83007c4cf9b8 0000000000000018 000000000000001c
(XEN)    000000008851cf8c 000000008838cb0e 000000000000016e 0000000000000000
(XEN)    000000000032bc48 000000f100000000 ffff82c48014b45c 000000000000e008
(XEN)    0000000000000206 ffff83007c4cfd80 0000000000000202 ffff82c4802d5d00
(XEN)    0000000000000001 ffff83007c4cfe28 0000000000000001 0000000000000002
(XEN)    000000674badefea ffff82c4801ff320 ffff82c4801bd73d 0000000000000000
(XEN)    ffff83007c4cfe28 ffff82c49f7e7170 ffff830058080000 ffff82c4801ff441
(XEN)    ffff83007c4cfe28 ffff82c48017c007 ffff8300107e8000 0000000000000001
(XEN)    ffff8300107e8000 ffff83007c4d6000 ffff83007ab60080 ffff82c4801f4628
(XEN)    000000674badefea ffff83007ab60080 ffff83007c4d6000 ffff8300107e8000
(XEN)    0000000000000001 ffff8300107e8000 ffff83007ab62558 ffff83007ab60180
(XEN)    ffff830058080448 ffff83007ab62530 00000000fa889380 ffff83007c4cff90
(XEN)    0000000000000000 0000000000000001 ffff8300107e97f0 0000000e00000002
(XEN)    ffff82c4801f473e 000000000000e008 0000000000010002 ffff83007c4cfed8
(XEN)    000000000000e010 000000fc00000000 ffff83007c4cff18 0000000000000000
(XEN)    ffff82c4801bbf91 000000000000e008 0000000000000286 ffff83007c4cff10
(XEN)    000000000000e010 ffff82c4801bbd9b 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN) Xen call trace:
(XEN)    [<ffff82c4801fe9bf>] kdb_trap_immed+0x3f/0x80
(XEN)    [<ffff82c48020114c>] kdb_cmdf_kdbf+0x1c/0x50
(XEN)    [<ffff82c48020640d>] kdb_do_cmds+0x15d/0x230
(XEN)    [<ffff82c4801371d0>] ns16550_putc+0x0/0x20
(XEN)    [<ffff82c48014b45c>] __udelay+0x2c/0x40
(XEN)    [<ffff82c4801ff320>] kdbmain_fatal+0xd0/0x1e0
(XEN)    [<ffff82c4801bd73d>] vmx_do_resume+0x12d/0x1e0
(XEN)    [<ffff82c4801ff441>] kdb_trap_fatal+0x11/0x20
(XEN)    [<ffff82c48017c007>] do_page_fault+0x437/0x470
(XEN)    [<ffff82c4801f4628>] handle_exception_saved+0x30/0x6e
(XEN)    [<ffff82c4801f473e>] int3+0x1e/0x40
(XEN)    [<ffff82c4801bbf91>] vmx_intr_assist+0x1/0x250
(XEN)    [<ffff82c4801bbd9b>] vmx_asm_do_vmentry+0x5/0xea

Please help me out with this issue. Is kdb actively supported and in use now?

On Mon, Feb 14, 2011 at 11:05 AM, Daniel J Mathew <danieljmathew@xxxxxxxxx> wrote:

Hi,

I forgot to mention that the following lines are printed to console after I hit 'go' and start the guest, before I get the error:
[421093.000014] Clocksource tsc unstable (delta = 30119955364 ns)
(XEN) tmem: all pools frozen for all domains
(XEN) tmem: all pools thawed for all domains
(XEN) tmem: all pools frozen for all domains
(XEN) tmem: all pools thawed for all domains

Does this have to do anything with the error?

Daniel.
--
Daniel J Mathew
Indian Institute of Technology Delhi

On Mon, Feb 14, 2011 at 10:57 AM, Daniel J Mathew <danieljmathew@xxxxxxxxx> wrote:

Hi,

I am trying to debug some HVM code I wrote for recording and replaying VM execution (on xen-unstable). For this, I set up kdb and a serial connection to another machine. However, most of the times when the breakpoint is hit, a fatal error occurs.

Here's what I'm doing:
bp vmx_intr_assist
go
[Started HVM guest from the other machine. The guest OS is a dummy OS called Pintos.]
*** kdb (Fatal Error on cpu:1 vec:14 Page Fault):
ffff82c4801f473e: int3+1e                        lock bts %rax, 0xe17b9(%rip)

Another example (with the call stack):
bp hvmemul_read_io
[1]xkdb> go
[Started HVM guest from the other machine.]
(XEN) read_ins_ring_dom0: cleared ring
(XEN) HVM1: HVM Loader
(XEN) setmode: Initialized ring
(XEN) HVM1: Detected Xen v4.1-unstable
(XEN) HVM1: CPU speed is 3325 MHz
(XEN) HVM1: Xenbus rings @0xfeffc000, event channel 2
(XEN) irq.c:243: Dom1 PCI link 0 changed 0 -> 5
(XEN) HVM1: PCI-ISA link 0 routed to IRQ5
(XEN) irq.c:243: Dom1 PCI link 1 changed 0 -> 10
(XEN) HVM1: PCI-ISA link 1 routed to IRQ10
(XEN) irq.c:243: Dom1 PCI link 2 changed 0 -> 11
(XEN) HVM1: PCI-ISA link 2 routed to IRQ11
(XEN) irq.c:243: Dom1 PCI link 3 changed 0 -> 5
(XEN) HVM1: PCI-ISA link 3 routed to IRQ5
*** kdb (Fatal Error on cpu:1 vec:14 Page Fault):
ffff82c4801f473e: int3+1e                        lock bts %rax, 0xe17b9(%rip)

[1]xkdb> f
(XEN) Xen call trace:
(XEN)    [<ffff82c4801f473e>] int3+0x1e/0x40
(XEN)    [<ffff82c4801a2171>] hvmemul_read_io+0x1/0x1f0
(XEN)    [<ffff82c480188ec5>] x86_emulate+0xb8e5/0x12bd0
(XEN)    [<ffff82c4801d9922>] sh_gva_to_gfn__guest_2+0x112/0x180
(XEN)    [<ffff82c4801a8000>] __hvm_copy+0x240/0x3b0
(XEN)    [<ffff82c480137900>] __serial_putc+0x50/0x190
(XEN)    [<ffff82c480149619>] smp_apic_timer_interrupt+0x49/0x80
(XEN)    [<ffff82c48011830f>] csched_vcpu_wake+0x12f/0x2c0
(XEN)    [<ffff82c48014e5ed>] vcpu_kick+0x1d/0x80
(XEN)    [<ffff82c480106065>] evtchn_set_pending+0x145/0x1d0
(XEN)    [<ffff82c4801d9922>] sh_gva_to_gfn__guest_2+0x112/0x180
(XEN)    [<ffff82c480106175>] notify_via_xen_event_channel+0x85/0xa0
(XEN)    [<ffff82c4801a6a10>] hvm_send_assist_req+0xa0/0x120
(XEN)    [<ffff82c4801a80b6>] __hvm_copy+0x2f6/0x3b0
(XEN)    [<ffff82c4801a16d9>] hvm_emulate_one+0xc9/0x1b0
(XEN)    [<ffff82c4801ac165>] handle_mmio+0x285/0x320
(XEN)    [<ffff82c480130001>] unshare_xenoprof_page_with_guest+0xc1/0x140
(XEN)    [<ffff82c48011d2b3>] vcpu_runstate_get+0x63/0xd0
(XEN)    [<ffff82c48011d340>] get_cpu_idle_time+0x20/0x30
(XEN)    [<ffff82c4801ac2c7>] hvm_io_assist+0xc7/0xd0
(XEN)    [<ffff82c4801a7075>] hvm_do_resume+0x185/0x1b0
(XEN)    [<ffff82c4801a6f21>] hvm_do_resume+0x31/0x1b0
(XEN)    [<ffff82c4801bd73d>] vmx_do_resume+0x12d/0x1e0
(XEN)    [<ffff82c48014f577>] context_switch+0x147/0xe40
(XEN)    [<ffff82c48014f577>] context_switch+0x147/0xe40
(XEN)    [<ffff82c480174668>] __update_vcpu_system_time+0x258/0x2e0
(XEN)    [<ffff82c48011e480>] schedule+0x230/0x570
(XEN)    [<ffff82c48014907a>] event_check_interrupt+0x2a/0x30
(XEN)    [<ffff82c48011f8af>] __do_softirq+0x6f/0xb0
(XEN)    [<ffff82c48015255d>] idle_loop+0x2d/0x60                                                                


The line where the error occurs is in arch/x86/x86_64/entry.S. I couldn't find any way to get Xen back to running after this error happens, so I usually end up doing a forced reboot.
Can someone please shed some light on what's happening? Is there anything I can do differently to get around this?

Thanks,
Daniel.
--
Daniel J Mathew
Indian Institute of Technology Delhi

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel