[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [Xen-devel] Serial port GDB for Xen debugging
Hello,
I have had a bit of success in using GDB over the serial port to debug Xen itself. If anyone else is interested in debugging in this fashion, please read on.
I have managed to get the gdb stubs working for x86_64 which enable the user to break into Xen (via the '%' debug key from the Xen console), set/clear break points, and to resume execution without a Xen panic. There are still some stability issues, depending on where you set breakpoints. These will probably require someone with more x86 and Xen experience than myself.
In summary:
- fixed register reads/writes for x86_64. Different register numbering/format than x86_32.
- added SMP pause/resume.
- fixed serial port high-bit multiplexing to avoid sending carriage returns down the wrong pipe.
- misc. other changes such as watchdog enable/disable and do_int3 entry points.
I have tested these changes on xen-unstable/x86_64 using HVM. I have successfully booted up a WinXP guest and broke in and resumed with no errors.
Currently, when stopping execution and resuming, the dom0 linux kernel outputs the 'softlockup' back traces, but these have no permanent effect. I did look into the HVM parameters, specifically the 'delay_for_missed_ticks' option, but this seems to not help the passage of time in dom0.
I am using gdb and the Xen console through a single serial port. In order to get this working, you need to fix the serial port driver (patch below). Also, for serial port demuxing on the gdb host side, I could not get the nsplitd working, and didnt understand how it could be made to work.
So after googling for an hour or so and not finding a replacement or simple instructions, I created my own. Probably re-invented the wheel, but just in case someone else is in the same situation, I have attached the source code. It does not require any system configuration or mucking with /etc/xinit.d or /etc/services or anything else.
To compile:
% gcc -Wall ssplitd.c -o ssplitd
To use:
% ./ssplitd -t /dev/ttyS0 -b 115200 -l 4001 -h 4002
will setup on ttyS0 at baud rate 115200 and will listen on localhost sockets 4001 for the 'high-bit-unset' serial stream and on 4002 for the 'high-bit-set' serial stream.
For example, if you setup Xen as follows (from grub menu):
kernel /xen-3.2.gz console=com1H com1=115200,8n1 gdb=com1
The console will use the 'high-bit-set' stream and would be available on localhost port 4002
% ./tools/misc/xenconsole 127.0.0.1 4002
GDB would use the 'high-bit-unset' stream and would be available on localhost port 4001
% gdb xen-syms
(gdb) target remote 127.0.0.1:4001
If you are interested in debugging the gdb stubs, you can use the '-g X' option to ssplitd. This will cause it to output the gdb protocol strings. I found this a bit helpful when getting things working. You simply specify which serial port stream gdb is on. For the above example you would specify '-g 0' because gdb is using the 'high-bit-unset' serial stream.
The changes required are:
[root@localhost xen-unstable.hg]# hg status
M xen/arch/x86/gdbstub.c M xen/arch/x86/nmi.c M xen/arch/x86/setup.c M xen/arch/x86/traps.c M xen/arch/x86/x86_64/Makefile M xen/common/gdbstub.c M xen/common/keyhandler.c M xen/drivers/char/serial.c M xen/include/xen/gdbstub.h ? xen/arch/x86/x86_32/gdbstub.c ? xen/arch/x86/x86_64/gdbstub.c arch/x86/gdbstub.c
-- Remove the register read/write routines. These are specific to x86_32/x86_64 so moved them to a separate gdbstub.c file in the arch/x86/x86_32/64 directories.
-- Added SMP routines to pause/resume CPUs.
diff -r 966a6d3b7408 xen/arch/x86/gdbstub.c
--- a/xen/arch/x86/gdbstub.c Fri Dec 14 11:50:24 2007 +0000 +++ b/xen/arch/x86/gdbstub.c Mon Dec 17 11:02:36 2007 -0800 @@ -20,55 +20,26 @@ * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA */ #include <asm/debugger.h> +#include <xen/delay.h> + + +struct gdb_cpu_info +{ + atomic_t paused; + atomic_t ack; +}; + +static struct gdb_cpu_info gdb_cpu[NR_CPUS]; +static atomic_t gdb_smp_paused_count; + +static void gdb_smp_pause(void); +static void gdb_smp_resume(void); + u16 gdb_arch_signal_num(struct cpu_user_regs *regs, unsigned long cookie) { - /* XXX */ - return 1; -} - -void -gdb_arch_read_reg_array(struct cpu_user_regs *regs, struct gdb_context *ctx) -{ -#define GDB_REG(r) gdb_write_to_packet_hex(r, sizeof(r), ctx); - GDB_REG(regs->eax); - GDB_REG(regs->ecx); - GDB_REG(regs->edx); - GDB_REG(regs->ebx); - GDB_REG(regs->esp); - GDB_REG(regs->ebp); - GDB_REG(regs->esi); - GDB_REG(regs->edi); - GDB_REG(regs->eip); - GDB_REG(regs->eflags); -#undef GDB_REG -#define GDB_SEG_REG(s) gdb_write_to_packet_hex(s, sizeof(u32), ctx); - /* sizeof(segment) = 16bit */ - /* but gdb requires its return value as 32bit value */ - GDB_SEG_REG(regs->cs); - GDB_SEG_REG(regs->ss); - GDB_SEG_REG(regs->ds); - GDB_SEG_REG(regs->es); - GDB_SEG_REG(regs->fs); - GDB_SEG_REG(regs->gs); -#undef GDB_SEG_REG - gdb_send_packet(ctx); -} - -void -gdb_arch_write_reg_array(struct cpu_user_regs *regs, const char* buf, - struct gdb_context *ctx) -{ - /* XXX TODO */ - gdb_send_reply("E02", ctx); -} - -void -gdb_arch_read_reg(unsigned long regnum, struct cpu_user_regs *regs, - struct gdb_context *ctx) -{ - gdb_send_reply("", ctx); + return 5; /* TRAP signal. see include/gdb/signals.h */ } /* @@ -87,17 +58,6 @@ gdb_arch_copy_to_user(void *dest, const return __copy_to_user(dest, src, len); } -void -gdb_arch_resume(struct cpu_user_regs *regs, - unsigned long addr, unsigned long type, - struct gdb_context *ctx) -{ - /* XXX */ - if (type == GDB_STEP) { - gdb_send_reply("S01", ctx); - } -} - void gdb_arch_print_state(struct cpu_user_regs *regs) { @@ -107,13 +67,107 @@ void void gdb_arch_enter(struct cpu_user_regs *regs) { - /* nothing */ + /* + * We must pause all other CPUs. + */ + gdb_smp_pause(); } void gdb_arch_exit(struct cpu_user_regs *regs) { - /* nothing */ + /* + * Resume all CPUs. + */ + gdb_smp_resume(); +} + +static void gdb_pause_this_cpu(void *unused) +{ + unsigned long flags; + + local_irq_save(flags); + + atomic_set(&gdb_cpu[smp_processor_id()].ack, 1); + atomic_inc(&gdb_smp_paused_count); + + while (atomic_read(&gdb_cpu[smp_processor_id()].paused)) + mdelay(1); + + atomic_dec(&gdb_smp_paused_count); + atomic_set(&gdb_cpu[smp_processor_id()].ack, 0); + + /* Restore interrupts */ + local_irq_restore(flags); +} + +static void gdb_smp_pause(void) +{ + int timeout = 100; + int cpu; + + for_each_online_cpu(cpu) + { + atomic_set(&gdb_cpu[cpu].ack, 0); + atomic_set(&gdb_cpu[cpu].paused, 1); + } + + atomic_set(&gdb_smp_paused_count, 0); + + smp_call_function(gdb_pause_this_cpu, NULL, /* dont wait! */0, 0); + + /* Wait 100ms for all other CPUs to enter pause loop */ + while ( (atomic_read(&gdb_smp_paused_count) < (num_online_cpus() - 1)) + && (timeout-- > 0) ) + mdelay(1); + + if ( atomic_read(&gdb_smp_paused_count) < (num_online_cpus() - 1) ) + { + printk("GDB: Not all CPUs have paused, missing CPUs "); + for_each_online_cpu(cpu) + { + if ( cpu == smp_processor_id() ) + continue; + + if ( !atomic_read(&gdb_cpu[cpu].ack) ) + { + printk("%d ", cpu); + } + } + printk("\n"); + } +} + +static void gdb_smp_resume(void) +{ + int cpu; + int timeout = 100; + + for_each_online_cpu(cpu) + { + atomic_set(&gdb_cpu[cpu].paused, 0); + } + + /* Make sure all CPUs resume */ + while ( (atomic_read(&gdb_smp_paused_count) > 0) + && (timeout-- > 0) ) + mdelay(1); + + if ( atomic_read(&gdb_smp_paused_count) > 0 ) + { + printk("GDB: Not all CPUs have resumed execution, missing CPUs "); + for_each_online_cpu(cpu) + { + if ( cpu == smp_processor_id() ) + continue; + + if ( !atomic_read(&gdb_cpu[cpu].ack) ) + { + printk("%d ", cpu); + } + } + printk("\n"); + } } arch/x86/x86_64/gdbstub.c
-- new file to capture 64 bit specific gdb stubs.
/*
* x86_64 -specific gdb stub routines * * Copyright (C) 2007 Dan Doucette ddoucette@xxxxxxxxxxxx * * This program is free software; you can redistribute it and/or modify * it under the terms of the GNU General Public License as published by * the Free Software Foundation; either version 2 of the License, or * (at your option) any later version. * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * GNU General Public License for more details. * * You should have received a copy of the GNU General Public License * along with this program; if not, write to the Free Software * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA */ #include <asm/debugger.h> #define GDB_REG64(r) gdb_write_to_packet_hex(r, sizeof(u64), ctx)
#define GDB_REG32(r) gdb_write_to_packet_hex(r, sizeof(u32), ctx) void
gdb_arch_read_reg_array(struct cpu_user_regs *regs, struct gdb_context *ctx) { GDB_REG64(regs->rax); GDB_REG64(regs->rbx); GDB_REG64(regs->rcx); GDB_REG64(regs->rdx); GDB_REG64(regs->rsi); GDB_REG64(regs->rdi); GDB_REG64(regs->rbp); GDB_REG64(regs->rsp); GDB_REG64(regs->r8);
GDB_REG64(regs->r9); GDB_REG64(regs->r10); GDB_REG64(regs->r11); GDB_REG64(regs->r12); GDB_REG64(regs->r13); GDB_REG64(regs->r14); GDB_REG64(regs->r15); GDB_REG64(regs->rip);
GDB_REG32(regs->eflags); GDB_REG32(regs->cs);
GDB_REG32(regs->ss); GDB_REG32(regs->ds); GDB_REG32(regs->es); GDB_REG32(regs->fs); GDB_REG32(regs->gs); gdb_send_packet(ctx);
} void
gdb_arch_write_reg_array(struct cpu_user_regs *regs, const char* buf, struct gdb_context *ctx) { gdb_send_reply("", ctx); } void
gdb_arch_read_reg(unsigned long regnum, struct cpu_user_regs *regs, struct gdb_context *ctx) { switch (regnum) { case 0: GDB_REG64(regs->rax); break; case 1: GDB_REG64(regs->rbx); break; case 2: GDB_REG64(regs->rcx); break; case 3: GDB_REG64(regs->rdx); break; case 4: GDB_REG64(regs->rsi); break; case 5: GDB_REG64(regs->rdi); break; case 6: GDB_REG64(regs->rbp); break; case 7: GDB_REG64(regs->rsp); break; case 8: GDB_REG64(regs->r8); break;
case 9: GDB_REG64(regs->r9); break; case 10: GDB_REG64(regs->r10); break; case 11: GDB_REG64(regs->r11); break; case 12: GDB_REG64(regs->r12); break; case 13: GDB_REG64(regs->r13); break; case 14: GDB_REG64(regs->r14); break; case 15: GDB_REG64(regs->r15); break; case 16: GDB_REG64(regs->rip); break;
case 17: GDB_REG32(regs->rflags); break; case 18: GDB_REG32(regs->cs); break; case 19: GDB_REG32(regs->ss); break; case 20: GDB_REG32(regs->ds); break; case 21: GDB_REG32(regs->es); break; case 22: GDB_REG32(regs->fs); break; case 23: GDB_REG32(regs->gs); break; default: GDB_REG64(0xbaadf00ddeadbeef); break; } gdb_send_packet(ctx); } void
gdb_arch_write_reg(unsigned long regnum, unsigned long val, struct cpu_user_regs *regs, struct gdb_context *ctx) { switch (regnum) { case 0: regs->rax = val; break; case 1: regs->rbx = val; break; case 2: regs->rcx = val; break; case 3: regs->rdx = val; break; case 4: regs->rsi = val; break; case 5: regs->rdi = val; break; case 6: regs->rbp = val; break; case 7: regs->rsp = val; break; case 8: regs->r8 = val; break;
case 9: regs->r9 = val; break; case 10: regs->r10 = val; break; case 11: regs->r11 = val; break; case 12: regs->r12 = val; break; case 13: regs->r13 = val; break; case 14: regs->r14 = val; break; case 15: regs->r15 = val; break; case 16: regs->rip = val; break;
case 17: regs->rflags = (u32)val; break; case 18: regs->cs = (u16)val; break; case 19: regs->ss = (u16)val; break; case 20: regs->ds = (u16)val; break; case 21: regs->es = (u16)val; break; case 22: regs->fs = (u16)val; break; case 23: regs->gs = (u16)val; break; default: break; } gdb_send_reply("OK", ctx); } void
gdb_arch_resume(struct cpu_user_regs *regs, unsigned long addr, unsigned long type, struct gdb_context *ctx) { if ( addr != -1UL ) { regs->rip = addr; } regs->rflags &= ~X86_EFLAGS_TF;
/* * Set eflags.RF to ensure we do not re-enter. */ regs->rflags |= X86_EFLAGS_RF; /*
* Set the trap flag if we are single stepping. */ if ( type == GDB_STEP ) regs->rflags |= X86_EFLAGS_TF; } xen/arch/x86/x86_64/Makefile
-- add in the newly created gdbstub.o object
diff -r 966a6d3b7408 xen/arch/x86/x86_64/Makefile
--- a/xen/arch/x86/x86_64/Makefile Fri Dec 14 11:50:24 2007 +0000 +++ b/xen/arch/x86/x86_64/Makefile Mon Dec 17 11:14:28 2007 -0800 @@ -5,6 +5,7 @@ obj-y += mm.o obj-y += mm.o obj-y += traps.o +obj-$(crash_debug) += gdbstub.o obj-$(CONFIG_COMPAT) += compat.o obj-$(CONFIG_COMPAT) += compat_kexec.o obj-$(CONFIG_COMPAT) += domain.o xen/arch/x86/nmi.c
-- externed the opt_watchdog flag to prevent gdb stubs from enabling the watchdog, even though we have selected not to use it via watchdog=0 flag.
diff -r 966a6d3b7408 xen/arch/x86/nmi.c
--- a/xen/arch/x86/nmi.c Fri Dec 14 11:50:24 2007 +0000 +++ b/xen/arch/x86/nmi.c Mon Dec 17 11:10:23 2007 -0800 @@ -38,6 +38,8 @@ static unsigned int nmi_p4_cccr_val; static unsigned int nmi_p4_cccr_val; static DEFINE_PER_CPU(struct timer, nmi_timer); static DEFINE_PER_CPU(unsigned int, nmi_timer_ticks); + +extern int opt_watchdog; /* * lapic_nmi_owner tracks the ownership of the lapic NMI hardware: @@ -169,6 +171,9 @@ static void disable_lapic_nmi_watchdog(v static void enable_lapic_nmi_watchdog(void) { + if ( !opt_watchdog ) + return; + if (nmi_active < 0) { nmi_watchdog = NMI_LOCAL_APIC; setup_apic_nmi_watchdog(); @@ -351,13 +356,17 @@ static atomic_t watchdog_disable_count = void watchdog_disable(void) { - atomic_inc(&watchdog_disable_count); + if ( opt_watchdog ) + atomic_inc(&watchdog_disable_count); } void watchdog_enable(void) { static unsigned long heartbeat_initialised; unsigned int cpu; + + if ( !opt_watchdog ) + return; if ( !atomic_dec_and_test(&watchdog_disable_count) || test_and_set_bit(0, &heartbeat_initialised) ) xen/arch/x86/setup.c
-- extern the opt_watchdog flag for use in nmi.c
diff -r 966a6d3b7408 xen/arch/x86/setup.c
--- a/xen/arch/x86/setup.c Fri Dec 14 11:50:24 2007 +0000 +++ b/xen/arch/x86/setup.c Mon Dec 17 11:12:08 2007 -0800 @@ -72,7 +72,7 @@ integer_param("maxcpus", max_cpus); integer_param("maxcpus", max_cpus); /* opt_watchdog: If true, run a watchdog NMI on each processor. */ -static int opt_watchdog = 0; +int opt_watchdog = 0; boolean_param("watchdog", opt_watchdog); /* **** Linux config option: propagated to domain0. */ @@ -952,8 +952,7 @@ void __init __start_xen(unsigned long mb do_initcalls(); - if ( opt_watchdog ) - watchdog_enable(); + watchdog_enable(); /* Create initial domain 0. */ dom0 = domain_create(0, 0, DOM0_SSIDREF); xen/arch/x86/traps.c
-- added handing for gdb stubs in do_int3
-- changed handler from a system gate to an interrupt gate to ensure interrupts are immediately disabled.
diff -r 966a6d3b7408 xen/arch/x86/traps.c
--- a/xen/arch/x86/traps.c Fri Dec 14 11:50:24 2007 +0000 +++ b/xen/arch/x86/traps.c Mon Dec 17 11:13:00 2007 -0800 @@ -800,9 +800,8 @@ asmlinkage void do_int3(struct cpu_user_ if ( !guest_mode(regs) ) { - DEBUGGER_trap_fatal(TRAP_int3, regs); - show_execution_state(regs); - panic("FATAL TRAP: vector = 3 (Int3)\n"); + __trap_to_gdb(regs, TRAP_int3); + return; } do_guest_trap(TRAP_int3, regs, 0); @@ -2753,7 +2752,7 @@ void __init trap_init(void) set_intr_gate(TRAP_divide_error,÷_error); set_intr_gate(TRAP_debug,&debug); set_intr_gate(TRAP_nmi,&nmi); - set_system_gate(TRAP_int3,&int3); /* usable from all privileges */ + set_intr_gate(TRAP_int3,&int3); /* usable from all privileges */ set_system_gate(TRAP_overflow,&overflow); /* usable from all privileges */ set_intr_gate(TRAP_bounds,&bounds); set_intr_gate(TRAP_invalid_op,&invalid_op); xen/common/gdbstub.c
-- remove calls to smp_send_stop. gdb_arch_enter/exit now implicitly handle SMP pause/resume.
-- add register writes and continue/single step commands to allow them to work.
diff -r 966a6d3b7408 xen/common/gdbstub.c
--- a/xen/common/gdbstub.c Fri Dec 14 11:50:24 2007 +0000 +++ b/xen/common/gdbstub.c Mon Dec 17 11:15:55 2007 -0800 @@ -396,8 +396,9 @@ process_command(struct cpu_user_regs *re process_command(struct cpu_user_regs *regs, struct gdb_context *ctx) { const char *ptr; - unsigned long addr, length; + unsigned long addr, length, val; int resume = 0; + unsigned long type = GDB_CONTINUE; /* XXX check ctx->in_bytes >= 2 or similar. */ @@ -460,30 +461,40 @@ process_command(struct cpu_user_regs *re } gdb_arch_read_reg(addr, regs, ctx); break; + case 'P': /* write register */ + addr = simple_strtoul(ctx->in_buf + 1, &ptr, 16); + if ( ptr == (ctx->in_buf + 1) ) + { + gdb_send_reply("E03", ctx); + return 0; + } + if ( ptr[0] != '=' ) + { + gdb_send_reply("E04", ctx); + return 0; + } + ptr++; + val = str2ulong(ptr, sizeof(unsigned long)); + gdb_arch_write_reg(addr, val, regs, ctx); + break; case 'D': + case 'k': gdbstub_detach(ctx); gdb_send_reply("OK", ctx); - /* fall through */ - case 'k': ctx->connected = 0; - /* fall through */ + resume = 1; + break; case 's': /* Single step */ + type = GDB_STEP; case 'c': /* Resume at current address */ - { - unsigned long addr = ~((unsigned long)0); - unsigned long type = GDB_CONTINUE; - if ( ctx->in_buf[0] == 's' ) - type = GDB_STEP; - if ( ((ctx->in_buf[0] == 's') || (ctx->in_buf[0] == 'c')) && - ctx->in_buf[1] ) + addr = ~((unsigned long)0); + + if ( ctx->in_buf[1] ) addr = str2ulong(&ctx->in_buf[1], sizeof(unsigned long)); - if ( ctx->in_buf[0] != 'D' ) - gdbstub_attach(ctx); + gdbstub_attach(ctx); resume = 1; gdb_arch_resume(regs, addr, type, ctx); break; - } - default: gdb_send_reply("", ctx); break; @@ -555,10 +566,13 @@ __trap_to_gdb(struct cpu_user_regs *regs gdb_ctx->connected = 1; } - smp_send_stop(); - /* Try to make things a little more stable by disabling - interrupts while we're here. */ + /* + * We will enter this function through 2 mechanisms. + * 1) int3. Entered through intr_gate. + * 2) direct call of trap_to_gdb via the '%' debug key. + * we must disable interrupts here for case 2. + */ local_irq_save(flags); watchdog_disable(); xen/common/keyhandler.c
-- modified to simply call into gdb stub, rather than use trap fatal.
diff -r 966a6d3b7408 xen/common/keyhandler.c
--- a/xen/common/keyhandler.c Fri Dec 14 11:50:24 2007 +0000 +++ b/xen/common/keyhandler.c Mon Dec 17 11:18:21 2007 -0800 @@ -276,7 +276,7 @@ static void do_debug_key(unsigned char k static void do_debug_key(unsigned char key, struct cpu_user_regs *regs) { printk("'%c' pressed -> trapping into debugger\n", key); - (void)debugger_trap_fatal(0xf001, regs); + __trap_to_gdb(regs, TRAP_int3); nop(); /* Prevent the compiler doing tail call optimisation, as that confuses xendbg a bit. */ xen/drivers/char/serial.c
-- fixed 'cooked' mode insert of carriage returns to send them down the correct multiplexed character stream.
diff -r 966a6d3b7408 xen/drivers/char/serial.c
--- a/xen/drivers/char/serial.c Fri Dec 14 11:50:24 2007 +0000 +++ b/xen/drivers/char/serial.c Mon Dec 17 11:19:31 2007 -0800 @@ -126,6 +126,7 @@ void serial_putc(int handle, char c) { struct serial_port *port; unsigned long flags; + char cr; if ( handle == -1 ) return; @@ -137,7 +138,12 @@ void serial_putc(int handle, char c) spin_lock_irqsave(&port->tx_lock, flags); if ( (c == '\n') && (handle & SERHND_COOKED) ) - __serial_putc(port, '\r'); + { + cr = '\r'; + if ( handle & SERHND_HI ) + cr |= 0x80; + __serial_putc(port, cr); + } if ( handle & SERHND_HI ) c |= 0x80; @@ -153,7 +159,7 @@ void serial_puts(int handle, const char { struct serial_port *port; unsigned long flags; - char c; + char c, cr; if ( handle == -1 ) return; @@ -167,7 +173,12 @@ void serial_puts(int handle, const char while ( (c = *s++) != '\0' ) { if ( (c == '\n') && (handle & SERHND_COOKED) ) - __serial_putc(port, '\r'); + { + cr = '\r'; + if ( handle & SERHND_HI ) + cr |= 0x80; + __serial_putc(port, cr); + } if ( handle & SERHND_HI ) c |= 0x80; Attachment:
ssplitd.c _______________________________________________ Xen-devel mailing list Xen-devel@xxxxxxxxxxxxxxxxxxx http://lists.xensource.com/xen-devel
|
Lists.xenproject.org is hosted with RackSpace, monitoring our |