Re: [Xen-API] Debugging XAPI daemon crash

[David Scott <dave.scott@xxxxxxxxxxxxx>]

On 24/03/14 10:30, Ranjeet R wrote:
> Hello Dave
>
> The binaries did not have debug symbols but I managed to rebuild the binaries 
> with debug enabled.
Great.

> I tried starting the xapi process as it was started in the init.d scripts under 
> gdb. However, in gdb, the xapi process forks another process and I am not able 
> to debug it further (I tried setting detach_on_fork to off in gdb, but the 
> primary process just goes to end of execution).
>
> I am using the following gdb command to debug
>
> gdb --args /usr/sbin/xapi -daemon -writeinitcomplete 
> /var/run/xapi_init_complete.cookie -writereadyfile /var/run/xapi_startup.cookie 
> -onsystemboot"
>
> Can you please help me in the steps that you use in debugging the XAPI process.
Ah, I think xapi forks a "watchdog" process near the start -- this is 
probably what you're seeing.
Try adding a "-nowatchdog" option to the command-line.

Dave

> Thanks for your help,
>
> -Ranjeet
>
>
>
> -----Original Message-----
> From: Dave Scott [mailto:Dave.Scott@xxxxxxxxxx]
> Sent: Saturday, March 22, 2014 12:36 PM
> To: Ranjeet R
> Cc: xen-api@xxxxxxxxxxxxx
> Subject: Re: [Xen-API] Debugging XAPI daemon crash
>
> Hi,
>
> I suspect the segfault is being caused by a bad C function binding. I've seen a 
> similar crash before when querying an interface IP via getifaddrs (I think that 
> was the function name) Could you run xapi in gdb and reproduce the crash? 
> Printing the call stack would help to confirm this hypothesis. Provided the 
> xapi binary still has debug symbols (ie hasn't been stripped) the ocaml 
> functions (with fairly obvious mangled names) should also be on the stack too.
>
> Cheers,
> Dave
>
> > On Mar 22, 2014, at 3:47 AM, "Ranjeet R" <rranjeet@xxxxxxxxxxx> wrote:
> >
> > Hello all
> >
> > I am trying to bring a DevCloud setup which has an XCP Kronos based XAPI 
> > daemon. I had changed the underlying network implementation (it is not a 
> > bridge, but an openvswitch-like network implementation) and the XAPI daemon 
> > crashes during bootup. Please find the XAPI logs below.
> >
> >
> > starting up database engine D:72969b3eaf8e|redo_log] Flushing database
> > to all active redo-logs starting up database engine
> > D:72969b3eaf8e|xapi] About to flush database: /var/lib/xcp/state.db
> > starting up database engine D:72969b3eaf8e|redo_log] Flushing database
> > to all active redo-logs starting up database engine
> > D:72969b3eaf8e|xapi] Performing initial DB GC thread_zero|dbsync
> > (update_env) D:fd0aec7399c9|dbsync] Sync: sync_create_localhost dbsync
> > (update_env) D:fd0aec7399c9|dbsync] creating localhost
> >
> > dmesg logs seem to suggest that xapi is crashing during startup.
> >
> > [    9.092377] xapi[2813]: segfault at 0 ip 085bc286 sp bf80ae30 error 4 in 
> > xapi[8048000+59f000]
> > [    9.869971] xapi[2943]: segfault at 0 ip 085bc286 sp bf8ec450 error 4 in 
> > xapi[8048000+59f000]
> >
> > I looked the XAPI code to see where it fails and I don't see any logs
> > after the following code point in ocaml / xapi / dbsync_slave.ml
> >
> > let create_localhost ~__context info =
> >    let ip = get_my_ip_addr ~__context in
> >
> > I confirmed to see that "ifconfig xenbr0" has a valid management IP address and 
> > should not fail.
> >
> > How do I debug this crash further. Are there any ways to look at the stack 
> > trace where XAPI crashed. Any pointers to debug this further will be very 
> > helpful.
> >
> > -Ranjeet
> >
> >
> > _______________________________________________
> > Xen-api mailing list
> > Xen-api@xxxxxxxxxxxxx
> > http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api

_______________________________________________
Xen-api mailing list
Xen-api@xxxxxxxxxxxxx
http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api