[Xen-API] xapi will no longer start - what are my options?

[Lars Seeliger <meesterlars@xxxxxxxxx>]

Hi there

A scripted backup running on one of our XCP installations failed a few days ago, while exporting a snapshot. After the failure I tried to delete the snapshot in question, to no avail (something about the VDI being in use).

Anyway, I thought an xe-toolstack-restart would reset any lock on that file, allowing me to delete the no longer needed snapshot. Unfortunately, xapi is now unable to start.

I've just tried again, and this appears in /var/log/messages:

Jan 20 11:01:21 xcp-hoppe xapi: [ info|xcp-hoppe|0 thread_zero||watchdog] (Re)starting xapi... 
Jan 20 11:04:26 xcp-hoppe fe: 20120120T10:04:25.850Z||1172|About to bind to /var/xapi/forker/fd_e8f89481-9aae-05a5-1d73-fbb713f58ea3 

Jan 20 11:04:26 xcp-hoppe fe: 20120120T10:04:25.850Z||1172|bound, listening 
Jan 20 11:04:26 xcp-hoppe fe: 20120120T10:04:25.851Z||2300|Child here! 
Jan 20 11:04:26 xcp-hoppe fe: 20120120T10:04:25.851Z||2301|Grandchild here! 

Jan 20 11:04:26 xcp-hoppe fe: 20120120T10:04:25.851Z||2301|Started: state.cmdargs = [/sbin/service;xapissl;restart] 
Jan 20 11:04:26 xcp-hoppe fe: 20120120T10:04:25.851Z||2301|Started: state.env = [PATH=/sbin:/usr/sbin:/bin:/usr/bin] 

Jan 20 11:04:26 xcp-hoppe fe: 20120120T10:04:25.852Z||2301|Selecting in handle_comms_no_fd_sock2 
Jan 20 11:04:26 xcp-hoppe fe: 20120120T10:04:25.852Z||2301|Done 
Jan 20 11:04:26 xcp-hoppe fe: 20120120T10:04:25.852Z||2301|fd sock 

Jan 20 11:04:26 xcp-hoppe fe: 20120120T10:04:25.852Z||2301|Selecting in handle_comms_with_fd_sock2 
Jan 20 11:04:26 xcp-hoppe fe: 20120120T10:04:25.852Z||2301|Done 
Jan 20 11:04:26 xcp-hoppe fe: 20120120T10:04:25.852Z||2301|fd sock2 

Jan 20 11:04:26 xcp-hoppe fe: 20120120T10:04:25.852Z||2301|Received fd named: ed350b47-3eb6-63e0-38c5-3beaaefb65dd - duping to 1 (from 6) 
Jan 20 11:04:26 xcp-hoppe fe: 20120120T10:04:25.852Z||2301|Selecting in handle_comms_with_fd_sock2 

Jan 20 11:04:26 xcp-hoppe fe: 20120120T10:04:25.852Z||2301|Done 
Jan 20 11:04:26 xcp-hoppe fe: 20120120T10:04:25.852Z||2301|fd sock2 
Jan 20 11:04:26 xcp-hoppe fe: 20120120T10:04:25.852Z||2301|Received fd named: d810a903-961d-bb1e-aeb7-b39c98e5eefa - duping to 2 (from 6) 

Jan 20 11:04:26 xcp-hoppe fe: 20120120T10:04:25.852Z||2301|Selecting in handle_comms_with_fd_sock2 
Jan 20 11:04:26 xcp-hoppe fe: 20120120T10:04:25.852Z||2301|Done 
Jan 20 11:04:26 xcp-hoppe fe: 20120120T10:04:25.852Z||2301|comms sock 

Jan 20 11:04:26 xcp-hoppe fe: 20120120T10:04:25.852Z||2301|Exec 
Jan 20 11:04:26 xcp-hoppe fe: 20120120T10:04:25.852Z||2301|Finished... 
Jan 20 11:04:26 xcp-hoppe fe: 20120120T10:04:25.852Z||2301|Args after replacement = [/sbin/service;xapissl;restart] 

Jan 20 11:04:26 xcp-hoppe fe: 20120120T10:04:25.852Z||2301|I've received the following fds: [2;1]Â 
Jan 20 11:04:26 xcp-hoppe fe: 20120120T10:04:26.942Z||2301|Caught unexpected exception: Unix.Unix_error(31, "write", "") 

Jan 20 10:05:41 xcp-hoppe HVM5652[12007]:ÂÂÂ XENNET: WARNING: No handler for oid 0xffda0014. 
Jan 20 10:05:41 xcp-hoppe HVM5652[12007]:ÂÂÂ XENNET: WARNING: No handler for oid 0xffa0ffa1. 
Jan 20 10:05:41 xcp-hoppe HVM5652[12007]:ÂÂÂ XENNET: WARNING: Bad tcp task offload header? 

Jan 20 10:05:41 xcp-hoppe HVM5652[12007]:ÂÂÂ XENNET: WARNING: Bad tcp task offload header? 
Jan 20 11:06:16 xcp-hoppe snmpd[15173]: Received SNMP packet(s) from UDP: [192.168.1.1]:2227 
Jan 20 11:06:16 xcp-hoppe snmpd[15173]: Received SNMP packet(s) from UDP: [192.168.1.1]:2228 

Jan 20 11:06:17 xcp-hoppe snmpd[15173]: Received SNMP packet(s) from UDP: [192.168.1.1]:2229 
Jan 20 11:06:17 xcp-hoppe snmpd[15173]: Received SNMP packet(s) from UDP: [192.168.1.1]:2230 
Jan 20 11:06:25 xcp-hoppe python: PERFMON: caught socket.error: (111 Connection refused) - restarting XAPI session

Jan 20 10:08:36 xcp-hoppe HVM5652[12007]: Time offset set 3569, added offset -1 
Jan 20 11:08:54 xcp-hoppe python: PERFMON: Caught signal 15 - exiting
Jan 20 11:08:54 xcp-hoppe python: PERFMON: 11 Resource temporarily unavailable

Jan 20 11:08:54 xcp-hoppe python: PERFMON: Traceback (most recent call last): 
Jan 20 11:08:54 xcp-hoppe python: PERFMON:ÂÂ File "/opt/xensource/bin/perfmon", line 930, in ?ÂÂÂÂ rc = main() 
Jan 20 11:08:54 xcp-hoppe python: PERFMON:ÂÂ File "/opt/xensource/bin/perfmon", line 880, in mainÂÂÂÂ cmd = cmdsock.recv(cmdmaxlen) 

Jan 20 11:08:54 xcp-hoppe python: PERFMON: error: (11, 'Resource temporarily unavailable') 
Jan 20 11:08:54 xcp-hoppe python: PERFMON: caught socket.error: (111 Connection refused) - restarting XAPI session

Jan 20 11:08:55 xcp-hoppe v6d: [ info|xcp-hoppe|0||watchdog] (Re)starting v6d... 
Jan 20 11:08:55 xcp-hoppe xapi: [ info|xcp-hoppe|0 thread_zero||watchdog] (Re)starting xapi... 
Jan 20 10:10:45 xcp-hoppe HVM5641[28792]: Time offset set 3563, added offset -1 

Jan 20 11:13:45 xcp-hoppe python: PERFMON: caught socket.error: (111 Connection refused) - restarting XAPI session


/var/log/SMI contains:

[2598] 2012-01-20 11:08:55.277830ÂÂÂ VASSR run ['/opt/xensource/sm/VASSR', '<methodCall><methodName>sr_get_driver_info</methodName><params><param><value><struct><member><name>host_ref</name><value>OpaqueRef:NULL</value></member><member><name>command</name><value>sr_get_driver_info</value></member><member><name>args</name><value><array><data/></array></value></member><member><name>device_config</name><value><struct/></value></member></struct></value></param></params></methodCall>']

[2598] 2012-01-20 11:08:55.278332ÂÂÂ Warning: vdi_[de]activate present for vastsky
[2619] 2012-01-20 11:08:55.858537ÂÂÂ Warning: vdi_[de]activate present for dummy

Not sure there's anything of value in those logs...

I'm somewhat desperate, as I'm unable to perform any xe commands and am worried a reboot of the server will not magically bring xapi back online, meaning the VMs will not start. This XCP host is critical to one of our customer's infrastructure. It's Friday and I could possibly go there this evening and perform tasks necessary to bring everything back online, I just don't quite know what my options are, aside from reboot and perhaps an in-place install of XCP.

Does anyone have any bright ideas? I'm all ears!
_______________________________________________
xen-api mailing list
xen-api@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/mailman/listinfo/xen-api