[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Xen-API] xapi will no longer start - what are my options?
Dave, the reboot was successful; the VMs will take a while to come online as an auto fsck is running. I reckon we're good.
Thanks again. Without your advice we'd have been far less sure about what we needed to do.
There should be a support subscription for us XCP users. We'd happily contribute a few hundred Euros a year for fast, competent support! Pass that on to your sales people :p
On 20 January 2012 18:30, Dave Scott <Dave.Scott@xxxxxxxxxxxxx> wrote:
Hi,
 The storage metadata should be fine (itâs only changed on VDI create/destroy etc, which isnât happening since xapi is offline). The storage data will also be fine if you shut the VMs down cleanly.
 Good luck!
 Dave
Â
 Killed both, but the xapi service stubbornly refuses to start. I guess I'll reboot. Arg.
My local storage shouldn't be affected by this, right? So long as I shut down each VM cleanly before rebooting the XCP host, all should be well?
Thanks a million for your help so far, Dave. It's much appreciated.
On 20 January 2012 18:19, Dave Scott <Dave.Scott@xxxxxxxxxxxxx> wrote: Hi,
 Before you reboot itâs worth manually killing the two non-zombie processes: (donât trust the init.d script to do this for you)
 Kill -9 Â8737
Kill -9 10346 Â
And then retrying âservice xapi startâ Â
If that still doesnât work, then reboot is probably your best option.
 Dave
Â
 So:
strace -p 10346
Process 10346 attached - interrupt to quit
select(0, [], [], [], {19456, 924155}
[root@xcp-hoppe ~]# strace -p 8737
Process 8737 attached - interrupt to quit
waitpid(10346,
This PID doesn't seem to be doing anything. Just, you know, waiting...
As per your instructions: - [root@xcp-hoppe ~]# service xapi stop
Stopping xapi: cannot stop xapi: xapi is not running.ÂÂÂÂÂ [FAILED] -
[root@xcp-hoppe ~]# ps -fe | grep xapi
rootÂÂÂÂÂ 7598ÂÂÂÂ 1Â 0Â 2011 ?ÂÂÂÂÂÂÂ 00:32:46 /opt/xensource/libexec/cdrommon /dev/xapi/cd/sr0
rootÂÂÂÂÂ 8737ÂÂÂÂ 1Â 0 17:07 ?ÂÂÂÂÂÂÂ 00:00:00 /opt/xensource/bin/xapi -daemon -writereadyfile /var/run/xapi_startup.cookie -writeinitcomplete /var/run/xapi_init_complete.cookie
rootÂÂÂÂ 10346Â 8737Â 0 17:27 ?ÂÂÂÂÂÂÂ 00:00:09 /opt/xensource/bin/xapi -daemon -writereadyfile /var/run/xapi_startup.cookie -writeinitcomplete /var/run/xapi_init_complete.cookie
rootÂÂÂÂ 10521ÂÂÂÂ 1Â 0 17:30 ?ÂÂÂÂÂÂÂ 00:00:00 /usr/bin/python /etc/xapi.d/plugins/vmpr <methodCall><methodName>protect</methodName><params><param><value>OpaqueRef:f0e3258d-be17-f1e8-b059-f50b7ae6eea9</value></param><param><value><struct/></value></param></params></methodCall>
rootÂÂÂÂ 10794ÂÂÂÂ 1Â 0 17:47 ?ÂÂÂÂÂÂÂ 00:00:00 /usr/sbin/stunnel /var/xapi/xapi-ssl.conf
rootÂÂÂÂ 12128Â 6638Â 0 18:02 pts/8ÂÂÂ 00:00:00 grep xapi
rootÂÂÂÂ 24767ÂÂÂÂ 1Â 1 Jan15 ?ÂÂÂÂÂÂÂ 01:28:38 [xapi] <defunct>
Still xapi processes running... still, I'll go on - [root@xcp-hoppe ~]# /opt/xensource/bin/xapi -nowatchdog
Fatal error: exception Unix.Unix_error(50, "bind", "")
I think we're going to have to reboot. I reckon the zombie process is blocking something...
On 20 January 2012 17:54, Dave Scott <Dave.Scott@xxxxxxxxxxxxx> wrote: Hm, ok, letâs leave that one alone. Itâs not a good sign but OTOH a zombie process uses no real resources and canât really do anything.
 I see you have some healthier-looking xapiâs here:
 #
rootÂÂÂÂÂ 8737Â 0.0Â 1.0Â 21508Â 6052 ?ÂÂÂÂÂÂÂ S<ÂÂ 17:07ÂÂ 0:00 /opt/xensource/bin/xapi -daemon -writereadyfile /var/run/xapi_startup.cookie -writeinitcompl
# root 10346 0.4 2.4 111896 13976 ? S<l 17:27 0:02 \_ /opt/xensource/bin/xapi -daemon -writereadyfile /var/run/xapi_startup.cookie âwriteinitc
 Are these generating any additional logs? Are they stuck? If you âstrace âp 10346â (the child)âdoes it seem busy? If you âservice xapi stopâ do they both go away (it might be that the zombie process could confuse the init script?).
Another diagnostic possibility is: 1.ÂÂÂÂÂÂ Run âservice xapi stopâ
2.ÂÂÂÂÂÂ Verify all (non-zombie) xapis are gone
3.ÂÂÂÂÂÂ Run â/opt/xensource/bin/xapi -nowatchdogâ on the terminal
Some types of error might appear on the terminal. Â
Although I canât see why itâs got stuck, I also canât see why a reboot wouldnât work (if that makes sense). Certainly the zombie process would go away. So would other runtime state that might be causing a hard-to-spot problem. ÂHowever I understand why you want to exhaust all other possibilities before doing that.
 Cheers,
Dave Â
From: Lars Seeliger [mailto:meesterlars@xxxxxxxxx]
Sent: 20 January 2012 16:39
To: Dave Scott
Subject: Re: [Xen-API] xapi will no longer start - what are my options?
 The parent of PID 24767 is PID 1, which is init. I don't believe I can kill that :p
Have a look here:
http://pastebin.com/V7fpLDik
Does it look like reboot time? I get the horrors... On 20 January 2012 17:33, Dave Scott <Dave.Scott@xxxxxxxxxxxxx> wrote:
Hi, Â
Youâre probably right about the zombie process. Its parent process must still be alive â if you kill that then the zombie should also go away. The parent process is probably another âxapiâ (since xapi forks a subprocess and leaves the parent around as a âwatchdogâ. The watchdog may be stuck, ironically).
 So I would
1.ÂÂÂÂÂÂ Kill whatever the xapi parent process is (it should be a xapi)
2.ÂÂÂÂÂÂ Verify that all xapis have gone
3.ÂÂÂÂÂÂ âservice xapissl startâ
4.ÂÂÂÂÂÂ Verify that stunnel is running
5.ÂÂÂÂÂÂ âservice xapi startâ
 Â
Cheers, Dave
Â
 Here you go:
[root@xcp-hoppe ~]# /sbin/service xapissl restart
Stopping xapi SSL:
Starting xapi SSL:ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ [Â OKÂ ]
[root@xcp-hoppe ~]# echo $?
0
[root@xcp-hoppe ~]# /sbin/service xapissl stop
Stopping xapi SSL: .
[root@xcp-hoppe ~]# ps aux | grep stunnel
rootÂÂÂÂÂ 9745Â 0.0Â 0.1ÂÂ 5100ÂÂ 716 pts/8ÂÂÂ S+ÂÂ 17:24ÂÂ 0:00 grep stunnel
So nothing to kill there.
[root@xcp-hoppe ~]# ps axf | grep xapi
Â7598 ?ÂÂÂÂÂÂÂ SÂÂÂÂ 32:46 /opt/xensource/libexec/
cdrommon /dev/xapi/cd/sr0
Â6840 pts/8ÂÂÂ S+ÂÂÂÂ 0:00Â |ÂÂÂÂÂÂ \_ grep xapi
24767 ?ÂÂÂÂÂÂÂ Z<lÂÂ 88:38 [xapi] <defunct> Â
There's a zombie process there I can't kill. I have a hunch this is the problem. Â [root@xcp-hoppe ~]# ps -fe | grep xapi
rootÂÂÂÂÂ 7598ÂÂÂÂ 1Â 0Â 2011 ?ÂÂÂÂÂÂÂ 00:32:46 /opt/xensource/libexec/cdrommon /dev/xapi/cd/sr0
rootÂÂÂÂÂ 8401ÂÂÂÂ 1Â 0 16:52 ?ÂÂÂÂÂÂÂ 00:00:00 /usr/sbin/stunnel /var/xapi/xapi-ssl.conf
rootÂÂÂÂÂ 8737ÂÂÂÂ 1Â 0 17:07 ?ÂÂÂÂÂÂÂ 00:00:00 /opt/xensource/bin/xapi -daemon -writereadyfile /var/run/xapi_startup.cookie -writeinitcomplete /var/run/xapi_init_complete.cookie
rootÂÂÂÂÂ 8739Â 8737Â 0 17:07 ?ÂÂÂÂÂÂÂ 00:00:02 /opt/xensource/bin/xapi -daemon -writereadyfile /var/run/xapi_startup.cookie -writeinitcomplete /var/run/xapi_init_complete.cookie
rootÂÂÂÂÂ 9480ÂÂÂÂ 1Â 0 17:15 ?ÂÂÂÂÂÂÂ 00:00:00 /usr/bin/python /etc/xapi.d/plugins/vmpr <methodCall><methodName>protect</methodName><params><param><value>OpaqueRef:5212aed4-a93b-3012-3988-e4aad29c5bbd</value></param><param><value><struct/></value></param></params></methodCall>
rootÂÂÂÂÂ 9614Â 6638Â 0 17:21 pts/8ÂÂÂ 00:00:00 grep xapi
rootÂÂÂÂ 24767ÂÂÂÂ 1Â 1 Jan15 ?ÂÂÂÂÂÂÂ 01:28:38 [xapi] <defunct> Â
 On 20 January 2012 17:21, Dave Scott <Dave.Scott@xxxxxxxxxxxxx> wrote:
Hi, Â
Thanks for the updated pastebin. The last line says: Â
[20120120T14:48:18.726Z|debug|xcp-hoppe|0 thread_zero|bringing up management interface D:c515ea9445f2|xapi] Restarting stunnel
 Which correlates with some of the original /var/log/messages lines you posted
 Jan 20 11:04:26 xcp-hoppe fe: 20120120T10:04:25.852Z||2301|Args after replacement = [/sbin/service;xapissl;restart]
Jan 20 11:04:26 xcp-hoppe fe: 20120120T10:04:25.852Z||2301|I've received the following fds: [2;1]Â
Jan 20 11:04:26 xcp-hoppe fe: 20120120T10:04:26.942Z||2301|Caught unexpected exception: Unix.Unix_error(31, "write", "")
 I suggest a quick experimentâ if you run:
  /sbin/service xapissl restart
 echo $? Â
Does it succeed and print â0â? Â
Another quick experiment would be: Â
 /sbin/service xapissl stop Â
Then check to see if stunnel really has gone away. If not (it might be stuck somehow) Iâd recommend shutting it down with âkill -9 <pid>â. Then
  /sbin/service xapissl start
 After that I would try
 /sbin/service xapi stop
 Then make sure that all xapis really have gone away. If not (they might also be stuck somehow) Iâd recommend shutting them down with âkill -9 <pid>â. Then
  /sbin/service xapi start
 Cheers,
Dave Â
Â
Here you go:
[root@xcp-hoppe ~]# ps axf | grep xapi
Â7598 ?ÂÂÂÂÂÂÂ SÂÂÂÂ 32:46 /opt/xensource/libexec/cdrommon /dev/xapi/cd/sr0
Â6840 pts/8ÂÂÂ S+ÂÂÂÂ 0:00Â |ÂÂÂÂÂÂ \_ grep xapi
24767 ?ÂÂÂÂÂÂÂ Z<lÂÂ 88:38 [xapi] <defunct>Â <<<<<<<<<<<< defunct??????
Â6757 ?ÂÂÂÂÂÂÂ S<sÂÂÂ 0:00 /usr/sbin/stunnel /var/xapi/xapi-ssl.conf
[root@xcp-hoppe ~]# ps axf | grep stunnel
Â6677 pts/8ÂÂÂ S+ÂÂÂÂ 0:00ÂÂÂÂÂÂÂÂÂ \_ grep stunnel
Â6397 ?ÂÂÂÂÂÂÂ S<sÂÂÂ 0:00 /usr/sbin/stunnel /var/xapi/xapi-ssl.conf
There is something listening on port 443, though my skills don't allow me to ascertain what. Same goes for 80:
[root@xcp-hoppe ~]# netstat -an
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local AddressÂÂÂÂÂÂÂÂÂÂÂÂÂÂ Foreign AddressÂÂÂÂÂÂÂÂÂÂÂÂ State
tcpÂÂÂÂÂÂÂ 0ÂÂÂÂÂ 0 127.0.0.1:199ÂÂÂÂÂÂÂÂÂÂÂÂÂÂ 0.0.0.0:*ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ LISTEN
tcpÂÂÂÂÂÂÂ 0ÂÂÂÂÂ 0 127.0.0.1:5900ÂÂÂÂÂÂÂÂÂÂÂÂÂ 0.0.0.0:*ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ LISTEN
tcpÂÂÂÂÂÂÂ 0ÂÂÂÂÂ 0 127.0.0.1:5901ÂÂÂÂÂÂÂÂÂÂÂÂÂ 0.0.0.0:*ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ LISTEN
tcpÂÂÂÂÂÂÂ 0ÂÂÂÂÂ 0 127.0.0.1:5902ÂÂÂÂÂÂÂÂÂÂÂÂÂ 0.0.0.0:*ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ LISTEN
tcpÂÂÂÂÂÂÂ 0ÂÂÂÂÂ 0 127.0.0.1:5903ÂÂÂÂÂÂÂÂÂÂÂÂÂ 0.0.0.0:*ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ LISTEN
tcpÂÂÂÂÂÂÂ 0ÂÂÂÂÂ 0 127.0.0.1:111ÂÂÂÂÂÂÂÂÂÂÂÂÂÂ 0.0.0.0:*ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ LISTEN
tcpÂÂÂÂÂÂÂ 0ÂÂÂÂÂ 0 0.0.0.0:22ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ 0.0.0.0:*ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ LISTEN
tcpÂÂÂÂÂÂÂ 0ÂÂÂÂÂ 0 0.0.0.0:600ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ 0.0.0.0:*ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ LISTEN
tcpÂÂÂÂÂÂÂ 0ÂÂÂÂÂ 0 192.168.1.10:443ÂÂÂÂÂÂÂÂÂÂÂ 0.0.0.0:*ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ LISTEN
tcpÂÂÂÂÂÂÂ 0ÂÂÂÂÂ 0 127.0.0.1:443ÂÂÂÂÂÂÂÂÂÂÂÂÂÂ 0.0.0.0:*ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ LISTEN
tcpÂÂÂÂÂÂÂ 0ÂÂÂÂÂ 0 0.0.0.0:3551ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ 0.0.0.0:*ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ LISTEN
tcpÂÂÂÂÂÂÂ 0ÂÂÂÂÂ 0 192.168.1.10:22ÂÂÂÂÂÂÂÂÂÂÂÂ 192.168.1.1:4930ÂÂÂÂÂÂÂÂÂÂÂ ESTABLISHED
tcpÂÂÂÂÂÂÂ 0ÂÂÂÂÂ 0 192.168.1.10:51388ÂÂÂÂÂÂÂÂÂ 192.168.1.1:445ÂÂÂÂÂÂÂÂÂÂÂÂ ESTABLISHED
tcpÂÂÂÂÂÂÂ 0ÂÂÂÂÂ 0 192.168.1.10:22ÂÂÂÂÂÂÂÂÂÂÂÂ 10.168.1.9:61784ÂÂÂÂÂÂÂÂÂÂÂ ESTABLISHED
tcpÂÂÂÂÂÂÂ 1ÂÂÂÂÂ 0 127.0.0.1:80ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ 127.0.0.1:48318ÂÂÂÂÂÂÂÂÂÂÂÂ CLOSE_WAIT
tcpÂÂÂÂÂÂÂ 0ÂÂÂÂ 52 192.168.1.10:22ÂÂÂÂÂÂÂÂÂÂÂÂ 192.168.1.1:1093ÂÂÂÂÂÂÂÂÂÂÂ ESTABLISHED
udpÂÂÂÂÂÂÂ 0ÂÂÂÂÂ 0 0.0.0.0:58882ÂÂÂÂÂÂÂÂÂÂÂÂÂÂ 0.0.0.0:*
udpÂÂÂÂÂÂÂ 0ÂÂÂÂÂ 0 0.0.0.0:161ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ 0.0.0.0:*
udpÂÂÂÂÂÂÂ 0ÂÂÂÂÂ 0 0.0.0.0:600ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ 0.0.0.0:*
udpÂÂÂÂÂÂÂ 0ÂÂÂÂÂ 0 0.0.0.0:601ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ 0.0.0.0:*
udpÂÂÂÂÂÂÂ 0ÂÂÂÂÂ 0 127.0.0.1:111ÂÂÂÂÂÂÂÂÂÂÂÂÂÂ 0.0.0.0:*
udpÂÂÂÂÂÂÂ 0ÂÂÂÂÂ 0 192.168.1.10:123ÂÂÂÂÂÂÂÂÂÂÂ 0.0.0.0:*
udpÂÂÂÂÂÂÂ 0ÂÂÂÂÂ 0 127.0.0.1:123ÂÂÂÂÂÂÂÂÂÂÂÂÂÂ 0.0.0.0:*
udpÂÂÂÂÂÂÂ 0ÂÂÂÂÂ 0 0.0.0.0:123ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ 0.0.0.0:*
netstat -an | grep stunnel returns nothing.
[root@xcp-hoppe ~]# netstat -an | grep xapi
unix 2 [ ACC ] STREAM LISTENING 708872842 /var/run/openvswitch/xapi1.snoop
unix 2 [ ACC ] STREAM LISTENING 708872844 /var/run/openvswitch/xapi1.mgmt
unix 2 [ ACC ] STREAM LISTENING 834141817 /var/xapi/forker/main
unix 2 [ ACC ] STREAM LISTENING 834141947 /var/xapi/v6
unix 2 [ ] STREAM CONNECTED 831867057 /var/xapi/xapi
unix 2 [ ] STREAM CONNECTED 831832667 /var/xapi/xapi
unix 2 [ ] STREAM CONNECTED 831831438 /var/xapi/xapi
unix 2 [ ] STREAM CONNECTED 831825575 /var/xapi/xapi
unix 2 [ ] STREAM CONNECTED 827814254 /var/xapi/forker/main
unix 2 [ ] STREAM CONNECTED 806731261 /var/xapi/forker/main
unix 2 [ ] STREAM CONNECTED 797242460 /var/xapi/forker/main
New pastebin is up, with 1749 lines. Enjoy :p
http://pastebin.com/gHTeNqXb
Thanks! On 20 January 2012 16:21, Dave Scott <Dave.Scott@xxxxxxxxxxxxx> wrote:
Hi, Â
Thanks for the info. Â
Could you check whether stunnel is running in âps axfâ? Also check in ânetstat âanâ to see whether something is listening on port 443 (should be stunnel) and port 80 (should be xapi).
 The pastebin text stops at line 310, which looks inoccuous. Is there anything after that?
 Cheers,
Dave Â
 Â
Â
It's a full install from an XCP CD.
# service xapissl status
xapissl (pid 2323) is running...
I performed a restart of xapissl anyway, and xe-toolstack-restart still fails starting the xapi service. Any other ideas? What would happen if I were to restart the system or perform a reinstall?
Our customer and us are pretty nervous now, as they have not had a successful backup since last week Friday. So what else can I try?
Thanks, Dave! On 20 January 2012 15:41, Dave Scott <Dave.Scott@xxxxxxxxxxxxx> wrote:
Hi, Â
I should have asked earlier: is this a host installed via the XCP CD, or is this a Debian system running the xcp- packages?
 It looks like xapi canât find a running stunnel, and it looks like the xe-toolstack-restart failed to run â/sbin/service xapissl restartâ. Do you have stunnel running, and listening on port 443? On a system installed via the XCP CD, âservice xapissl restartâ should start stunnel. On a Debian system â I believe â the xapi init.d script itself starts stunnel.
 Dave
Â
 Hey, Dave
Thanks for the prompt response. For fear of not including enough info, I've pasted the entire xensource.log contents here:
http://pastebin.com/AW12gfM0
If you need anything else, just shout; this problem has ruined my day! :p On 20 January 2012 12:53, Dave Scott <Dave.Scott@xxxxxxxxxxxxx> wrote:
Hi Lars, Â
Have a look in the main xapi logs (in the confusingly-named file /var/log/xensource.log). Start from the bottom and reverse-search to the string âXAPI SERVER STARTINGâ. The lines after that will show how far the startup sequence got.
 Cheers,
Dave Â
 Â
 Hi there
A scripted backup running on one of our XCP installations failed a few days ago, while exporting a snapshot. After the failure I tried to delete the snapshot in question, to no avail (something about the VDI being in use).
Anyway, I thought an xe-toolstack-restart would reset any lock on that file, allowing me to delete the no longer needed snapshot. Unfortunately, xapi is now unable to start.
I've just tried again, and this appears in /var/log/messages:
Jan 20 11:01:21 xcp-hoppe xapi: [ info|xcp-hoppe|0 thread_zero||watchdog] (Re)starting xapi...
Jan 20 11:04:26 xcp-hoppe fe: 20120120T10:04:25.850Z||1172|About to bind to /var/xapi/forker/fd_e8f89481-9aae-05a5-1d73-fbb713f58ea3
Jan 20 11:04:26 xcp-hoppe fe: 20120120T10:04:25.850Z||1172|bound, listening
Jan 20 11:04:26 xcp-hoppe fe: 20120120T10:04:25.851Z||2300|Child here!
Jan 20 11:04:26 xcp-hoppe fe: 20120120T10:04:25.851Z||2301|Grandchild here!
Jan 20 11:04:26 xcp-hoppe fe: 20120120T10:04:25.851Z||2301|Started: state.cmdargs = [/sbin/service;xapissl;restart]
Jan 20 11:04:26 xcp-hoppe fe: 20120120T10:04:25.851Z||2301|Started: state.env = [PATH=/sbin:/usr/sbin:/bin:/usr/bin]
Jan 20 11:04:26 xcp-hoppe fe: 20120120T10:04:25.852Z||2301|Selecting in handle_comms_no_fd_sock2
Jan 20 11:04:26 xcp-hoppe fe: 20120120T10:04:25.852Z||2301|Done
Jan 20 11:04:26 xcp-hoppe fe: 20120120T10:04:25.852Z||2301|fd sock
Jan 20 11:04:26 xcp-hoppe fe: 20120120T10:04:25.852Z||2301|Selecting in handle_comms_with_fd_sock2
Jan 20 11:04:26 xcp-hoppe fe: 20120120T10:04:25.852Z||2301|Done
Jan 20 11:04:26 xcp-hoppe fe: 20120120T10:04:25.852Z||2301|fd sock2
Jan 20 11:04:26 xcp-hoppe fe: 20120120T10:04:25.852Z||2301|Received fd named: ed350b47-3eb6-63e0-38c5-3beaaefb65dd - duping to 1 (from 6)
Jan 20 11:04:26 xcp-hoppe fe: 20120120T10:04:25.852Z||2301|Selecting in handle_comms_with_fd_sock2
Jan 20 11:04:26 xcp-hoppe fe: 20120120T10:04:25.852Z||2301|Done
Jan 20 11:04:26 xcp-hoppe fe: 20120120T10:04:25.852Z||2301|fd sock2
Jan 20 11:04:26 xcp-hoppe fe: 20120120T10:04:25.852Z||2301|Received fd named: d810a903-961d-bb1e-aeb7-b39c98e5eefa - duping to 2 (from 6)
Jan 20 11:04:26 xcp-hoppe fe: 20120120T10:04:25.852Z||2301|Selecting in handle_comms_with_fd_sock2
Jan 20 11:04:26 xcp-hoppe fe: 20120120T10:04:25.852Z||2301|Done
Jan 20 11:04:26 xcp-hoppe fe: 20120120T10:04:25.852Z||2301|comms sock
Jan 20 11:04:26 xcp-hoppe fe: 20120120T10:04:25.852Z||2301|Exec
Jan 20 11:04:26 xcp-hoppe fe: 20120120T10:04:25.852Z||2301|Finished...
Jan 20 11:04:26 xcp-hoppe fe: 20120120T10:04:25.852Z||2301|Args after replacement = [/sbin/service;xapissl;restart]
Jan 20 11:04:26 xcp-hoppe fe: 20120120T10:04:25.852Z||2301|I've received the following fds: [2;1]Â
Jan 20 11:04:26 xcp-hoppe fe: 20120120T10:04:26.942Z||2301|Caught unexpected exception: Unix.Unix_error(31, "write", "")
Jan 20 10:05:41 xcp-hoppe HVM5652[12007]:ÂÂÂ XENNET: WARNING: No handler for oid 0xffda0014.
Jan 20 10:05:41 xcp-hoppe HVM5652[12007]:ÂÂÂ XENNET: WARNING: No handler for oid 0xffa0ffa1.
Jan 20 10:05:41 xcp-hoppe HVM5652[12007]:ÂÂÂ XENNET: WARNING: Bad tcp task offload header?
Jan 20 10:05:41 xcp-hoppe HVM5652[12007]:ÂÂÂ XENNET: WARNING: Bad tcp task offload header?
Jan 20 11:06:16 xcp-hoppe snmpd[15173]: Received SNMP packet(s) from UDP: [192.168.1.1]:2227
Jan 20 11:06:16 xcp-hoppe snmpd[15173]: Received SNMP packet(s) from UDP: [192.168.1.1]:2228
Jan 20 11:06:17 xcp-hoppe snmpd[15173]: Received SNMP packet(s) from UDP: [192.168.1.1]:2229
Jan 20 11:06:17 xcp-hoppe snmpd[15173]: Received SNMP packet(s) from UDP: [192.168.1.1]:2230
Jan 20 11:06:25 xcp-hoppe python: PERFMON: caught socket.error: (111 Connection refused) - restarting XAPI session
Jan 20 10:08:36 xcp-hoppe HVM5652[12007]: Time offset set 3569, added offset -1
Jan 20 11:08:54 xcp-hoppe python: PERFMON: Caught signal 15 - exiting
Jan 20 11:08:54 xcp-hoppe python: PERFMON: 11 Resource temporarily unavailable
Jan 20 11:08:54 xcp-hoppe python: PERFMON: Traceback (most recent call last):
Jan 20 11:08:54 xcp-hoppe python: PERFMON:ÂÂ File "/opt/xensource/bin/perfmon", line 930, in ?ÂÂÂÂ rc = main()
Jan 20 11:08:54 xcp-hoppe python: PERFMON:ÂÂ File "/opt/xensource/bin/perfmon", line 880, in mainÂÂÂÂ cmd = cmdsock.recv(cmdmaxlen)
Jan 20 11:08:54 xcp-hoppe python: PERFMON: error: (11, 'Resource temporarily unavailable')
Jan 20 11:08:54 xcp-hoppe python: PERFMON: caught socket.error: (111 Connection refused) - restarting XAPI session
Jan 20 11:08:55 xcp-hoppe v6d: [ info|xcp-hoppe|0||watchdog] (Re)starting v6d...
Jan 20 11:08:55 xcp-hoppe xapi: [ info|xcp-hoppe|0 thread_zero||watchdog] (Re)starting xapi...
Jan 20 10:10:45 xcp-hoppe HVM5641[28792]: Time offset set 3563, added offset -1
Jan 20 11:13:45 xcp-hoppe python: PERFMON: caught socket.error: (111 Connection refused) - restarting XAPI session
/var/log/SMI contains:
[2598] 2012-01-20 11:08:55.277830ÂÂÂ VASSR run ['/opt/xensource/sm/VASSR', '<methodCall><methodName>sr_get_driver_info</methodName><params><param><value><struct><member><name>host_ref</name><value>OpaqueRef:NULL</value></member><member><name>command</name><value>sr_get_driver_info</value></member><member><name>args</name><value><array><data/></array></value></member><member><name>device_config</name><value><struct/></value></member></struct></value></param></params></methodCall>']
[2598] 2012-01-20 11:08:55.278332ÂÂÂ Warning: vdi_[de]activate present for vastsky
[2619] 2012-01-20 11:08:55.858537ÂÂÂ Warning: vdi_[de]activate present for dummy
Not sure there's anything of value in those logs...
I'm somewhat desperate, as I'm unable to perform any xe commands and am worried a reboot of the server will not magically bring xapi back online, meaning the VMs will not start. This XCP host is critical to one of our customer's infrastructure. It's Friday and I could possibly go there this evening and perform tasks necessary to bring everything back online, I just don't quite know what my options are, aside from reboot and perhaps an in-place install of XCP.
Does anyone have any bright ideas? I'm all ears! Â Â
  Â
 Â
_______________________________________________
xen-api mailing list
xen-api@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/mailman/listinfo/xen-api
|
