[Xen-users] Debugging (possible) Xen-related hang-issues without the possibility of attaching serial console to capture Xen output

[Heiko Wundram <modelnine@xxxxxxxxxxxxx>]

Hey all!

I'm currently in the process of trying to debug a (possibly!) 
Xen-related issue (with Kernel 3.2.9-vanilla as Dom0, and Xen 4.1.2 
[almost] vanilla, except for a patch for CVE-2012-0029), where a system 
freezes without me being able to ascertain a triggering event. The 
system console is completely unresponsive in the event of hangs (i.e., 
displays motd and the login prompt, but doesn't display a Dom0 traceback 
or anything related), and I'm unable to pinpoint the exact source of the 
hang condition due to not being able to reproduce the hang under 
laboratory-conditions (which is the worst part about these hangs) with 
similar hardware (so, basically, it might actually not be Xen- but 
[hosting-]environment-related).
I'd already written about this some time ago, and the suggestion to 
enable network logging (through netconsole) is implemented on the system 
in question, but when the system locks up I'm also not seeing any output 
to the netconsole connected process on the remote logging machine - it 
seems that the Dom0-kernel is not "the guilty party" when these hangs 
occur.
Generally, I'd have started to attach a serial console to the system 
now to check whether the hypervisor logs any fatal errors when the 
system goes to a hang, but as the machine in question is located at a 
service provider (which I can't change easily and which adamantly 
refuses to help with debugging in the form of attaching a serial 
console), I'm unable to get at Xen serial output on the host in 
question.
Is there any, _any_ other way to get at hypervisor output in the case 
of a crash? i.e., some form of memory buffering of the output messages 
which I might later be able to find in the host memory after rebooting 
to a "working" state, or something similar? I know that FreeBSD (and 
lately) Linux implement this kind of crash debugging helpers - is there 
any possibility to get something similar working with Xen? Reverting to 
a kdump/kexec-enabled kernel (i.e., 2.6.18) for the Dom0 is (pretty 
much) impossible due to the hardware of the system in use, but if 
someone points me at a 2.6.34+ Xen-kexec/kdump-enabled Dom0-kernel, I'd 
also be happy to give that a try (I've not found any, or rather: 
kdump/kexec also doesn't work with the OpenSUSE-up-patched xenified 
2.6.38 I had in use before switching to 3.2.x-vanilla, but the problems 
[hangs] were similar with the older kernel).
Thanks for any answer!

--
--- Heiko.

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users