[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] HVM domains crash after upgrade from XEN 4.5.1 to 4.5.2



Am 17.11.15 um 00:10 schrieb Atom2:
Am 17.11.15 um 00:01 schrieb Andrew Cooper:
On 16/11/2015 19:16, Atom2 wrote:

Am 16.11.15 um 16:31 schrieb Konrad Rzeszutek Wilk:
Your analysis was absolutely spot on. After re-thinking this for a
moment, I thought going down that route first would make a lot of
sense
as PV guests still do work and one of the differences to HVM domUs is that the former do _not_ require SeaBIOS. Looking at my log files of
installed packages confirmed an upgrade from SeaBIOS 1.7.5 to
1.8.2 in
the relevant timeframe which obviously had not made it to the
hvmloader
of xen-4.5.1 as I did not re-compile xen after the upgrade of
SeaBIOS.

So I re-compiled xen-4.5.1 (obviously now using the installed SeaBIOS
1.8.2) and the same error as with xen-4.5.2 popped up - and that
seemed
to strongly indicate that there indeed might be an issue with
SeaBIOS as
this probably was the only variable that had changed from the
original
install of xen-4.5.1.
I recall seeing this way back in Fedora 20 days. I narrowed it down the
SeaBIOS version that was a standalone package to not have CONFIG_XEN.

Having that fixed in the SeaBIOS package fixed it.
Hi Konrad, Doug, Andrew (specifically added to this part of the thread)!
Konrad, you might have found an interesting point. I did have a look
at the ebuild for the failing version and in there I found the
following comment:
====== comment from ebuild =======
     # Upstream hasn't released a new binary.  We snipe ours from
Fedora for now.
# http://code.coreboot.org/p/seabios/downloads/get/bios.bin-${PV}.gz
====== end comment from ebuild =======
which might in fact underline that there might be an issue similar to
what you described above.

What is also pretty interesting is the fact that the old (working)
SeaBIOS version 1.7.5 installed as "bios.bin" under /usr/share/seabios
is actually 262.144 bytes in size whereas the new (invalid) SeaBIOS
1.8.2 installed in the same location is only half as big: 131.072 bytes.

I checked at the download site and the 1.8.2 binary version is indeed
not available from http://code.coreboot.org/p/seabios/downloads/. But
both the binary versions for 1.7.5 and 1.8.0 are available and both
are acutually 262.144 bytes in size, so I'd be very surprised if the
1.8.2 version is really only half that size. By the way, the old
working version (according to the ebuild) was directly downloaded from
the above url and also shows an identical SHA1 digest to that version
available for download there.

To me this looks as if something is really wrong here. If anybody of
you has access to a 1.8.2 version, could you please confirm whether
there's really that big a size difference between the 1.7.5 and the
1.8.2 versions? Or is that difference probably attributable to the
missing CONFIG_XEN option?

Andrew: I havent't gotten around to run the debug version of the
hypervisor again, but if the current suspicion turns out to be true,
there's probably not much value in that anyways. Would you agree?
Sadly not.
Fair enough. I'll try to get things done, hopefully somewhen tomorrow or, in case that doesn't work out, on Wednesday and will send you the requested information.

Many thanks for your support, Atom2
I accept that this issue is possibly fixed in newer SeaBIOS by working
around the issue.

However, I stand by my original point.  *There is no way the guest
should be able to get into this situation in the first place*, and its
implication of *there is a genuine hypervisor bug which we should track
down*, irrespective of whether the issue has been worked around elsehow
Hi Andrew,
as promised I have again tried with a debug build and the results are very mixed. I initially tried to better understand what the debug USE flag actually does in gentoo and my understanding (after reading the so called ebuilds) is now that the XEN hypervisor will be built by adding a gcc option of "debug=y" - and that's what should compile a debug build - right? So I went on and again enabled the debug USE flag plus gdb symbols and rebuilt the hypervisor in the hope that this created a valid and working debug build.

It, however, seems there's another problem lurking somewhere which only manifests itself when I boot from the debug build of the hypervisor. The system crashes early on with a DOUBLE FAULT in doIRQ - we have had this already earlier in that thread. I am however a step further as the disass in gdb now seems to provide not just an empty page full of NULL values but rather something that might give you a hint why it crashes that early on: Please see the attached disass file (doIRQ) together with the serial console output (serial.dbg). The old NULL value file was probably because I did not include gdb symbols in the debug build at that time - my bad.

So I am at loss why a debug build actually results in a crash early on, but probably Doug could step in here and explain that ... it initially thought that it might have to do with the hardened sources that I am using, but discarded that option because linux does not play any role that early on in the boot process (albeit all lines in the serial log are prefixed with "(XEN)").

Because using a debug build of the hypervisor did not work out, I at least used the debug build of the xen-tools (which includes hvmloader) and run this on the working hypervisor (i.e. the standard, non-debug build) in the hope that this might prove useful.

Please find attached the "xl dmesg" from an attempt to start a HVM domU. There's a bit more information included compared to the previous xl dmesg output, but I don't know whether that's of any help. Unfortunately I could not spot any lines "prefixed with [HVM:$DOMID.$VCPUID]" which you had expected. But I assume that's because I was forced to boot from a non-debug build of the hypervisor in order to get the system up at all.

So to me it looks as if we are currently stuck - unless you (or Doug) are able to find the root cause of the early crash when using the debug build of the hypervisor. If that is resolved, I could then move on to the next step and hopefully be able to provide the information that you are after.

Sorry for not being able to provide better input right now.

Thanks Atom2

Attachment: serial.dbg
Description: Text document

Attachment: doIRQ
Description: Text document

Attachment: xl.dmesg
Description: Text document

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel

 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.