[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] Re: [Xen-devel] HVM domains crash after upgrade from XEN 4.5.1 to 4.5.2
Am 17.11.15 um 00:10 schrieb Atom2: Am 17.11.15 um 00:01 schrieb Andrew Cooper:Fair enough. I'll try to get things done, hopefully somewhen tomorrow or, in case that doesn't work out, on Wednesday and will send you the requested information.On 16/11/2015 19:16, Atom2 wrote:Am 16.11.15 um 16:31 schrieb Konrad Rzeszutek Wilk:Hi Konrad, Doug, Andrew (specifically added to this part of the thread)!I recall seeing this way back in Fedora 20 days. I narrowed it down theYour analysis was absolutely spot on. After re-thinking this for a moment, I thought going down that route first would make a lot of senseas PV guests still do work and one of the differences to HVM domUs is that the former do _not_ require SeaBIOS. Looking at my log files ofinstalled packages confirmed an upgrade from SeaBIOS 1.7.5 to 1.8.2 in the relevant timeframe which obviously had not made it to the hvmloader of xen-4.5.1 as I did not re-compile xen after the upgrade of SeaBIOS.So I re-compiled xen-4.5.1 (obviously now using the installed SeaBIOS1.8.2) and the same error as with xen-4.5.2 popped up - and that seemed to strongly indicate that there indeed might be an issue with SeaBIOS as this probably was the only variable that had changed from the original install of xen-4.5.1.SeaBIOS version that was a standalone package to not have CONFIG_XEN. Having that fixed in the SeaBIOS package fixed it.Konrad, you might have found an interesting point. I did have a look at the ebuild for the failing version and in there I found the following comment: ====== comment from ebuild ======= # Upstream hasn't released a new binary. We snipe ours from Fedora for now.# http://code.coreboot.org/p/seabios/downloads/get/bios.bin-${PV}.gz====== end comment from ebuild ======= which might in fact underline that there might be an issue similar to what you described above. What is also pretty interesting is the fact that the old (working) SeaBIOS version 1.7.5 installed as "bios.bin" under /usr/share/seabios is actually 262.144 bytes in size whereas the new (invalid) SeaBIOS1.8.2 installed in the same location is only half as big: 131.072 bytes.I checked at the download site and the 1.8.2 binary version is indeed not available from http://code.coreboot.org/p/seabios/downloads/. But both the binary versions for 1.7.5 and 1.8.0 are available and both are acutually 262.144 bytes in size, so I'd be very surprised if the 1.8.2 version is really only half that size. By the way, the old working version (according to the ebuild) was directly downloaded from the above url and also shows an identical SHA1 digest to that version available for download there. To me this looks as if something is really wrong here. If anybody of you has access to a 1.8.2 version, could you please confirm whether there's really that big a size difference between the 1.7.5 and the 1.8.2 versions? Or is that difference probably attributable to the missing CONFIG_XEN option? Andrew: I havent't gotten around to run the debug version of the hypervisor again, but if the current suspicion turns out to be true, there's probably not much value in that anyways. Would you agree?Sadly not.Many thanks for your support, Atom2I accept that this issue is possibly fixed in newer SeaBIOS by working around the issue. However, I stand by my original point. *There is no way the guest should be able to get into this situation in the first place*, and its implication of *there is a genuine hypervisor bug which we should track down*, irrespective of whether the issue has been worked around elsehow Hi Andrew,as promised I have again tried with a debug build and the results are very mixed. I initially tried to better understand what the debug USE flag actually does in gentoo and my understanding (after reading the so called ebuilds) is now that the XEN hypervisor will be built by adding a gcc option of "debug=y" - and that's what should compile a debug build - right? So I went on and again enabled the debug USE flag plus gdb symbols and rebuilt the hypervisor in the hope that this created a valid and working debug build. It, however, seems there's another problem lurking somewhere which only manifests itself when I boot from the debug build of the hypervisor. The system crashes early on with a DOUBLE FAULT in doIRQ - we have had this already earlier in that thread. I am however a step further as the disass in gdb now seems to provide not just an empty page full of NULL values but rather something that might give you a hint why it crashes that early on: Please see the attached disass file (doIRQ) together with the serial console output (serial.dbg). The old NULL value file was probably because I did not include gdb symbols in the debug build at that time - my bad. So I am at loss why a debug build actually results in a crash early on, but probably Doug could step in here and explain that ... it initially thought that it might have to do with the hardened sources that I am using, but discarded that option because linux does not play any role that early on in the boot process (albeit all lines in the serial log are prefixed with "(XEN)"). Because using a debug build of the hypervisor did not work out, I at least used the debug build of the xen-tools (which includes hvmloader) and run this on the working hypervisor (i.e. the standard, non-debug build) in the hope that this might prove useful. Please find attached the "xl dmesg" from an attempt to start a HVM domU. There's a bit more information included compared to the previous xl dmesg output, but I don't know whether that's of any help. Unfortunately I could not spot any lines "prefixed with [HVM:$DOMID.$VCPUID]" which you had expected. But I assume that's because I was forced to boot from a non-debug build of the hypervisor in order to get the system up at all. So to me it looks as if we are currently stuck - unless you (or Doug) are able to find the root cause of the early crash when using the debug build of the hypervisor. If that is resolved, I could then move on to the next step and hopefully be able to provide the information that you are after. Sorry for not being able to provide better input right now. Thanks Atom2 Attachment:
serial.dbg Attachment:
doIRQ Attachment:
xl.dmesg _______________________________________________ Xen-devel mailing list Xen-devel@xxxxxxxxxxxxx http://lists.xen.org/xen-devel
|
Lists.xenproject.org is hosted with RackSpace, monitoring our |