Re: [Xen-devel] PV guest with PCI passthrough crash on Xen 4.8.3 inside KVM when booted through OVMF

[Marek Marczykowski-Górecki <marmarek@xxxxxxxxxxxxxxxxxxxxxx>]

On Mon, Feb 19, 2018 at 06:30:14PM +0100, Juergen Gross wrote:
> On 16/02/18 20:02, Andrew Cooper wrote:
> > On 16/02/18 18:51, Marek Marczykowski-Górecki wrote:
> >> On Fri, Feb 16, 2018 at 05:52:50PM +0000, Andrew Cooper wrote:
> >>> On 16/02/18 17:48, Marek Marczykowski-Górecki wrote:
> >>>> Hi,
> >>>>
> >>>> As in the subject, the guest crashes on boot, before kernel output
> >>>> anything. I've isolated this to the conditions below:
> >>>>  - PV guest have PCI device assigned (e1000e emulated by QEMU in this 
> >>>> case),
> >>>>    without PCI device it works
> >>>>  - Xen (in KVM) is started through OVMF; with seabios it works
> >>>>  - nested HVM is disabled in KVM
> >>>>  - AMD IOMMU emulation is disabled in KVM; when enabled qemu crashes on
> >>>>    boot (looks like qemu bug, unrelated to this one)
> >>>>
> >>>> Version info:
> >>>>  - KVM host: OpenSUSE 42.3, qemu 2.9.1, 
> >>>> ovmf-2017+git1492060560.b6d11d7c46-4.1, AMD
> >>>>  - Xen host: Xen 4.8.3, dom0: Linux 4.14.13
> >>>>  - Xen domU: Linux 4.14.13, direct boot
> >>>>
> >>>> Not sure if relevant, but initially I've tried booting xen.efi /mapbs
> >>>> /noexitboot and then dom0 kernel crashed saying something about conflict
> >>>> between e820 and kernel mapping. But now those options are disabled.
> >>>>
> >>>> The crash message:
> >>>> (XEN) d1v0 Unhandled invalid opcode fault/trap [#6, ec=0000]
> >>>> (XEN) domain_crash_sync called from entry.S: fault at ffff82d080218720 
> >>>> entry.o#create_bounce_frame+0x137/0x146
> >>>> (XEN) Domain 1 (vcpu#0) crashed on cpu#1:
> >>>> (XEN) ----[ Xen-4.8.3  x86_64  debug=n   Not tainted ]----
> >>>> (XEN) CPU:    1
> >>>> (XEN) RIP:    e033:[<ffffffff826d9156>]
> >>> This is #UD, which is most probably hitting a BUG().  addr2line this ^
> >>> to find some code to look at.
> >> addr2line failed me
> > 
> > By default, vmlinux is stripped and compressed.  Ideally you want to
> > addr2line the vmlinux artefact in the root of your kernel build, which
> > is the plain elf with debugging symbols.
> > 
> > Alternatively, use scripts/extract-vmlinux on the binary you actually
> > booted, which might get you somewhere.
> > 
> >> , but System.map says its xen_memory_setup. And it
> >> looks like the BUG() is the same as I had in dom0 before:
> >> "Xen hypervisor allocated kernel memory conflicts with E820 map".
> > 
> > Juergen: Is there anything we can do to try and insert some dummy
> > exception handlers right at PV start, so we could at least print out a
> > oneliner to the host console which is a little more helpful than Xen
> > saying "something unknown went wrong" ?
> 
> You mean something like commit 42b3a4cb5609de757f5445fcad18945ba9239a07
> added to kernel 4.15?
> 
> > 
> >>
> >> Disabling e820_host in guest config solved the problem. Thanks!
> >>
> >> Is this some bug in Xen or OVMF, or is it expected behavior and e820_host
> >> should be avoided?
> > 
> > I don't really know.  e820_host is a gross hack which shouldn't really
> > be present.  The actually problem is that Linux can't cope with the
> > memory layout it was given (and I can't recall if there is anything
> > Linux could potentially to do cope).  OTOH, the toolstack, which knew
> > about e820_host and chose to lay the guest out in an overlapping way is
> > probably also at fault.
> 
> The kernel can cope with lots of E820 scenarios (e.g. by relocating
> initrd or the p2m map), but moving itself out of the way is not
> possible.

I'm afraid I need to resurrect this thread...

With recent kernel (6.6+), the host_e820=0 workaround is not an option
anymore. It makes Linux not initialize xen-swiotlb (due to
f9a38ea5172a3365f4594335ed5d63e15af2fd18), so PCI passthrough doesn't
work at all. While I can add yet another layer of workaround (force
xen-swiotlb with iommu=soft), that's getting unwieldy.

Furthermore, I don't get the crash message anymore, even with debug
hypervisor and guest_loglvl=all. Not even "Domain X crashed" in `xl
dmesg`. It looks like the "crash" shutdown reason doesn't reach Xen, and
it's considered clean shutdown (I can confirm it by changing various
`on_*` settings (via libvirt) and observing which gets applied).

Most tests I've done with 6.7-rc1, but the issue I observed on 6.6.1
already.

This is on Xen 4.17.2. And the L0 is running Linux 6.6.1, and then uses
QEMU 8.1.2 + OVMF 202308 to run Xen as L1.

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

Attachment: signature.asc
Description: PGP signature