Re: Design session "MSI-X support with Linux stubdomain" notes

[Marek Marczykowski-Górecki <marmarek@xxxxxxxxxxxxxxxxxxxxxx>]

On Thu, Sep 22, 2022 at 08:00:00PM +0200, Jan Beulich wrote:
> On 22.09.2022 18:05, Anthony PERARD wrote:
> > WARNING: Notes missing at the beginning of the meeting.
> > 
> > session description:
> >> Currently a HVM with PCI passthrough and Qemu Linux stubdomain doesn’t
> >> support MSI-X. For the device to (partially) work, Qemu needs a patch 
> >> masking
> >> MSI-X from the PCI config space. Some drivers are not happy about that, 
> >> which
> >> is understandable (device natively supports MSI-X, so fallback path are
> >> rarely tested).
> >>
> >> This is mostly (?) about qemu accessing /dev/mem directly (here:
> >> https://github.com/qemu/qemu/blob/master/hw/xen/xen_pt_msi.c#L579) - lets
> >> discuss alternative interface that stubdomain could use.
> > 
> > 
> > 
> > when qemu forward interrupt,
> >     for correct mask bit, it read physical mask bit.
> >     an hypercall would make sense.
> >     -> benefit, mask bit in hardware will be what hypervisor desire, and 
> > device model desire.
> >     from guest point of view, interrupt should be unmask.
> > 
> > interrupt request are first forwarded to qemu, so xen have to do some post 
> > processing once request comes back from qemu.
> >     it's weird..
> > 
> > someone should have a look, and rationalize this weird path.
> > 
> > Xen tries to not forward everything to qemu.
> > 
> > why don't we do that in xen.
> >     there's already code in xen for that.
> 
> So what I didn't pay enough attention to when talking was that the
> completion logic in Xen is for writes only. Maybe something similar
> can be had for reads as well, but that's to be checked ...

I spent some time trying to follow that part of qemu, and I think it
reads vector control only on the write path, to keep some bits
unchanged, and also detect whether Xen masked it behind qemu's back.
My understanding is, since 484d7c852e "x86/MSI-X: track host and guest
mask-all requests separately" it is unnecessary, because Xen will
remember guest's intention, so qemu can simply use its own internal
state and act on that (guest writes will go through qemu, so it should
have up to date view from guest's point of view).

As for PBA access, it is read by qemu only to pass it to the guest. I'm
not sure whether qemu should use hypercall to retrieve it, or maybe
Xen should fixup value itself on the read path.

I did some preliminary patch here:
https://github.com/marmarek/qubes-vmm-xen-stubdom-linux/commit/80cf769f3659aa0d7f2b5598bf862d83da28807e

but it does not work yet. It seems I haven't undo MSI-X hiding enough
(lspci inside the guest doesn't report MSI-X at all). This I will figure
out, but I'd appreciate comments about how to handle PBA best.

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

Attachment: signature.asc
Description: PGP signature