Re: MSI-X cleanup(?) issue with passthrough after domU restart

[Roger Pau Monné <roger.pau@xxxxxxxxxx>]

On Tue, Aug 26, 2025 at 03:55:05PM +0200, Marek Marczykowski-Górecki wrote:
> On Tue, Aug 26, 2025 at 12:57:50PM +0200, Marek Marczykowski-Górecki wrote:
> > On Tue, Aug 26, 2025 at 10:28:56AM +0200, Roger Pau Monné wrote:
> > > On Tue, Aug 26, 2025 at 08:16:56AM +0200, Jan Beulich wrote:
> > > > On 26.08.2025 03:49, Marek Marczykowski-Górecki wrote:
> > > > > Hi,
> > > > > 
> > > > > I'm hitting an MSI-X issue after rebooting the domU. The symptoms are
> > > > > rather boring: on initial domU start the device (realtek eth card) 
> > > > > works
> > > > > fine, but after domU restart, the link doesn't come up (there is no
> > > > > "Link is Up" message anymore). No errors from domU driver or Xen. I
> > > > > tracked it down to MSI-X - if I force INTx (via pci=nomsi on domU
> > > > > cmdline) it works fine. Convincing the driver to poll instead of 
> > > > > waiting
> > > > > for an interrupt also workarounds the issue.
> > > > > 
> > > > > I noticed also some interrupts are not cleaned up on restart. The list
> > > > > of MSIs in 'Q' debug key output grows:
> > > > > 
> > > > >     (XEN) 0000:03:00.0 - d22 - node -1  - MSIs < 41 42 43 44 45 46 47 
> > > > > >
> > > > >     restart sys-net domU
> > > > >     (XEN) 0000:03:00.0 - d24 - node -1  - MSIs < 41 42 43 44 45 46 47 
> > > > > 48 >
> > > > >     restart sys-net domU
> > > > >     (XEN) 0000:03:00.0 - d26 - node -1  - MSIs < 41 42 43 44 45 46 47 
> > > > > 48 49 >
> > > > > 
> > > > > and 'M' output is:
> > > > > 
> > > > >     (XEN)  MSI-X   41 vec=b1 lowest  edge   assert  log lowest 
> > > > > dest=00000001 mask=1/H /1
> > > > >     (XEN)  MSI-X   42 vec=b9 lowest  edge   assert  log lowest 
> > > > > dest=00000004 mask=1/HG/1
> > > > >     (XEN)  MSI-X   43 vec=c1 lowest  edge   assert  log lowest 
> > > > > dest=00000010 mask=1/HG/1
> > > > >     (XEN)  MSI-X   44 vec=d9 lowest  edge   assert  log lowest 
> > > > > dest=00000001 mask=1/HG/1
> > > > >     (XEN)  MSI-X   45 vec=e1 lowest  edge   assert  log lowest 
> > > > > dest=00000001 mask=1/HG/1
> > > > >     (XEN)  MSI-X   46 vec=e9 lowest  edge   assert  log lowest 
> > > > > dest=00000040 mask=1/HG/1
> > > > >     (XEN)  MSI-X   47 vec=32 lowest  edge   assert  log lowest 
> > > > > dest=00000004 mask=1/HG/1
> > > > >     (XEN)  MSI-X   48 vec=3a lowest  edge   assert  log lowest 
> > > > > dest=00000040 mask=1/HG/1
> > > > >     (XEN)  MSI-X   49 vec=42 lowest  edge   assert  log lowest 
> > > > > dest=00000010 mask=1/ G/1
> > > > > 
> > > > > And also, after starting and stopping the domU, `xl 
> > > > > pci-assignable-remove 03:00.0`
> > > > > makes pciback to complain:
> > > > > 
> > > > >     [ 1180.919874] pciback 0000:03:00.0: xen_pciback: MSI-X release 
> > > > > failed (-16)
> > > > > 
> > > > > This is all running on Xen 4.19.3, but I don't see much changes in 
> > > > > this
> > > > > area since then.
> > > > > 
> > > > > Some more info collected at 
> > > > > https://github.com/QubesOS/qubes-issues/issues/9335
> > > > > 
> > > > > My question is: what should be responsible for this cleanup on domain
> > > > > destroy? Xen, or maybe device model (which is QEMU in stubdomain 
> > > > > here)?
> > > > 
> > > > The expectation is that qemu invokes the necessary cleanup, but of 
> > > > course ...
> > > > 
> > > > > I see some cleanup (apparently not enough) happening via QEMU when the
> > > > > domU driver is unloaded, but logically correct cleanup shouldn't 
> > > > > depend
> > > > > on correct domU operation...
> > > > 
> > > > ... Xen may not make itself dependent upon either DomU or QEMU.
> > > 
> > > AFAICT free_domain_pirqs() called by arch_domain_destroy() should take
> > > care of unbinding and freeing pirqs (but obviously not in this case).
> > > Can you repeat the test with a debug=y hypervisor and post the
> > > resulting serial or dmesg here?  Some of the errors on those paths are
> > > printed with dprintk() and won't be visible unless using a Xen debug
> > > build.
> > 
> > Sure, will do.
> 
> Output collected during domU shutdown and subsequent startup (dom0 logs
> to Xen console here too):
> https://gist.github.com/marmarek/6dc3ac14d3ba840482e6361fcbd37c30
> 
> I don't see any errors there...

Hm, yes, I don't see any errors either.  Do you think you could
instrument unmap_domain_pirq() and figure out whether it gets called,
and if such call to unmap the PIRQ succeeds?

There's one silent error path in the return value of
xsm_unmap_domain_irq(), but that shouldn't be taken since the call to
unmap_domain_pirq() executed when destroying the domain is done with
d->is_dying == true.

> As for the domU-triggered cleanup, I just checked - if I unload the
> driver in domU before restarting, it works fine on subsequent startup.

Yeah, the unbind an unmap is then done by QEMU, and that seems to
work fine.  What doesn't seem to work fine is the cleanup done as part
of domain destroy.

Don't we have any Gitlab tests that do PCI passthrough?  Maybe those
don't do any domU resets?

Thanks, Roger.