MSI-X cleanup(?) issue with passthrough after domU restart

[Marek Marczykowski-Górecki <marmarek@xxxxxxxxxxxxxxxxxxxxxx>]

Hi,

I'm hitting an MSI-X issue after rebooting the domU. The symptoms are
rather boring: on initial domU start the device (realtek eth card) works
fine, but after domU restart, the link doesn't come up (there is no
"Link is Up" message anymore). No errors from domU driver or Xen. I
tracked it down to MSI-X - if I force INTx (via pci=nomsi on domU
cmdline) it works fine. Convincing the driver to poll instead of waiting
for an interrupt also workarounds the issue.

I noticed also some interrupts are not cleaned up on restart. The list
of MSIs in 'Q' debug key output grows:

    (XEN) 0000:03:00.0 - d22 - node -1  - MSIs < 41 42 43 44 45 46 47 >
    restart sys-net domU
    (XEN) 0000:03:00.0 - d24 - node -1  - MSIs < 41 42 43 44 45 46 47 48 >
    restart sys-net domU
    (XEN) 0000:03:00.0 - d26 - node -1  - MSIs < 41 42 43 44 45 46 47 48 49 >

and 'M' output is:

    (XEN)  MSI-X   41 vec=b1 lowest  edge   assert  log lowest dest=00000001 
mask=1/H /1
    (XEN)  MSI-X   42 vec=b9 lowest  edge   assert  log lowest dest=00000004 
mask=1/HG/1
    (XEN)  MSI-X   43 vec=c1 lowest  edge   assert  log lowest dest=00000010 
mask=1/HG/1
    (XEN)  MSI-X   44 vec=d9 lowest  edge   assert  log lowest dest=00000001 
mask=1/HG/1
    (XEN)  MSI-X   45 vec=e1 lowest  edge   assert  log lowest dest=00000001 
mask=1/HG/1
    (XEN)  MSI-X   46 vec=e9 lowest  edge   assert  log lowest dest=00000040 
mask=1/HG/1
    (XEN)  MSI-X   47 vec=32 lowest  edge   assert  log lowest dest=00000004 
mask=1/HG/1
    (XEN)  MSI-X   48 vec=3a lowest  edge   assert  log lowest dest=00000040 
mask=1/HG/1
    (XEN)  MSI-X   49 vec=42 lowest  edge   assert  log lowest dest=00000010 
mask=1/ G/1

And also, after starting and stopping the domU, `xl pci-assignable-remove 
03:00.0`
makes pciback to complain:

    [ 1180.919874] pciback 0000:03:00.0: xen_pciback: MSI-X release failed (-16)

This is all running on Xen 4.19.3, but I don't see much changes in this
area since then.

Some more info collected at https://github.com/QubesOS/qubes-issues/issues/9335

My question is: what should be responsible for this cleanup on domain
destroy? Xen, or maybe device model (which is QEMU in stubdomain here)?
I see some cleanup (apparently not enough) happening via QEMU when the
domU driver is unloaded, but logically correct cleanup shouldn't depend
on correct domU operation...

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

Attachment: signature.asc
Description: PGP signature