Re: [Xen-devel] pci-passthrough loses msi-x interrupts ability after domain destroy

[Sander Eikelenboom <linux@xxxxxxxxxxxxxx>]

Thursday, September 21, 2017, 10:39:52 AM, you wrote:

> On Wed, Sep 20, 2017 at 03:50:35PM -0400, Jérôme Oufella wrote:
>> Hi Xen-devel, 
>> 
>> I'm using PCI pass-through to map a PCIe (intel i210) controller into 
>> a HVM domain. The system uses xen-pciback to hide the appropriate PCI 
>> device from Dom0. 
>> 
>> When creating the HVM domain after an hypervisor cold boot, the HVM 
>> domain can access and use the PCIe controller without problem. 
>> 
>> However, if the HVM domain is destroyed then restarted, it won't be 
>> able to use the pass-through PCI device anymore. The PCI device is 
>> seen and can be mapped, however, the interrupts will not be passed to 
>> the HVM domain anymore (this is visible under a Linux guest as 
>> /proc/interrupts counters remain 0). The behavior on a Windows10 guest 
>> is the same. 
>> 
>> A few interesting hints I noticed: 
>> 
>> - On Dom0, 'lspci -vv' on that PCIe device between the "working" and 
>> the "muted interrupts" states, I noted a difference between the 
>> MSI-X caps: 
>> 
>> - Capabilities: [70] MSI-X: Enable- Count=5 Masked- <-- IRQs will work if 
>> domain started 
>> + Capabilities: [70] MSI-X: Enable- Count=5 Masked+ <-- IRQs won't work if 
>> domain started
>>                                             ^^^^^^^

> IMHO it seems that either your device is not able to perform a reset
> successfully, or Linux is not correctly performing such reset. I don't
> think there's a lot that can be done from the Xen side.

Unfortunately for a lot of pci-devices a simple reset as performed by default 
isn't enough, 
but also almost none support a real pci FLR.
 
In the distant past Konrad has made a patchset that implemented a bus reset and
reseting config space. (It piggy backed on already existing libxl mechanism of 
trying to call on a syfs "do_flr" attribute which triggers pciback to perform 
the busreset and rewrite of config space for the device.

I use that patchset ever since for my pci-passtrough needs and it works pretty 
well. I can shutdown an restart VM's with pci devices passed trhough (also AMD 
Radeon graphic cards).

Jérôme:
Although your mileage may vary, you could try the attached patch, it's
for a > 4.9 linux kernel's pciback, although it probably also would apply with 
for a > minimal adjustments (mostly non matching line numbers) to an some what 
earlier kernel. 

If it works for you as well, perhaps it deserves a mention on the Xen wiki in 
the 
pci-passtrough section.

Roger: 
I follow your PVH (dom0) patches shallowly, from my understanding it will 
result 
in Xen having more inteference with the handling of PCI devices ? 
If that's correct will this also impact the resetting logic, or will most stay 
in  the dom0 kernel/pciback ?

--
Sander

> Thanks, Roger.

Attachment: pciback_do_flr_4_9.patch
Description: Binary data

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel