Re: [Xen-devel] Issues with PCI-Passtrough (VT-d) in HVM with Xen 4.6

["Jürgen Walter" <juwalter@xxxxxxxxx>]

On 3 Jun 2016, at 15:26, Jan Beulich wrote:
> I'll see to put together a patch for the recognizable problem,
> but I'm still unclear about your actual one.
regarding the actual one: while it is still not working, I managed to 
dig deeper and found a few observations
background: there is a driver/kernel module, and a user space library, 
and a "test console"
the "test console" uses the library, and invokes "cifXInit" (initialise 
the PCI card)
during init, it does
a) read pci config (space)
b) send reset sequence to the PCI device
c) writes back the pci config saved from step "a)" (restore)

I used "lspci -xxx -s BDF" to read that config space, now

- on the Dom0 - when I run that "test console", everything works as 
expected, and the "lspci -xxx" output is the same before and after the 
reset sequence (there is a delay built-in to allow rebooting of the ARM 
chip on the device)
before the reset:

06:00.0 Unassigned class [ff00]: Hilscher GmbH CIFX 50E-DP(M/S)
00: cf 15 00 00 02 01 00 02 00 00 00 ff 00 40 00 00
10: 00 00 a0 f7 00 00 00 00 00 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00
30: 00 00 00 00 00 00 00 00 00 00 00 00 0b 01 00 00
40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
50: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

diff before-reset after-reset:

2,3c2,3
< 00: cf 15 00 00 02 01 00 02 00 00 00 ff 00 40 00 00
< 10: 00 00 a0 f7 00 00 00 00 00 00 00 00 00 00 00 00
---
> 00: cf 15 00 00 00 00 00 02 00 00 00 ff 00 00 00 00
> 10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
5c5
< 30: 00 00 00 00 00 00 00 00 00 00 00 00 0b 01 00 00
---
> 30: 00 00 00 00 00 00 00 00 00 00 00 00 00 01 00 00
and it is 100% identical to "before reset" after the configuration 
restore operation

- on the DomU - when I run that "test console" tool, the "lspci -xxx" 
output is different from before/after
not much, though, only one register(?)

diff lspci.before-testconsole lspci.after-testconsole
2c2
< 00: cf 15 00 00 02 01 00 02 00 00 00 ff 00 40 00 00
---
> 00: cf 15 00 00 02 01 00 02 00 00 00 ff 00 00 00 00

Now, I my desperation I found out about "setpci" and thought I give it a 
shot, alas no dice, it does not seem to write
root@DomU: setpci -s 00:00.0 D.B # the device is at 00.00.0

log from Dom0 # the device is at 06.00.0
        xen-pciback: 0000:06:00.0: read 1 bytes at 0xd
        xen-pciback: 0000:06:00.0: read 1 bytes at 0xd = 0

root@DomU: setpci -s 00:00.0 D.B=40 # the device is at 00.00.0

log from Dom0 # the device is at 06.00.0
        xen-pciback: 0000:06:00.0: write request 1 bytes at 0xd = 40
        xen-pciback: 0000:06:00.0: read 1 bytes at 0xd
        xen-pciback: 0000:06:00.0: read 1 bytes at 0xd = 0



Even though, when I do "cat /sys/bus/pci/drivers/pciback/quirks" it duly 
lists this address as writeable:
06:00.0
        15cf:0000:0000:0000
                00000000:2:00000000
                00000002:2:00000000
                00000004:2:00000000
                0000003c:1:00000000
                0000003d:1:00000000
                0000000c:1:00000000
                0000000d:1:00000000  <------- ok to write 1 byte
                0000000f:1:00000000
                00000010:4:00000000
                00000014:4:00000000
                00000018:4:00000000
                0000001c:4:00000000
                00000020:4:00000000
                00000024:4:00000000
                00000030:4:00000000



Lastly, this is the only difference of "lspci -xxx" (after a _PCI_ 
device reset) between Dom0 and DomU
diff lspci.before-domU lspci.before-dom0
< 30: 00 00 00 00 00 00 00 00 00 00 00 00 10 01 00 00
---
> 30: 00 00 00 00 00 00 00 00 00 00 00 00 00 01 00 00


Long story short, not sure where to look anymore :(

It just seems like this thing is not "coming back" properly after the 
driver send the device reset.
The comment in the driver source say, if this address has 0xFFFFFFFF 
"This happens if memory is not available."
(but I doubt, this is necessarily the correct error message)

Very lastly, when I do "xl dmesg | grep VT-d" - there is some msg re: no 
Dom0 DMA passtrough
(XEN) Intel VT-d iommu 0 supported page sizes: 4kB.
(XEN) Intel VT-d iommu 1 supported page sizes: 4kB.
(XEN) Intel VT-d Snoop Control not enabled.
(XEN) Intel VT-d Dom0 DMA Passthrough not enabled.         <------ ?? 
important ???
(XEN) Intel VT-d Queued Invalidation enabled.
(XEN) Intel VT-d Interrupt Remapping enabled.
(XEN) Intel VT-d Shared EPT tables not enabled.


Thanks again for reading that far! Jürgen

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel