Re: [Xen-devel] Issues with PCI-Passtrough (VT-d) in HVM with Xen 4.6

["Jürgen Walter" <juwalter@xxxxxxxxx>]

Hi Jan,

On 6 Jun 2016, at 10:41, Jan Beulich wrote:

> > > > On 04.06.16 at 17:15, <juwalter@xxxxxxxxx> wrote:
> > On 3 Jun 2016, at 15:26, Jan Beulich wrote:
> > > I'll see to put together a patch for the recognizable problem,
> > > but I'm still unclear about your actual one.
> > regarding the actual one: while it is still not working, I managed to
> > dig deeper and found a few observations
> >
> > background: there is a driver/kernel module, and a user space 
> > library,
> > and a "test console"
> > the "test console" uses the library, and invokes "cifXInit" 
> > (initialise
> > the PCI card)
> >
> > during init, it does
> > a) read pci config (space)
> > b) send reset sequence to the PCI device
> > c) writes back the pci config saved from step "a)" (restore)
> I'm sorry to mention that, but so far you still didn't provide a
> matching pciback log for that portion of the operation, at least
> afaict.
sorry about that- we managed to pull this together. note: we changed the 
user space library to perform another read after the write/restore
we did on Dom0 and DomU, "inside permissive mode" printk statement we 
added if it goes into "if (!handled ...)"
DOM0

# read PCI config and save for later restore

(gdb) x/64wx 0x622190
0x622190:       0x000015cf      0x02000107      0xff000000      0x00004000
0x6221a0:       0xf7a00000      0x00000000      0x00000000      0x00000000
0x6221b0:       0x00000000      0x00000000      0x00000001      0x00000000
0x6221c0:       0x00000000      0x00000000      0x00000000      0x0000010b
0x6221d0:       0x00000000      0x00000000      0x00000000      0x00000000
0x6221e0:       0x00000000      0x00000000      0x00000000      0x00000000
0x6221f0:       0x00000000      0x00000000      0x00000000      0x00000000
0x622200:       0x00000000      0x00000000      0x00000000      0x00000000
0x622210:       0x00000000      0x00000000      0x00000000      0x00000000
0x622220:       0x00000000      0x00000000      0x00000000      0x00000000
0x622230:       0x00000000      0x00000000      0x00000000      0x00000000
0x622240:       0x00000000      0x00000000      0x00000000      0x00000000
0x622250:       0x00000000      0x00000000      0x00000000      0x00000000
0x622260:       0x00000000      0x00000000      0x00000000      0x00000000
0x622270:       0x00000000      0x00000000      0x00000000      0x00000000
0x622280:       0x00000000      0x00000000      0x00000000      0x00000000

# write/restore PCI config
# ...
# read PCI config again and see what is there now

(gdb) x/64w 0x622190
0x622190:       0x000015cf      0x02000107      0xff000000      0x00004000 <--- 
worked
0x6221a0:       0xf7a00000      0x00000000      0x00000000      0x00000000
0x6221b0:       0x00000000      0x00000000      0x00000001      0x00000000
0x6221c0:       0x00000000      0x00000000      0x00000000      0x0000010b
0x6221d0:       0x00000000      0x00000000      0x00000000      0x00000000
0x6221e0:       0x00000000      0x00000000      0x00000000      0x00000000
0x6221f0:       0x00000000      0x00000000      0x00000000      0x00000000
0x622200:       0x00000000      0x00000000      0x00000000      0x00000000
0x622210:       0x00000000      0x00000000      0x00000000      0x00000000
0x622220:       0x00000000      0x00000000      0x00000000      0x00000000
0x622230:       0x00000000      0x00000000      0x00000000      0x00000000
0x622240:       0x00000000      0x00000000      0x00000000      0x00000000
0x622250:       0x00000000      0x00000000      0x00000000      0x00000000
0x622260:       0x00000000      0x00000000      0x00000000      0x00000000
0x622270:       0x00000000      0x00000000      0x00000000      0x00000000
0x622280:       0x00000000      0x00000000      0x00000000      0x00000000
(gdb)


DOMU

# read PCI config and save for later restore

(gdb) p pvPCIConfig
$2 = (void *) 0x61ef90
(gdb) x/64wx 0x61ef90
0x61ef90:       0x000015cf      0x02000102      0xff000000      0x00004000
0x61efa0:       0xf7a00000      0x00000000      0x00000000      0x00000000
0x61efb0:       0x00000000      0x00000000      0x00000001      0x00000000
0x61efc0:       0x00000000      0x00000000      0x00000000      0x00000110
0x61efd0:       0x00000000      0x00000000      0x00000000      0x00000000
0x61efe0:       0x00000000      0x00000000      0x00000000      0x00000000
0x61eff0:       0x00000000      0x00000000      0x00000000      0x00000000
0x61f000:       0x00000000      0x00000000      0x00000000      0x00000000
0x61f010:       0x00000000      0x00000000      0x00000000      0x00000000
0x61f020:       0x00000000      0x00000000      0x00000000      0x00000000
0x61f030:       0x00000000      0x00000000      0x00000000      0x00000000
0x61f040:       0x00000000      0x00000000      0x00000000      0x00000000
0x61f050:       0x00000000      0x00000000      0x00000000      0x00000000
0x61f060:       0x00000000      0x00000000      0x00000000      0x00000000
0x61f070:       0x00000000      0x00000000      0x00000000      0x00000000
0x61f080:       0x00000000      0x00000000      0x00000000      0x00000000

# write/restore PCI config
# ...
# read PCI config again and see what is there now

(gdb) p OS_ReadPCIConfig(ptDevInstance->pvOSDependent)
$3 = (void *) 0x61ef90
(gdb) x/64wx 0x61ef90
0x61ef90:       0x000015cf      0x02000102      0xff000000      0x00000000 <--- 
failed
0x61efa0:       0xf7a00000      0x00000000      0x00000000      0x00000000
0x61efb0:       0x00000000      0x00000000      0x00000001      0x00000000
0x61efc0:       0x00000000      0x00000000      0x00000000      0x00000110
0x61efd0:       0x00000000      0x00000000      0x00000000      0x00000000
0x61efe0:       0x00000000      0x00000000      0x00000000      0x00000000
0x61eff0:       0x00000000      0x00000000      0x00000000      0x00000000
0x61f000:       0x00000000      0x00000000      0x00000000      0x00000000
0x61f010:       0x00000000      0x00000000      0x00000000      0x00000000
0x61f020:       0x00000000      0x00000000      0x00000000      0x00000000
0x61f030:       0x00000000      0x00000000      0x00000000      0x00000000
0x61f040:       0x00000000      0x00000000      0x00000000      0x00000000
0x61f050:       0x00000000      0x00000000      0x00000000      0x00000000
0x61f060:       0x00000000      0x00000000      0x00000000      0x00000000
0x61f070:       0x00000000      0x00000000      0x00000000      0x00000000
0x61f080:       0x00000000      0x00000000      0x00000000      0x00000000



[ 1466.964782 <  139.522689>] xen-pciback: 0000:06:00.0: write request 4 
bytes at 0x0 = 15cf
[ 1466.964786 <    0.000004>] xen-pciback: 0000:06:00.0: read 2 bytes at 
0x0
[ 1466.964799 <    0.000013>] xen-pciback: 0000:06:00.0: read 2 bytes at 
0x0 = 15cf
[ 1466.964801 <    0.000002>] xen-pciback: 0000:06:00.0: read 2 bytes at 
0x2
[ 1466.964807 <    0.000006>] xen-pciback: 0000:06:00.0: read 2 bytes at 
0x2 = 0
[ 1466.964828 <    0.000021>] xen-pciback: 0000:06:00.0: write request 4 
bytes at 0x4 = 2000102
[ 1466.964829 <    0.000001>] xen-pciback: 0000:06:00.0: read 2 bytes at 
0x4
[ 1466.964841 <    0.000012>] xen-pciback: 0000:06:00.0: read 2 bytes at 
0x4 = 102
[ 1466.964870 <    0.000029>] xen-pciback: 0000:06:00.0: write request 4 
bytes at 0x8 = ff000000
[ 1466.964872 <    0.000002>] xen-pciback: 0000:06:00.0: inside 
permissive mode - write request 4 bytes at 0x8 = ff000000
[ 1466.964895 <    0.000023>] xen-pciback: 0000:06:00.0: write request 4 
bytes at 0xc = 4000
[ 1466.964897 <    0.000002>] xen-pciback: 0000:06:00.0: read 1 bytes at 
0xc
[ 1466.964907 <    0.000010>] xen-pciback: 0000:06:00.0: read 1 bytes at 
0xc = 0
[ 1466.964914 <    0.000007>] xen-pciback: 0000:06:00.0: read 1 bytes at 
0xf
[ 1466.964925 <    0.000011>] xen-pciback: 0000:06:00.0: read 1 bytes at 
0xf = 0
[ 1466.964947 <    0.000022>] xen-pciback: 0000:06:00.0: write request 4 
bytes at 0x10 = f7a00000
[ 1466.964948 <    0.000001>] xen-pciback: 0000:06:00.0: read 4 bytes at 
0x10
[ 1466.964955 <    0.000007>] xen-pciback: 0000:06:00.0: read 4 bytes at 
0x10 = f7a00000
[ 1466.964972 <    0.000017>] xen-pciback: 0000:06:00.0: write request 4 
bytes at 0x14 = 0
[ 1466.964973 <    0.000001>] xen-pciback: 0000:06:00.0: read 4 bytes at 
0x14
[ 1466.964980 <    0.000007>] xen-pciback: 0000:06:00.0: read 4 bytes at 
0x14 = 0
[ 1466.964998 <    0.000018>] xen-pciback: 0000:06:00.0: write request 4 
bytes at 0x18 = 0
[ 1466.964999 <    0.000001>] xen-pciback: 0000:06:00.0: read 4 bytes at 
0x18
[ 1466.965006 <    0.000007>] xen-pciback: 0000:06:00.0: read 4 bytes at 
0x18 = 0

> > - on the DomU - when I run that "test console" tool, the "lspci -xxx"
> > output is different from before/after
> > not much, though, only one register(?)
> >
> > diff lspci.before-testconsole lspci.after-testconsole
> > 2c2
> > < 00: cf 15 00 00 02 01 00 02 00 00 00 ff 00 40 00 00
> > ---
> > > 00: cf 15 00 00 02 01 00 02 00 00 00 ff 00 00 00 00
> Okay, that's the Latency Timer field, and I think just like BARs we
> may need to permit restoring this field. However, please note that
> the reset being done behind the back of pciback really is the
> problem here: pciback assumes (for a reason, as you can certainly
> understand) that it has full control over config space of a passed
> through device.
I agree that it is very ugly to reset the device behind the back of 
pciback.
However, this is how the vendor decided to do it, and have no better 
idea how
to accomplish this.
While I cannot image that there is, but can you think think of
any "correct" way to reset a pci device from
        a) inside a DomU
        b) from the user space library?

What I cannot do: modify the firmware or the HW of the PCI device
(which would be good, because then it could simply do
all memory cleaning etc. w/o affecting its PCI config known to the host)

What I can do: rewrite all parts of the driver (uio_netx) and user space 
library.
> And actually the latency timer would, as a side effect of enabling
> bus mastering on the device (via the pci_set_master() call from
> command_write()) set the Latency Timer field properly, just that
> again pciback (and the rest of Dom0's PCI subsystem) thinks that
> bus mastering is already enabled on the device. So perhaps in
> permissive mode we should simply allow the latency timer field to
> be written, just like we allow writing various of the Command
> Register bits in that mode. Maintainers, what do you think?
(is that something that /sys/bus/pci/drivers/pciback/quirks could
help with? according to docs, this is only used/in effect when
in permissive mode)

> If we decide to go that route, I would then wonder whether
> Cache Line Size being unconditionally writable right now would
> also better be restricted to permissive mode.
> In any event, Jürgen, it would be helpful if you could confirm
If I understood you correctly, I modify xen_pcibk_config_write
in "drivers/xen/xen-pciback/conf_space.c" to specifically allow only
this field (latency timer) to be written and log everything.

Will do and report back!

Thanks for your time!!!

Jürgen

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel