|
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] Re: [Xen-devel] [PATCH v4] PCI back fixes for 3.17.
On Mon, Jul 14, 2014 at 09:01:29PM +0200, Sander Eikelenboom wrote:
>
> Monday, July 14, 2014, 8:45:47 PM, you wrote:
>
> > On Mon, Jul 14, 2014 at 08:24:33PM +0200, Sander Eikelenboom wrote:
> >>
> >> Monday, July 14, 2014, 7:45:25 PM, you wrote:
> >>
> >> > On Mon, Jul 14, 2014 at 07:43:04PM +0200, Sander Eikelenboom wrote:
> >> >>
> >> >> Monday, July 14, 2014, 7:37:53 PM, you wrote:
> >> >>
> >> >> >> >> Ad B)
> >> >> >> >>
> >> >> >> >> root@dom0:~# xl pci-list router
> >> >> >> >> Vdev Device
> >> >> >> >> 05.0 0000:00:1b.0
> >> >> >> >>
> >> >> >> >> root@dom0:~# xl pci-assignable-list
> >> >> >> >> 0000:02:00.0
> >> >> >> >>
> >> >> >> >> root@dom0:~# xl pci-detach router 00:1b.0
> >> >> >> >> dmesg shows:
> >> >> >> >> [ 199.742668] pciback 0000:00:1b.0: restoring config space at
> >> >> >> >> offset 0x10 (was 0x4, writing 0xf7d30004)
> >> >> >> >> [ 199.743527] pciback 0000:00:1b.0: restoring config space at
> >> >> >> >> offset 0xc (was 0x0, writing 0x10)
> >> >> >> >> [ 199.744321] pciback 0000:00:1b.0: restoring config space at
> >> >> >> >> offset 0x4 (was 0x100000, writing 0x100006)
> >> >> >> >> [ 199.757184] xen-pciback pci-1-0: xen_pcibk_xenbus_remove
> >> >> >> >> freeing pdev @ 0xffff8800589fce40
> >> >> >> >> [ 199.758139] xen-pciback pci-1-0: xen_pcibk_disconnect pdev @
> >> >> >> >> 0xffff8800589fce40
> >> >> >> >> [ 199.862595] xen: xen_unregister_device_domain_owner
> >> >> >> >>
> >> >> >> >> xl dmesg shows:
> >> >> >> >> (XEN) [2014-07-14 16:28:29] memory_map:remove: dom1 gfn=f3070
> >> >> >> >> mfn=f7d30 nr=4
> >> >> >> >> (XEN) [2014-07-14 16:28:29] io.c:322: d1: unbind: m_gsi=22
> >> >> >> >> g_gsi=36 dev=00:00.5 intx=0
> >> >> >> >> (XEN) [2014-07-14 16:28:29] io.c:390: d1 unmap: m_irq=22
> >> >> >> >> dev=00:00.5 intx=0
> >> >> >> >> (XEN) [2014-07-14 16:28:29] [VT-D]iommu.c:1579: d1:PCIe: unmap
> >> >> >> >> 0000:00:1b.0
> >> >> >> >> (XEN) [2014-07-14 16:28:29] [VT-D]iommu.c:1440: d0:PCIe: map
> >> >> >> >> 0000:00:1b.0
> >> >> >> >>
> >> >> >> >> root@dom0:~# xl pci-list router
> >> >> >> >> root@dom0:~# xl pci-assignable-list
> >> >> >> >> 0000:00:1b.0
> >> >> >> >> 0000:02:00.0
> >> >> >> >>
> >> >> >> >> root@dom0:~# xl pci-assignable-remove 00:1b.0
> >> >> >> >> dmesg shows:
> >> >> >> >> [ 318.827415] xen: xen_unregister_device_domain_owner
> >> >> >> >> [ 318.828771] xen: xen_unregister_device_domain_owner: ENODEV
> >> >> >> >> [ 318.930869] pciback 0000:00:1b.0: restoring config space at
> >> >> >> >> offset 0x10 (was 0x4, writing 0xf7d30004)
> >> >> >> >> [ 318.933435] pciback 0000:00:1b.0: restoring config space at
> >> >> >> >> offset 0xc (was 0x0, writing 0x10)
> >> >> >> >> [ 318.935877] pciback 0000:00:1b.0: restoring config space at
> >> >> >> >> offset 0x4 (was 0x100000, writing 0x100006)
> >> >> >> >>
> >> >> >> >> root@dom0:~# xl pci-list router
> >> >> >> >> root@dom0:~# xl pci-assignable-list
> >> >> >> >> 0000:02:00.0
> >> >> >> >>
> >> >> >> >>
> >> >> >>
> >> >> >> > And if you do:
> >> >> >>
> >> >> >> > # xl pci-detach router 02:00.0
> >> >> >>
> >> >>
> >> >> > Err, I meant
> >> >> > # xl pci-assignable-remove 02:00.0
> >> >>
> >> >> Ah ok .. so that is:
> >> >> remove a device from pciback that has never been assigned to any guest
> >> >> ..
> >> >>
> >> >> will also give that a go .. although that probably won't be a problem.
> >>
> >> > Right. So I think it all works as expected? That is if you have
> >> > two PCI devices assigned to a guest and want to re-use them you
> >> > have to do the 'pci-detach' twice and then follow that with
> >> > 'pci-assignable-remove' twice as well?
> >>
> >> Well except that's not what i want :-) .. that's about the same as
> >> shutting down
> >> the guest .. and indeed that works :-)
> >>
> >> What i want to do is to remove just *one* device and leave the other one
> >> in the guest.
> >>
> >> The one that i remove .. i would like to be able to remove it from being
> >> assignable .. (rebind it in dom0) and/or assign it to another guest.
>
> > Aha
> >>
> >> And that fails because pciback still thinks the guest owns the device ..
> >> while libxl thinks it doesn't (outcome of xl pci-assignable-list).
>
> > Oh
> >>
> >> When doing the:
> >>
> >> # xl pci-assignable-list
> >> 0000:02:00.0
> >> # xl pci-assignable-remove 02:00.0
> >> dmesg shows:
> >> [ 443.292951] xen: xen_unregister_device_domain_owner
> >> [ 443.294308] xen: xen_unregister_device_domain_owner: ENODEV
> >> [ 443.398827] pciback 0000:02:00.0: restoring config space at offset 0x3c
> >> (was 0x100, writing 0x104)
> >> [ 443.400797] pciback 0000:02:00.0: restoring config space at offset 0x10
> >> (was 0x4, writing 0xf7c00004)
> >> [ 443.401713] pciback 0000:02:00.0: restoring config space at offset 0x4
> >> (was 0x100000, writing 0x100006)
> >>
> >> which i think is what is expected ..
>
> > Right.
>
> >>
> >>
> >> Hmm /sys/bus/pci/drivers/pciback doesn't shed much light, because in the
> >> state
> >> just after a xl pci-detach, the device is still registred by pciback (from
> >> dom0's
> >> view) which is correct, what is wrong is that pciback still seems to think
> >> that
> >> it is in turn still owned by the guest. That doesn't get reflected in the
> >> info
> >> available in the /sys dir.
>
> > Oh
> >>
> >> Ok over to xen-store ...
>
> > I think I had a patch for that in libxl. It was the order of operations it
> > was
> > doing - it did the 'unbind' first and then it did the XenBus operations.
>
> >>
> >> first a diff between the booted guest with two devices passed through and
> >> the
> >> state after the "xl pci-detach router 00:1b.0"
> >>
> >> root@creanuc:~# diff -U10 xs-before-detach-first.txt
> >> xs-after-detach-first.txt
> >> --- xs-before-detach-first.txt 2014-07-14 19:56:23.233576000 +0200
> >> +++ xs-after-detach-first.txt 2014-07-14 20:00:00.053576000 +0200
> >> @@ -185,35 +185,29 @@
> >> feature-rx-flip = "0"
> >> feature-split-event-channels = "1"
> >> multi-queue-max-queues = "4"
> >> hotplug-status = "connected"
> >> pci = ""
> >> 1 = ""
> >> 0 = ""
> >> frontend = "/local/domain/1/device/pci/0"
> >> frontend-id = "1"
> >> online = "1"
> >> - state = "3"
> >> + state = "7"
> >> domain = "router"
> >> key-0 = "0000:02:00.0"
> >> dev-0 = "0000:02:00.0"
> >> vdevfn-0 = "28"
> >> opts-0 = "msitranslate=0,power_mgmt=0,permissive=0"
> >> state-0 = "3"
> >> - key-1 = "0000:00:1b.0"
> >> - dev-1 = "0000:00:1b.0"
> >> - vdevfn-1 = "30"
> >> - opts-1 = "msitranslate=0,power_mgmt=0,permissive=0"
> >> - state-1 = "3"
> >> - num_devs = "2"
> >> + num_devs = "1"
> >> vdev-0 = "0000:00:00.00"
> >> - vdev-1 = "0000:00:01.00"
> >> root-0 = "0000:00"
> >> root_num = "1"
> >> 1 = ""
> >> vm = "/vm/a102d97e-8388-414e-97e5-cc394656c47d"
> >> name = "router"
> >> cpu = ""
> >> 0 = ""
> >> availability = "online"
> >> 1 = ""
> >> availability = "online"
> >>
> >> It seems ok, perhaps apart from the general pci state going from 3 to 7 ?
> >> Haven't looked up the exact states, but since only one domain is removes i
> >> would
> >> still expect it to be 3 ?
>
> > 7 is Reconfiguring which is what it should move to (to update the XenBus).
>
> Ok, but that should than be acknowledged somewhere by someone i suppose ..
> and
> not stay in that state ?
>
> >>
> >>
> >> Now a diff before and after the "xl pci-assignable-remove 00:1b.0"
> >> diff xs-after-detach-first.txt xs-after-pci-assignable-remove-first.txt is
> >> empty, as expected since this is a pciback only thing now involving any
> >> guest.
> >>
> >>
> >> And now the last diff, between the state before and after the "xl
> >> pci-detach
> >> 02:00.0" removing the last pci device from the guest.
> >>
> >> root@creanuc:~# diff -U5 xs-after-pci-assignable-remove-first.txt
> >> xs-after-pci-detach-second.txt
> >> --- xs-after-pci-assignable-remove-first.txt 2014-07-14
> >> 20:00:45.161576000 +0200
> >> +++ xs-after-pci-detach-second.txt 2014-07-14 20:01:20.177576000 +0200
> >> @@ -184,27 +184,10 @@
> >> feature-rx-copy = "1"
> >> feature-rx-flip = "0"
> >> feature-split-event-channels = "1"
> >> multi-queue-max-queues = "4"
> >> hotplug-status = "connected"
> >> - pci = ""
> >> - 1 = ""
> >> - 0 = ""
> >> - frontend = "/local/domain/1/device/pci/0"
> >> - frontend-id = "1"
> >> - online = "1"
> >> - state = "7"
> >> - domain = "router"
> >> - key-0 = "0000:02:00.0"
> >> - dev-0 = "0000:02:00.0"
> >> - vdevfn-0 = "28"
> >> - opts-0 = "msitranslate=0,power_mgmt=0,permissive=0"
> >> - state-0 = "3"
> >> - num_devs = "1"
> >> - vdev-0 = "0000:00:00.00"
> >> - root-0 = "0000:00"
> >> - root_num = "1"
> >> 1 = ""
> >> vm = "/vm/a102d97e-8388-414e-97e5-cc394656c47d"
> >> name = "router"
> >> cpu = ""
> >> 0 = ""
> >> @@ -249,15 +232,10 @@
> >> event-channel-rx = "25"
> >> request-rx-copy = "1"
> >> feature-rx-notify = "1"
> >> feature-sg = "1"
> >> feature-gso-tcpv4 = "1"
> >> - pci = ""
> >> - 0 = ""
> >> - backend = "/local/domain/0/backend/pci/1/0"
> >> - backend-id = "0"
> >> - state = "1"
> >> control = ""
> >> shutdown = ""
> >> platform-feature-multiprocessor-suspend = "1"
> >> platform-feature-xs_reset_watches = "1"
> >> hvmloader = ""
> >>
> >> That seems ok.
> >>
> >> So apart from that perhaps incorrect general pci state change, the info in
> >> xenstore seems to be OK.
> >>
> >> So it does seem to be a lack of pciback receiving a confirm from libxl /
> >> qemu that the
> >> device is indeed removed from the guest for the final deregistring of the
> >> ownership.
> >>
> >> Because after a pci-detach the pci device is:
> >> - removed from xen-store as attached to the guest
> >> - the iommu has assigned/mapped it back to dom0
> >> - xl pci-assignable-list shows it as assignable again
> >>
> >> So only pciback is the only one left out of the party :-)
> >> Couldn't it be somehow tied to the iommu events, since it should be in
> >> sync with that i suppose ?
>
>
> > Let me find out what is up with this. Once the device
> > is de-assigned from a guest (even if forcefully) it should
> > be possible to assign it to another.
>
> Yes .. at least for the rebinding to dom0 case, because the
> pci-assignable-remove seems to do it forcefully now. But i haven't tried what
> it
> does when reassigning to another guest (if it forcefully swaps ownership
> then,
> since the pci-assignable-remove isn't involved then).
>
> Will give the xen patch a try.
I found out what the problem is. It has been in the code for eons and
I think we never tested for it because the older toolstack (xend) would
do things differently.
The issue here is:
1) xl pci-detach <dom-id> <BDF>
'xl' removes the whole device from the XenStore, so from:
domain = "USB"
vdevfn-0 = "28"
opts-0 = "msitranslate=0,power_mgmt=0,permissive=0"
state-0 = "3"
vdev-0 = "0000:00:00.00"
root-0 = "0000:00"
key-0 = "0000:07:00.0"
dev-0 = "0000:07:00.0"
key-1 = "0000:04:00.0"
dev-1 = "0000:04:00.0"
vdevfn-1 = "30"
opts-1 = "msitranslate=0,power_mgmt=0,permissive=0"
state-1 = "3"
num_devs = "2"
vdev-1 = "0000:00:01.00"
root_num = "1"
it just yanks out the '*-1' values, so we have:
domain = "USB"
vdevfn-0 = "28"
opts-0 = "msitranslate=0,power_mgmt=0,permissive=0"
state-0 = "3"
vdev-0 = "0000:00:00.00"
root-0 = "0000:00"
key-0 = "0000:07:00.0"
dev-0 = "0000:07:00.0"
num_devs = "1"
While if the frontend was PV (so xen-pcifront) it would first but
the state-1 to 5 (Closed) which would let the Xen pciback deallocate this.
Anyhow, Xen pciback should be able to deal with this, but the patch
is going to take a bit of time to figure out.
Thank you for reporting it and helping me with the XenStore output to
narrow down the culprit!
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel
|
 |
Lists.xenproject.org is hosted with RackSpace, monitoring our |