[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] Re: [Xen-devel] [PATCH v4] PCI back fixes for 3.17.
On Mon, Jul 14, 2014 at 08:24:33PM +0200, Sander Eikelenboom wrote: > > Monday, July 14, 2014, 7:45:25 PM, you wrote: > > > On Mon, Jul 14, 2014 at 07:43:04PM +0200, Sander Eikelenboom wrote: > >> > >> Monday, July 14, 2014, 7:37:53 PM, you wrote: > >> > >> >> >> Ad B) > >> >> >> > >> >> >> root@dom0:~# xl pci-list router > >> >> >> Vdev Device > >> >> >> 05.0 0000:00:1b.0 > >> >> >> > >> >> >> root@dom0:~# xl pci-assignable-list > >> >> >> 0000:02:00.0 > >> >> >> > >> >> >> root@dom0:~# xl pci-detach router 00:1b.0 > >> >> >> dmesg shows: > >> >> >> [ 199.742668] pciback 0000:00:1b.0: restoring config space at > >> >> >> offset 0x10 (was 0x4, writing 0xf7d30004) > >> >> >> [ 199.743527] pciback 0000:00:1b.0: restoring config space at > >> >> >> offset 0xc (was 0x0, writing 0x10) > >> >> >> [ 199.744321] pciback 0000:00:1b.0: restoring config space at > >> >> >> offset 0x4 (was 0x100000, writing 0x100006) > >> >> >> [ 199.757184] xen-pciback pci-1-0: xen_pcibk_xenbus_remove freeing > >> >> >> pdev @ 0xffff8800589fce40 > >> >> >> [ 199.758139] xen-pciback pci-1-0: xen_pcibk_disconnect pdev @ > >> >> >> 0xffff8800589fce40 > >> >> >> [ 199.862595] xen: xen_unregister_device_domain_owner > >> >> >> > >> >> >> xl dmesg shows: > >> >> >> (XEN) [2014-07-14 16:28:29] memory_map:remove: dom1 gfn=f3070 > >> >> >> mfn=f7d30 nr=4 > >> >> >> (XEN) [2014-07-14 16:28:29] io.c:322: d1: unbind: m_gsi=22 g_gsi=36 > >> >> >> dev=00:00.5 intx=0 > >> >> >> (XEN) [2014-07-14 16:28:29] io.c:390: d1 unmap: m_irq=22 dev=00:00.5 > >> >> >> intx=0 > >> >> >> (XEN) [2014-07-14 16:28:29] [VT-D]iommu.c:1579: d1:PCIe: unmap > >> >> >> 0000:00:1b.0 > >> >> >> (XEN) [2014-07-14 16:28:29] [VT-D]iommu.c:1440: d0:PCIe: map > >> >> >> 0000:00:1b.0 > >> >> >> > >> >> >> root@dom0:~# xl pci-list router > >> >> >> root@dom0:~# xl pci-assignable-list > >> >> >> 0000:00:1b.0 > >> >> >> 0000:02:00.0 > >> >> >> > >> >> >> root@dom0:~# xl pci-assignable-remove 00:1b.0 > >> >> >> dmesg shows: > >> >> >> [ 318.827415] xen: xen_unregister_device_domain_owner > >> >> >> [ 318.828771] xen: xen_unregister_device_domain_owner: ENODEV > >> >> >> [ 318.930869] pciback 0000:00:1b.0: restoring config space at > >> >> >> offset 0x10 (was 0x4, writing 0xf7d30004) > >> >> >> [ 318.933435] pciback 0000:00:1b.0: restoring config space at > >> >> >> offset 0xc (was 0x0, writing 0x10) > >> >> >> [ 318.935877] pciback 0000:00:1b.0: restoring config space at > >> >> >> offset 0x4 (was 0x100000, writing 0x100006) > >> >> >> > >> >> >> root@dom0:~# xl pci-list router > >> >> >> root@dom0:~# xl pci-assignable-list > >> >> >> 0000:02:00.0 > >> >> >> > >> >> >> > >> >> > >> >> > And if you do: > >> >> > >> >> > # xl pci-detach router 02:00.0 > >> >> > >> > >> > Err, I meant > >> > # xl pci-assignable-remove 02:00.0 > >> > >> Ah ok .. so that is: > >> remove a device from pciback that has never been assigned to any guest .. > >> > >> will also give that a go .. although that probably won't be a problem. > > > Right. So I think it all works as expected? That is if you have > > two PCI devices assigned to a guest and want to re-use them you > > have to do the 'pci-detach' twice and then follow that with > > 'pci-assignable-remove' twice as well? > > Well except that's not what i want :-) .. that's about the same as shutting > down > the guest .. and indeed that works :-) > > What i want to do is to remove just *one* device and leave the other one in > the guest. > > The one that i remove .. i would like to be able to remove it from being > assignable .. (rebind it in dom0) and/or assign it to another guest. Aha > > And that fails because pciback still thinks the guest owns the device .. > while libxl thinks it doesn't (outcome of xl pci-assignable-list). Oh > > When doing the: > > # xl pci-assignable-list > 0000:02:00.0 > # xl pci-assignable-remove 02:00.0 > dmesg shows: > [ 443.292951] xen: xen_unregister_device_domain_owner > [ 443.294308] xen: xen_unregister_device_domain_owner: ENODEV > [ 443.398827] pciback 0000:02:00.0: restoring config space at offset 0x3c > (was 0x100, writing 0x104) > [ 443.400797] pciback 0000:02:00.0: restoring config space at offset 0x10 > (was 0x4, writing 0xf7c00004) > [ 443.401713] pciback 0000:02:00.0: restoring config space at offset 0x4 > (was 0x100000, writing 0x100006) > > which i think is what is expected .. Right. > > > Hmm /sys/bus/pci/drivers/pciback doesn't shed much light, because in the > state > just after a xl pci-detach, the device is still registred by pciback (from > dom0's > view) which is correct, what is wrong is that pciback still seems to think > that > it is in turn still owned by the guest. That doesn't get reflected in the > info > available in the /sys dir. Oh > > Ok over to xen-store ... I think I had a patch for that in libxl. It was the order of operations it was doing - it did the 'unbind' first and then it did the XenBus operations. > > first a diff between the booted guest with two devices passed through and the > state after the "xl pci-detach router 00:1b.0" > > root@creanuc:~# diff -U10 xs-before-detach-first.txt xs-after-detach-first.txt > --- xs-before-detach-first.txt 2014-07-14 19:56:23.233576000 +0200 > +++ xs-after-detach-first.txt 2014-07-14 20:00:00.053576000 +0200 > @@ -185,35 +185,29 @@ > feature-rx-flip = "0" > feature-split-event-channels = "1" > multi-queue-max-queues = "4" > hotplug-status = "connected" > pci = "" > 1 = "" > 0 = "" > frontend = "/local/domain/1/device/pci/0" > frontend-id = "1" > online = "1" > - state = "3" > + state = "7" > domain = "router" > key-0 = "0000:02:00.0" > dev-0 = "0000:02:00.0" > vdevfn-0 = "28" > opts-0 = "msitranslate=0,power_mgmt=0,permissive=0" > state-0 = "3" > - key-1 = "0000:00:1b.0" > - dev-1 = "0000:00:1b.0" > - vdevfn-1 = "30" > - opts-1 = "msitranslate=0,power_mgmt=0,permissive=0" > - state-1 = "3" > - num_devs = "2" > + num_devs = "1" > vdev-0 = "0000:00:00.00" > - vdev-1 = "0000:00:01.00" > root-0 = "0000:00" > root_num = "1" > 1 = "" > vm = "/vm/a102d97e-8388-414e-97e5-cc394656c47d" > name = "router" > cpu = "" > 0 = "" > availability = "online" > 1 = "" > availability = "online" > > It seems ok, perhaps apart from the general pci state going from 3 to 7 ? > Haven't looked up the exact states, but since only one domain is removes i > would > still expect it to be 3 ? 7 is Reconfiguring which is what it should move to (to update the XenBus). > > > Now a diff before and after the "xl pci-assignable-remove 00:1b.0" > diff xs-after-detach-first.txt xs-after-pci-assignable-remove-first.txt is > empty, as expected since this is a pciback only thing now involving any guest. > > > And now the last diff, between the state before and after the "xl pci-detach > 02:00.0" removing the last pci device from the guest. > > root@creanuc:~# diff -U5 xs-after-pci-assignable-remove-first.txt > xs-after-pci-detach-second.txt > --- xs-after-pci-assignable-remove-first.txt 2014-07-14 20:00:45.161576000 > +0200 > +++ xs-after-pci-detach-second.txt 2014-07-14 20:01:20.177576000 +0200 > @@ -184,27 +184,10 @@ > feature-rx-copy = "1" > feature-rx-flip = "0" > feature-split-event-channels = "1" > multi-queue-max-queues = "4" > hotplug-status = "connected" > - pci = "" > - 1 = "" > - 0 = "" > - frontend = "/local/domain/1/device/pci/0" > - frontend-id = "1" > - online = "1" > - state = "7" > - domain = "router" > - key-0 = "0000:02:00.0" > - dev-0 = "0000:02:00.0" > - vdevfn-0 = "28" > - opts-0 = "msitranslate=0,power_mgmt=0,permissive=0" > - state-0 = "3" > - num_devs = "1" > - vdev-0 = "0000:00:00.00" > - root-0 = "0000:00" > - root_num = "1" > 1 = "" > vm = "/vm/a102d97e-8388-414e-97e5-cc394656c47d" > name = "router" > cpu = "" > 0 = "" > @@ -249,15 +232,10 @@ > event-channel-rx = "25" > request-rx-copy = "1" > feature-rx-notify = "1" > feature-sg = "1" > feature-gso-tcpv4 = "1" > - pci = "" > - 0 = "" > - backend = "/local/domain/0/backend/pci/1/0" > - backend-id = "0" > - state = "1" > control = "" > shutdown = "" > platform-feature-multiprocessor-suspend = "1" > platform-feature-xs_reset_watches = "1" > hvmloader = "" > > That seems ok. > > So apart from that perhaps incorrect general pci state change, the info in > xenstore seems to be OK. > > So it does seem to be a lack of pciback receiving a confirm from libxl / qemu > that the > device is indeed removed from the guest for the final deregistring of the > ownership. > > Because after a pci-detach the pci device is: > - removed from xen-store as attached to the guest > - the iommu has assigned/mapped it back to dom0 > - xl pci-assignable-list shows it as assignable again > > So only pciback is the only one left out of the party :-) > Couldn't it be somehow tied to the iommu events, since it should be in sync > with that i suppose ? Let me find out what is up with this. Once the device is de-assigned from a guest (even if forcefully) it should be possible to assign it to another. > > -- > Sander > > BTW, i don't see the "do_flr" anywhere in /sys/*/pciback/ ? We been having a lengthy discussion about this and David is out on vacation so will have to wait until he is back. > Attachment:
libxl_pci.patch _______________________________________________ Xen-devel mailing list Xen-devel@xxxxxxxxxxxxx http://lists.xen.org/xen-devel
|
Lists.xenproject.org is hosted with RackSpace, monitoring our |