|
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] Re: [Xen-devel] [PATCH v4] PCI back fixes for 3.17.
Monday, July 14, 2014, 7:45:25 PM, you wrote:
> On Mon, Jul 14, 2014 at 07:43:04PM +0200, Sander Eikelenboom wrote:
>>
>> Monday, July 14, 2014, 7:37:53 PM, you wrote:
>>
>> >> >> Ad B)
>> >> >>
>> >> >> root@dom0:~# xl pci-list router
>> >> >> Vdev Device
>> >> >> 05.0 0000:00:1b.0
>> >> >>
>> >> >> root@dom0:~# xl pci-assignable-list
>> >> >> 0000:02:00.0
>> >> >>
>> >> >> root@dom0:~# xl pci-detach router 00:1b.0
>> >> >> dmesg shows:
>> >> >> [ 199.742668] pciback 0000:00:1b.0: restoring config space at offset
>> >> >> 0x10 (was 0x4, writing 0xf7d30004)
>> >> >> [ 199.743527] pciback 0000:00:1b.0: restoring config space at offset
>> >> >> 0xc (was 0x0, writing 0x10)
>> >> >> [ 199.744321] pciback 0000:00:1b.0: restoring config space at offset
>> >> >> 0x4 (was 0x100000, writing 0x100006)
>> >> >> [ 199.757184] xen-pciback pci-1-0: xen_pcibk_xenbus_remove freeing
>> >> >> pdev @ 0xffff8800589fce40
>> >> >> [ 199.758139] xen-pciback pci-1-0: xen_pcibk_disconnect pdev @
>> >> >> 0xffff8800589fce40
>> >> >> [ 199.862595] xen: xen_unregister_device_domain_owner
>> >> >>
>> >> >> xl dmesg shows:
>> >> >> (XEN) [2014-07-14 16:28:29] memory_map:remove: dom1 gfn=f3070
>> >> >> mfn=f7d30 nr=4
>> >> >> (XEN) [2014-07-14 16:28:29] io.c:322: d1: unbind: m_gsi=22 g_gsi=36
>> >> >> dev=00:00.5 intx=0
>> >> >> (XEN) [2014-07-14 16:28:29] io.c:390: d1 unmap: m_irq=22 dev=00:00.5
>> >> >> intx=0
>> >> >> (XEN) [2014-07-14 16:28:29] [VT-D]iommu.c:1579: d1:PCIe: unmap
>> >> >> 0000:00:1b.0
>> >> >> (XEN) [2014-07-14 16:28:29] [VT-D]iommu.c:1440: d0:PCIe: map
>> >> >> 0000:00:1b.0
>> >> >>
>> >> >> root@dom0:~# xl pci-list router
>> >> >> root@dom0:~# xl pci-assignable-list
>> >> >> 0000:00:1b.0
>> >> >> 0000:02:00.0
>> >> >>
>> >> >> root@dom0:~# xl pci-assignable-remove 00:1b.0
>> >> >> dmesg shows:
>> >> >> [ 318.827415] xen: xen_unregister_device_domain_owner
>> >> >> [ 318.828771] xen: xen_unregister_device_domain_owner: ENODEV
>> >> >> [ 318.930869] pciback 0000:00:1b.0: restoring config space at offset
>> >> >> 0x10 (was 0x4, writing 0xf7d30004)
>> >> >> [ 318.933435] pciback 0000:00:1b.0: restoring config space at offset
>> >> >> 0xc (was 0x0, writing 0x10)
>> >> >> [ 318.935877] pciback 0000:00:1b.0: restoring config space at offset
>> >> >> 0x4 (was 0x100000, writing 0x100006)
>> >> >>
>> >> >> root@dom0:~# xl pci-list router
>> >> >> root@dom0:~# xl pci-assignable-list
>> >> >> 0000:02:00.0
>> >> >>
>> >> >>
>> >>
>> >> > And if you do:
>> >>
>> >> > # xl pci-detach router 02:00.0
>> >>
>>
>> > Err, I meant
>> > # xl pci-assignable-remove 02:00.0
>>
>> Ah ok .. so that is:
>> remove a device from pciback that has never been assigned to any guest ..
>>
>> will also give that a go .. although that probably won't be a problem.
> Right. So I think it all works as expected? That is if you have
> two PCI devices assigned to a guest and want to re-use them you
> have to do the 'pci-detach' twice and then follow that with
> 'pci-assignable-remove' twice as well?
Well except that's not what i want :-) .. that's about the same as shutting
down
the guest .. and indeed that works :-)
What i want to do is to remove just *one* device and leave the other one in the
guest.
The one that i remove .. i would like to be able to remove it from being
assignable .. (rebind it in dom0) and/or assign it to another guest.
And that fails because pciback still thinks the guest owns the device ..
while libxl thinks it doesn't (outcome of xl pci-assignable-list).
When doing the:
# xl pci-assignable-list
0000:02:00.0
# xl pci-assignable-remove 02:00.0
dmesg shows:
[ 443.292951] xen: xen_unregister_device_domain_owner
[ 443.294308] xen: xen_unregister_device_domain_owner: ENODEV
[ 443.398827] pciback 0000:02:00.0: restoring config space at offset 0x3c (was
0x100, writing 0x104)
[ 443.400797] pciback 0000:02:00.0: restoring config space at offset 0x10 (was
0x4, writing 0xf7c00004)
[ 443.401713] pciback 0000:02:00.0: restoring config space at offset 0x4 (was
0x100000, writing 0x100006)
which i think is what is expected ..
Hmm /sys/bus/pci/drivers/pciback doesn't shed much light, because in the
state
just after a xl pci-detach, the device is still registred by pciback (from
dom0's
view) which is correct, what is wrong is that pciback still seems to think that
it is in turn still owned by the guest. That doesn't get reflected in the info
available in the /sys dir.
Ok over to xen-store ...
first a diff between the booted guest with two devices passed through and the
state after the "xl pci-detach router 00:1b.0"
root@creanuc:~# diff -U10 xs-before-detach-first.txt xs-after-detach-first.txt
--- xs-before-detach-first.txt 2014-07-14 19:56:23.233576000 +0200
+++ xs-after-detach-first.txt 2014-07-14 20:00:00.053576000 +0200
@@ -185,35 +185,29 @@
feature-rx-flip = "0"
feature-split-event-channels = "1"
multi-queue-max-queues = "4"
hotplug-status = "connected"
pci = ""
1 = ""
0 = ""
frontend = "/local/domain/1/device/pci/0"
frontend-id = "1"
online = "1"
- state = "3"
+ state = "7"
domain = "router"
key-0 = "0000:02:00.0"
dev-0 = "0000:02:00.0"
vdevfn-0 = "28"
opts-0 = "msitranslate=0,power_mgmt=0,permissive=0"
state-0 = "3"
- key-1 = "0000:00:1b.0"
- dev-1 = "0000:00:1b.0"
- vdevfn-1 = "30"
- opts-1 = "msitranslate=0,power_mgmt=0,permissive=0"
- state-1 = "3"
- num_devs = "2"
+ num_devs = "1"
vdev-0 = "0000:00:00.00"
- vdev-1 = "0000:00:01.00"
root-0 = "0000:00"
root_num = "1"
1 = ""
vm = "/vm/a102d97e-8388-414e-97e5-cc394656c47d"
name = "router"
cpu = ""
0 = ""
availability = "online"
1 = ""
availability = "online"
It seems ok, perhaps apart from the general pci state going from 3 to 7 ?
Haven't looked up the exact states, but since only one domain is removes i
would
still expect it to be 3 ?
Now a diff before and after the "xl pci-assignable-remove 00:1b.0"
diff xs-after-detach-first.txt xs-after-pci-assignable-remove-first.txt is
empty, as expected since this is a pciback only thing now involving any guest.
And now the last diff, between the state before and after the "xl pci-detach
02:00.0" removing the last pci device from the guest.
root@creanuc:~# diff -U5 xs-after-pci-assignable-remove-first.txt
xs-after-pci-detach-second.txt
--- xs-after-pci-assignable-remove-first.txt 2014-07-14 20:00:45.161576000
+0200
+++ xs-after-pci-detach-second.txt 2014-07-14 20:01:20.177576000 +0200
@@ -184,27 +184,10 @@
feature-rx-copy = "1"
feature-rx-flip = "0"
feature-split-event-channels = "1"
multi-queue-max-queues = "4"
hotplug-status = "connected"
- pci = ""
- 1 = ""
- 0 = ""
- frontend = "/local/domain/1/device/pci/0"
- frontend-id = "1"
- online = "1"
- state = "7"
- domain = "router"
- key-0 = "0000:02:00.0"
- dev-0 = "0000:02:00.0"
- vdevfn-0 = "28"
- opts-0 = "msitranslate=0,power_mgmt=0,permissive=0"
- state-0 = "3"
- num_devs = "1"
- vdev-0 = "0000:00:00.00"
- root-0 = "0000:00"
- root_num = "1"
1 = ""
vm = "/vm/a102d97e-8388-414e-97e5-cc394656c47d"
name = "router"
cpu = ""
0 = ""
@@ -249,15 +232,10 @@
event-channel-rx = "25"
request-rx-copy = "1"
feature-rx-notify = "1"
feature-sg = "1"
feature-gso-tcpv4 = "1"
- pci = ""
- 0 = ""
- backend = "/local/domain/0/backend/pci/1/0"
- backend-id = "0"
- state = "1"
control = ""
shutdown = ""
platform-feature-multiprocessor-suspend = "1"
platform-feature-xs_reset_watches = "1"
hvmloader = ""
That seems ok.
So apart from that perhaps incorrect general pci state change, the info in
xenstore seems to be OK.
So it does seem to be a lack of pciback receiving a confirm from libxl / qemu
that the
device is indeed removed from the guest for the final deregistring of the
ownership.
Because after a pci-detach the pci device is:
- removed from xen-store as attached to the guest
- the iommu has assigned/mapped it back to dom0
- xl pci-assignable-list shows it as assignable again
So only pciback is the only one left out of the party :-)
Couldn't it be somehow tied to the iommu events, since it should be in sync
with that i suppose ?
--
Sander
BTW, i don't see the "do_flr" anywhere in /sys/*/pciback/ ?
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel
|
 |
Lists.xenproject.org is hosted with RackSpace, monitoring our |