Xen project Mailing List

Hello,

I would like to paravirtualize several instances of a minimal Linux solution with xen. Guest OS have a PCI passthrough access on the ethernet interfaces.

I use a pvops kernel 4.4.122 , xen 4.11.2 and an x86 platform. My hardware platform doesn't support IOMMU but I doesn't need hard segmentation between OS with this platform.

I got a bug when the Ethernet driver configures the MSI for a domU from a domU.

pciback 0000:00:14.3: xen map irq failed -19 for 1 domain

Please find the bug description and maybe a solution :

1) dom1 is configured to use 0000:00:14.3

in file /etc/xen/dom1 :

pci=['00:14.3,permissive=1,seize=1,msitranslate=1']

and kernel bootparam include "iommu=soft"

2) xen-pciback is loaded in dom0 :

> modprobe xen-pciback passthrough=1 hide='(0000:00:14.3)'

> xl pci-assignable-list

0000:00:14.3

(xen-pciback is not loaded in dom1)

3) Xen debug terminal print that all the PCI devices are linked to dom0 before and after xen-pciback probing

...

(XEN) 0000:00:14.3 - dom 0 - node -1 - MSIs < >

...

4) no error are detected when I start dom1

> xl create /etc/xen/dom1

or with

> xl pci-attach dom1 00:14.3

5) For each Ethernet device I see the following call to the hypervisor

- xc_domain_iomem_permission

- xc_domain_ioport_permission

- xc_domain_iomem_permission

- xc_physdev_map_pirq

- xc_domain_irq_permission

- xc_assign_device

none of them update the pdev->domain or the pdev->domain->domain_id of the pdev element linked with the bfd 0000:00:14.3 (devfn 163)

6) Xen debug terminal prints again that all the PCI devices are linked to dom0

...

(XEN) 0000:00:14.3 - dom 0 - node -1 - MSIs < >

...

I guess that this is normal because Dom0 will allow dom1 to configure the device (IRQ & DMA) thanks to xen-pciback. Could you confirm me this assertion ?

7) When the Ethernet driver (igb) of dom1 configure MSI. xen failed to map MSI irqs because map_domain_pirq call pci_get_pdev_by_domain with domain_id 1 and the device is linked with domain_id 0

> modprobe igb

igb 0000:00:14.3: enabling device (0000 -> 0002)

igb 0000:00:14.3: Xen PCI mapped GSI23 to IRQ25

en_initdom_setup_msi_irqs:xen map irq map_irq.domid 3, map_irq.type 3 map_irq.index -1 , map_irq.pirq -1

xen_initdom_setup_msi_irqs:xen map irq map_irq.bus 0, map_irq.devfn 163

pciback 0000:00:14.3: PCI_CAP_ID_MSIX

pci_seg_supported

There,HYPERVISOR_physdev_op and PHYSDEVOP_map_pirq call map_domain_pirq :

(XEN) map_domain_pirq--> call pci_get_pdev_by_domain

(XEN) pci_get_pdev_by_domain-->

(XEN) domain->domain_id 1

(XEN) seg 0 bus 0 devfn 163

...

Unfortunately the device is already mapped to domain 0 and pci_get_pdev_by_domain don't found the device ... So map_domain_pirq & HYPERVISOR_physdev_op return -19 :

pciback 0000:00:14.3: xen map irq failed -19 for 2 domain

I don't find any hypervisor syscall able to assign my device to a domU. xc_assign_device is cancelled because iommu is disabled.

libxl_pci.c:1137:do_pci_add: Domain 3:do_pci_add --> call xc_assign_device -> hypervisor call arch_do_domctl--> iommu_do_domctl--> CANCELLED

To solve my issue , I just need to replace in map_domain_pirq in xen/arch/x86/irq.c

pdev = pci_get_pdev_by_domain(d, msi->seg, msi->bus, msi->devfn);

pdev = pci_get_pdev(msi->seg, msi->bus, msi->devfn);

With this "fix" in Xen, dom1 can use the ethernet device and its MSI that I see in /proc/interrupt

I use a customized cross-compilation of xen and xen-tools for a customized mininal linux solution thanks to buildroot. This solution has been built from scratch. So I'm not sure that all the xen components work perfectly. However, everything seem to work with my modifications (CPU, ethernet, memory, disk, interrupt ...)

So is it really a bug ? or just a configuration error ? maybe a parameter has been forgotten in /etc/xen/domU ?

Thank you for your help

[Xen-devel] bug suspcion and proposed modification when xen-pciback failed to map an irq (-19) to a domU