[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [BUG]Add PCIE devie to SMMUv3 fail




在 2022/12/15 09:30, sisyphean 写道:
Hi,

在 2022/12/14 21:05, Rahul Singh 写道:
Hi Sisphean,

On 13 Dec 2022, at 1:18 am, sisyphean <sisyphean@zlw.email> wrote:

Hi,

在 2022/12/13 00:30, Rahul Singh 写道:
Hi Sisyphean,

On 12 Dec 2022, at 5:49 am, sisyphean <sisyphean@zlw.email> wrote:

Hi,
在 2022/12/9 17:50, Rahul Singh 写道:
Hi Sisyphean,

On 9 Dec 2022, at 6:15 am, sisyphean <sisyphean@zlw.email> wrote:

Hi,

I try to run XEN on my ARM board(Sorry, for some commercial reasons, I can't tell you on which platform I run XEN)  and add PCIE device passthrough to DomU.But an error
occurred while adding the PCIE device to SMMUv3.
PCI passthrough support is not fully upstream to Xen on ARM. We have working
PCI passthrough branch that you can use to test it.

https://gitlab.com/xen-project/fusa/xen-integration/-/commits/integration/pci-passthrough

Through reading the code and tracing debugging, the error is found in the function arm_smmu_add_device, which will obtain and determine whether the fwspec of the device to be added to SMMU exists.But for the XEN of arm, the fwspec of the device is created and added by judging whether the iommu field exists in the device node when XEN parses the device tree.However, the PCIE device does not appear in the device tree, so there will be no fwspec for all PCIE devices. When attempting to add a PCIE device to
SMMU, a ENODEV error will be returned.
As of now Xen doesn’t support to add PCI device to IOMMU on ARM.
In addition, the code at xen/drivers/passthrough/pci.c also verifies the above view. For PCIE devices, pdev is alloc in function pci_add_device by alloc_pdev.However, the function alloc_pdev does not create and add fwspec to the PCIE device.Therefore, when function pci_add_device executes to iommu_add_device,it will get the error
return of ENODEV.

How can I resolve the above errors?
If you want to test the PCI passthrough please follow below steps.

Xen setup:
     • A checkout of the “integration/pci-passthrough” branch from the  gitlab https://gitlab.com/xen-project/fusa/xen-integration/-/commits/integration/pci-passthrough      • Pass iommu=yes  and pci-passthrough=on to Xen command line to enable PCI passthrough.

  Linux Kernel setup:
       • Some changes are required for the kernel to work with PCI passthrough. First are some configuration options, enable them in kernel config.
          CONFIG_XEN=y
          CONFIG_XEN_BACKEND=y
          CONFIG_XEN_PCIDEV_BACKEND=y
     • Then a patch needs to be applied for enabling the pciback driver. Patch is attached in this email.
  Using PCI passthrough:
       • In order to pass a device to a guest, you first need its PCI address(SBDF). You can either get it from a bare-metal        Linux running on the platform or by having pciutils  installed (if you are using a yocto-based dom0 or have apt available), which provides lspci.         For example, let's pass one ethernet interface to the guest. Running lspci gives us this output (truncated) :
        0000:00:00.0 Host bridge: Ampere Computing, LLC Device e100
        0000:00:01.0 PCI bridge: Ampere Computing, LLC Device e101 (rev 04)         0000:01:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710/X557-AT 10GBASE-T (rev 01)
                [...]
        We will pass one of the ethernet from the PCI network card : 0000:01:00.0 .
       • Add the following line to the guest configuration file :
           pci = ['0000:01:00.0']
       • Run the following command before starting the guest :
            xl pci-assignable-add 0000:01:00.0
     • Start the guest. The network interface should appear as 00:00.0  in the guest and be usable.
  Please let me know if you need more info.



Regards,
Rahul

Thank you for your reply.
After setting XEN and kernel as above, I tried the following two methods to add a PCIE device passthrough: 1. According to your suggestion, use the command xl pci-assignable-add 0002:21:00.0 to set in the Dom0. But in function iommu_do_pci_domctl,  after device_assigned is called, ENODEV error is obtained. 2. Add xen-pciback.hide=(0002:21:00.0) to dom0-bootargs in the device tree, I encountered the same problem as before when initializing the kernel. In function pci_add_device, PCIE devices cannot be added to SMMUv3.
It is hard to find what is happening without logs. Could you please share the Xen and Linux boot logs so that I can
check what is the root cause of this issue.

The kernel version I use is 5.10. Does this have an impact?
I am using the Linux version 5.15.44 but I don’t think is because of Linux.

Regards,
Rahul
The attachment is a log file. I have to delete some content that may involve commercial. Please forgive me.

In addition, I have forgotten to tell you a very important information. The PCIE controller used on my board is DesignWare. I referred to the code of ECAM under XEN and added some support related to DesignWare
(DBI space mapping and PCIE read/write).

The kernel.log file is from Dom0 boot. The PCIE device I trying to assign to domU is working fine in dom0.
What I understood from the logs is you shared the logs for domU boot. Could you please confirm that dom0 booted fine with the PCI passthrough branch and the PCI device you trying to assign to domU is working fine in dom0.

In that case we can confirm that code you added to support DesignWare PCIE controller is fine.

In addition, if needed ,I can pre initialized PCIE controller in the uboot stage, so  I can scan the PCIE device
in the uboot command line.


In order to verify my previous conjecture (there is no fwspec for the PCIE device, which makes it impossible to passthrough), I used the following code to test. The result is that the PCIE device can passthrough to the DomU, but the DomU kernel appears Data Abort when trying to read the PCIE device. This may be because I directly add the fwspec of the PCIE controller to the PCIE device, which leads to memory mapping problems.

diff --git a/xen/drivers/passthrough/device_tree.c b/xen/drivers/passthrough/device_tree.c
index 98f2aa0..28283d2 100644
--- a/xen/drivers/passthrough/device_tree.c
+++ b/xen/drivers/passthrough/device_tree.c
@@ -127,6 +127,7 @@ int iommu_release_dt_devices(struct domain *d)
     return 0;
 }

+struct device *smmu_dev_for_test;
 int iommu_add_dt_device(struct dt_device_node *np)
 {
     const struct iommu_ops *ops = iommu_get_ops();
@@ -167,7 +168,8 @@ int iommu_add_dt_device(struct dt_device_node *np)
         rc = iommu_fwspec_init(dev, &iommu_spec.np->dev);
         if ( rc )
             break;
-
+        else
+            smmu_dev_for_test = &iommu_spec.np->dev;
         /*
          * Provide DT IOMMU specifier which describes the IOMMU master
          * interfaces of that device (device IDs, etc) to the driver.


diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c
index da4ecda..cf8a099 100644
--- a/xen/drivers/passthrough/pci.c
+++ b/xen/drivers/passthrough/pci.c
@@ -1316,6 +1316,8 @@ static int __init setup_dump_pcidevs(void)
 }
 __initcall(setup_dump_pcidevs);

+extern int iommu_fwspec_init(struct device *dev, struct device *iommu_dev);
+extern struct device *smmu_dev_for_test;;
 static int iommu_add_device(struct pci_dev *pdev)
 {
     const struct domain_iommu *hd;
@@ -1331,6 +1333,10 @@ static int iommu_add_device(struct pci_dev *pdev)
     if ( !is_iommu_enabled(pdev->domain) )
         return 0;

+    if (pdev->bus != 0x20)
+    {
+        iommu_fwspec_init(pci_to_dev(pdev), smmu_dev_for_test);
+    }
     rc = hd->platform_ops->add_device(pdev->devfn, pci_to_dev(pdev));
     if ( rc || !pdev->phantom_stride )
         return rc;
@@ -1401,6 +1407,7 @@ static int device_assigned(u16 seg, u8 bus, u8 devfn)

     ASSERT(pcidevs_locked());
     pdev = pci_get_pdev(seg, bus, devfn);
+    printk("%s:%p,%p,%p\n",__FUNCTION__, pdev->domain, hardware_domain, dom_io);

     if ( !pdev )
         rc = -ENODEV;

The pci-passthrough  branch is xen-4.17-unstable, the xen tools does not work properly. I can't use xl create command to create DomU. After the command is executed, it seems
to be suspended, and the kernel and terminal have no output.

I will test the use of pci-passthrough  branch to verify the PCIE passthrough function

Yes, we can try that and use the below branch to check if it works. Below branch supports to add PCI device   during Xen boot as compared to previous branch where Linux inform Xen to add PCI devices.

You need to add "pci-scan=on” in Xen command line to add PCI devices during Xen boot.

https://gitlab.com/xen-project/people/bmarquis/xen-arm-poc/-/tree/poc/pci-passthrough

Regards,
Rahul

Cheers,
--
Sisyphean




 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.