Re: Proposal for virtual IOMMU binding b/w vIOMMU and passthrough devices

[Ayan Kumar Halder <ayankuma@xxxxxxx>]

On 26/10/2022 14:17, Rahul Singh wrote:
> CAUTION: This message has originated from an External Source. Please use proper 
> judgment and caution when opening attachments, clicking links, or responding to 
> this email.
>
>
> Hi All,
Hi Rahul,

I have a very basic question.

> At Arm, we started to implement the POC to support 2 levels of page 
> tables/nested translation in SMMUv3.
> To support nested translation for guest OS Xen needs to expose the virtual 
> IOMMU. If we passthrough the
> device to the guest that is behind an IOMMU and virtual IOMMU is enabled for 
> the guest there is a need to
> add IOMMU binding for the device in the passthrough node as per [1]. This email 
> is to get an agreement on
> how to add the IOMMU binding for guest OS.
>
> Before I will explain how to add the IOMMU binding let me give a brief overview 
> of how we will add support for virtual
> IOMMU on Arm. In order to implement virtual IOMMU Xen need SMMUv3 Nested 
> translation support. SMMUv3 hardware
> supports two stages of translation. Each stage of translation can be 
> independently enabled. An incoming address is logically
> translated from VA to IPA in stage 1, then the IPA is input to stage 2 which 
> translates the IPA to the output PA. Stage 1 is
> intended to be used by a software entity( Guest OS) to provide isolation or 
> translation to buffers within the entity, for example,
> DMA isolation within an OS. Stage 2 is intended to be available in systems 
> supporting the Virtualization Extensions and is
> intended to virtualize device DMA to guest VM address spaces. When both stage 1 
> and stage 2 are enabled, the translation
> configuration is called nesting.
>
> Stage 1 translation support is required to provide isolation between different 
> devices within the guest OS. XEN already supports
Doesn't this imply that there is support for Stage 1 translation for 
guests ? Otherwise, how will the guest provide isolation between 
different devices or dma-masters ?
- Ayan

> Stage 2 translation but there is no support for Stage 1 translation for guests.
> We will add support for guests to configure
> the Stage 1 transition via virtual IOMMU. XEN will emulate the SMMU hardware 
> and exposes the virtual SMMU to the guest.
> Guest can use the native SMMU driver to configure the stage 1 translation. When 
> the guest configures the SMMU for Stage 1,
> XEN will trap the access and configure the hardware accordingly.
>
> Now back to the question of how we can add the IOMMU binding between the 
> virtual IOMMU and the master devices so that
> guests can configure the IOMMU correctly. The solution that I am suggesting is 
> as below:
>
> For dom0, while handling the DT node(handle_node()) Xen will replace the phandle in the 
> "iommus" property with the virtual
> IOMMU node phandle.
>
> For domU guests, when passthrough the device to the guest as per [2],  add the 
> below property in the partial device tree
> node that is required to describe the generic device tree binding for IOMMUs 
> and their master(s)
>
> "iommus = < &magic_phandle 0xvMasterID>
>          • magic_phandle will be the phandle ( vIOMMU phandle in xl)  that will 
> be documented so that the user can set that in partial DT node (0xfdea).
>          • vMasterID will be the virtual master ID that the user will provide.
>
> The partial device tree will look like this:
> /dts-v1/;
>
> / {
>      /* #*cells are here to keep DTC happy */
>      #address-cells = <2>;
>      #size-cells = <2>;
>
>      aliases {
>          net = &mac0;
>      };
>
>      passthrough {
>          compatible = "simple-bus";
>          ranges;
>          #address-cells = <2>;
>          #size-cells = <2>;
>          mac0: ethernet@10000000 {
>              compatible = "calxeda,hb-xgmac";
>              reg = <0 0x10000000 0 0x1000>;
>              interrupts = <0 80 4  0 81 4  0 82 4>;
>             iommus = <0xfdea 0x01>;
>          };
>      };
> };
>
> In xl.cfg we need to define a new option to inform Xen about vMasterId to 
> pMasterId mapping and to which IOMMU device this
> the master device is connected so that Xen can configure the right IOMMU. This 
> is required if the system has devices that have
> the same master ID but behind a different IOMMU.
>
> iommu_devid_map = [ “PMASTER_ID[@VMASTER_ID],IOMMU_BASE_ADDRESS” , 
> “PMASTER_ID[@VMASTER_ID],IOMMU_BASE_ADDRESS”]
>
>          • PMASTER_ID is the physical master ID of the device from the physical 
> DT.
>          • VMASTER_ID is the virtual master Id that the user will configure in 
> the partial device tree.
>          • IOMMU_BASE_ADDRESS is the base address of the physical IOMMU device 
> to which this device is connected.
>
> Example: Let's say the user wants to assign the below physical device in DT to 
> the guest.
>
> iommu@4f000000 {
>                  compatible = "arm,smmu-v3";
>                  interrupts = <0x00 0xe4 0xf04>;
>                  interrupt-parent = <0x01>;
>                  #iommu-cells = <0x01>;
>                  interrupt-names = "combined";
>                  reg = <0x00 0x4f000000 0x00 0x40000>;
>                  phandle = <0xfdeb>;
>                  name = "iommu";
> };
>
> test@10000000 {
>          compatible = "viommu-test”;
>          iommus = <0xfdeb 0x10>;
>          interrupts = <0x00 0xff 0x04>;
>          reg = <0x00 0x10000000 0x00 0x1000>;
>          name = "viommu-test";
> };
>
> The partial Device tree node will be like this:
>
> / {
>      /* #*cells are here to keep DTC happy */
>      #address-cells = <2>;
>      #size-cells = <2>;
>
>      passthrough {
>          compatible = "simple-bus";
>          ranges;
>          #address-cells = <2>;
>          #size-cells = <2>;
>
>          test@10000000 {
>                  compatible = "viommu-test";
>                  reg = <0 0x10000000 0 0x1000>;
>                  interrupts = <0 80 4  0 81 4  0 82 4>;
>                  iommus = <0xfdea 0x01>;
>          };
>      };
> };
>
>   iommu_devid_map = [ “0x10@0x01,0x4f000000”]
>          • 0x10 is the real physical master id from the physical DT.
>          • 0x01 is the virtual master Id that the user defines as a partial 
> device tree.
>          • 0x4f000000 is the base address of the IOMMU device.
>
> [1] https://www.kernel.org/doc/Documentation/devicetree/bindings/iommu/iommu.txt
> [2] https://xenbits.xen.org/docs/unstable/misc/arm/passthrough.txt
>
> Regards,
> Rahul