Re: Proposal for virtual IOMMU binding b/w vIOMMU and passthrough devices

[Julien Grall <julien@xxxxxxx>]

On 26/10/2022 15:33, Rahul Singh wrote:
> Hi Julien,
Hi Rahul,

> > On 26 Oct 2022, at 2:36 pm, Julien Grall <julien@xxxxxxx> wrote:
> >
> >
> >
> > On 26/10/2022 14:17, Rahul Singh wrote:
> > > Hi All,
> > Hi Rahul,
> >
> > > At Arm, we started to implement the POC to support 2 levels of page 
> > > tables/nested translation in SMMUv3.
> > > To support nested translation for guest OS Xen needs to expose the virtual 
> > > IOMMU. If we passthrough the
> > > device to the guest that is behind an IOMMU and virtual IOMMU is enabled for 
> > > the guest there is a need to
> > > add IOMMU binding for the device in the passthrough node as per [1]. This email 
> > > is to get an agreement on
> > > how to add the IOMMU binding for guest OS.
> > > Before I will explain how to add the IOMMU binding let me give a brief overview 
> > > of how we will add support for virtual
> > > IOMMU on Arm. In order to implement virtual IOMMU Xen need SMMUv3 Nested 
> > > translation support. SMMUv3 hardware
> > > supports two stages of translation. Each stage of translation can be 
> > > independently enabled. An incoming address is logically
> > > translated from VA to IPA in stage 1, then the IPA is input to stage 2 which 
> > > translates the IPA to the output PA. Stage 1 is
> > > intended to be used by a software entity( Guest OS) to provide isolation or 
> > > translation to buffers within the entity, for example,
> > > DMA isolation within an OS. Stage 2 is intended to be available in systems 
> > > supporting the Virtualization Extensions and is
> > > intended to virtualize device DMA to guest VM address spaces. When both stage 1 
> > > and stage 2 are enabled, the translation
> > > configuration is called nesting.
> > > Stage 1 translation support is required to provide isolation between different 
> > > devices within the guest OS. XEN already supports
> > > Stage 2 translation but there is no support for Stage 1 translation for guests. 
> > > We will add support for guests to configure
> > > the Stage 1 transition via virtual IOMMU. XEN will emulate the SMMU hardware 
> > > and exposes the virtual SMMU to the guest.
> > > Guest can use the native SMMU driver to configure the stage 1 translation. When 
> > > the guest configures the SMMU for Stage 1,
> > > XEN will trap the access and configure the hardware accordingly.
> > > Now back to the question of how we can add the IOMMU binding between the 
> > > virtual IOMMU and the master devices so that
> > > guests can configure the IOMMU correctly. The solution that I am suggesting is 
> > > as below:
> > > For dom0, while handling the DT node(handle_node()) Xen will replace the phandle in the 
> > > "iommus" property with the virtual
> > > IOMMU node phandle.
> > Below, you said that each IOMMUs may have a different ID space. So shouldn't we 
> > expose one vIOMMU per pIOMMU? If not, how do you expect the user to specify the 
> > mapping?
> Yes you are right we need to create one vIOMMU per pIOMMU for dom0. This also 
> helps in the ACPI case
> where we don’t need to modify the tables to delete the pIOMMU entries and 
> create one vIOMMU.
> In this case, no need to replace the phandle as Xen create the vIOMMU with the 
> same pIOMMU
> phandle and same base address.
>
> For domU guests one vIOMMU per guest will be created.
IIRC, the SMMUv3 is using a ring like the GICv3 ITS. I think we need to 
be open here because this may end up to be tricky to security support it 
(we have N guest ring that can write to M host ring).
> > > For domU guests, when passthrough the device to the guest as per [2],  add the 
> > > below property in the partial device tree
> > > node that is required to describe the generic device tree binding for IOMMUs 
> > > and their master(s)
> > > "iommus = < &magic_phandle 0xvMasterID>
> > >         • magic_phandle will be the phandle ( vIOMMU phandle in xl)  that will 
> > > be documented so that the user can set that in partial DT node (0xfdea).
> > Does this mean only one IOMMU will be supported in the guest?
> Yes.
>
> > >         • vMasterID will be the virtual master ID that the user will provide.
> > > The partial device tree will look like this:
> > > /dts-v1/;
> > >   / {
> > >      /* #*cells are here to keep DTC happy */
> > >      #address-cells = <2>;
> > >      #size-cells = <2>;
> > >        aliases {
> > >          net = &mac0;
> > >      };
> > >        passthrough {
> > >          compatible = "simple-bus";
> > >          ranges;
> > >          #address-cells = <2>;
> > >          #size-cells = <2>;
> > >          mac0: ethernet@10000000 {
> > >              compatible = "calxeda,hb-xgmac";
> > >              reg = <0 0x10000000 0 0x1000>;
> > >              interrupts = <0 80 4  0 81 4  0 82 4>;
> > >             iommus = <0xfdea 0x01>;
> > >          };
> > >      };
> > > };
> > >   In xl.cfg we need to define a new option to inform Xen about vMasterId to 
> > > pMasterId mapping and to which IOMMU device this
> > > the master device is connected so that Xen can configure the right IOMMU. This 
> > > is required if the system has devices that have
> > > the same master ID but behind a different IOMMU.
> > In xl.cfg, we already pass the device-tree node path to passthrough. So Xen 
> > should already have all the information about the IOMMU and Master-ID. So it 
> > doesn't seem necessary for Device-Tree.
> >
> > For ACPI, I would have expected the information to be found in the IOREQ.
> >
> > So can you add more context why this is necessary for everyone?
> We have information for IOMMU and Master-ID but we don’t have information for 
> linking vMaster-ID to pMaster-ID.
I am confused. Below, you are making the virtual master ID optional. So 
shouldn't this be mandatory if you really need the mapping with the 
virtual ID?
> The device tree node will be used to assign the device to the guest and 
> configure the Stage-2 translation. Guest will use the
> vMaster-ID to configure the vIOMMU during boot. Xen needs information to link 
> vMaster-ID to pMaster-ID to configure
> the corresponding pIOMMU. As I mention we need vMaster-ID in case a system 
> could have 2 identical Master-ID but
> each one connected to a different SMMU and assigned to the guest.
I am afraid I still don't understand why this is a requirement. Libxl 
could have enough knowledge (which will be necessarry for the PCI case) 
to know the IOMMU and pMasterID associated with a device.
So libxl could allocate the vMasterID, tell Xen the corresponding 
mapping and update the device-tree.
IOW, it doesn't seem to be necessary to involve the user in the process 
here.
> > >   iommu_devid_map = [ “PMASTER_ID[@VMASTER_ID],IOMMU_BASE_ADDRESS” , 
> > > “PMASTER_ID[@VMASTER_ID],IOMMU_BASE_ADDRESS”]
> > >         • PMASTER_ID is the physical master ID of the device from the physical 
> > > DT.
> > >         • VMASTER_ID is the virtual master Id that the user will configure in 
> > > the partial device tree.
> > >         • IOMMU_BASE_ADDRESS is the base address of the physical IOMMU device 
> > > to which this device is connected.
> > Below you give an example for Platform device. How would that fit in the 
> > context of PCI passthrough?
> In PCI passthrough case, xl will create the "iommu-map" property in vpci host 
> bridge node with phandle to vIOMMU node.
> vSMMUv3 node will be created in xl.
This means that libxl will need to know the associated pMasterID to a 
PCI device. So, I don't understand why you can't do the same for 
platform devices.
> > >   Example: Let's say the user wants to assign the below physical device in DT 
> > > to the guest.
> > >   iommu@4f000000 {
> > >                  compatible = "arm,smmu-v3";
> > >                 interrupts = <0x00 0xe4 0xf04>;
> > >                  interrupt-parent = <0x01>;
> > >                  #iommu-cells = <0x01>;
> > >                  interrupt-names = "combined";
> > >                  reg = <0x00 0x4f000000 0x00 0x40000>;
> > >                  phandle = <0xfdeb>;
> > >                  name = "iommu";
> > > };
> > So I guess this node will be written by Xen. How will you the case where there 
> > are extra property to added (e.g. dma-coherent)?
> In this example this is physical IOMMU node. vIOMMU node wil be created by xl 
> during guest creation.
Ok. I think it would be better to use very different phandle in your 
example so it doesn't look like a mistake.
Cheers,

--
Julien Grall