Re: Proposal for virtual IOMMU binding b/w vIOMMU and passthrough devices

[Stefano Stabellini <sstabellini@xxxxxxxxxx>]

On Fri, 28 Oct 2022, Julien Grall wrote:
> On 28/10/2022 14:13, Bertrand Marquis wrote:
> > > On 28 Oct 2022, at 14:06, Julien Grall <julien@xxxxxxx> wrote:
> > > 
> > > Hi Rahul,
> > > 
> > > On 28/10/2022 13:54, Rahul Singh wrote:
> > > > > > > > > For ACPI, I would have expected the information to be found in
> > > > > > > > > the IOREQ.
> > > > > > > > > 
> > > > > > > > > So can you add more context why this is necessary for
> > > > > > > > > everyone?
> > > > > > > > We have information for IOMMU and Master-ID but we don’t have
> > > > > > > > information for linking vMaster-ID to pMaster-ID.
> > > > > > > 
> > > > > > > I am confused. Below, you are making the virtual master ID
> > > > > > > optional. So shouldn't this be mandatory if you really need the
> > > > > > > mapping with the virtual ID?
> > > > > > vMasterID is optional if user knows pMasterID is unique on the
> > > > > > system. But if pMasterId is not unique then user needs to provide
> > > > > > the vMasterID.
> > > > > 
> > > > > So the expectation is the user will be able to know that the pMasterID
> > > > > is uniq. This may be easy with a couple of SMMUs, but if you have 50+
> > > > > (as suggested above). This will become a pain on larger system.
> > > > > 
> > > > > IHMO, it would be much better if we can detect that in libxl (see
> > > > > below).
> > > > We can make the vMasterID compulsory to avoid complexity in libxl to
> > > > solve this
> > > 
> > > In general, complexity in libxl is not too much of problem.

I agree with this and also I agree with Julien's other statement:

"I am strongly in favor of libxl to modify it if it greatly improves the
user experience."

I am always in favor of reducing complexity for the user as they
typically can't deal with tricky details such as MasterIDs. In general,
I think we need more automation with our tooling.

However, it might not be as simple as adding support for automatically
generating IDs in libxl because we have 2 additional cases to support:
1) dom0less
2) statically built guests

For 1) we would need the same support also in Xen? Which means more
complexity in Xen.

2) are guests like Zephyr that consume a device tree at
build time instead of runtime. These guests are built specifically for a
given environment and it is not a problem to rebuild them for every Xen
release.

However I think it is going to be a problem if we have to run libxl to
get the device tree needed for the Zephyr build. That is because it
means that the Zephyr build system would have to learn how to compile
(or crosscompile) libxl in order to retrieve the data needed for its
input. Even for systems based on Yocto (Yocto already knows how to build
libxl) would cause issues because of internal dependencies this would
introduce.

So I think the automatic generation might be best done in another tool.

I think we need something like a script that takes a partial device tree
as input and provides a more detailed partial device tree as output with
the generated IDs.

If we did it that way, we could call the script from libxl, but also we
could call it separately from ImageBuilder for dom0less and Zephyr/Yocto
could also call it.

Basically we make it easier for everyone to use it. The only price to
pay is that it will be a bit less efficient for xl guests (one more
script to fork and exec) but I think is a good compromise.

Another advantage is that in fully static workflows we could call the
script ahead of time (e.g. from Lopper/ImageBuilder) and still have full
knowledge of the device tree of all the guests which is great from a
safety perspective.


> > I am a bit unsure about this strategy.
> > Currently xl has one configuration file where you put all Xen parameters.
> > The device tree is only needed by some guests to have a description of the
> > system they run on.
> > If we change the model and say that Xen configuration parameters are both in
> > the configuration and the device tree, we somehow enforce to have a device
> > tree even though some guests do not need it at all (for example Zephyr).
> 
> I think my approach was misunderstood because there is no change in the
> existing model.
> 
> What I am suggesting is to not introduce iommu_devid_map but instead let libxl
> allocate the virtual Master-ID and create the mapping with the physical
> Master-ID.
>
> Libxl would then update the property "iommus" in the device-tree with the
> allocated virtual Master-ID.
> 
> Each node in the partial device-tree would need to have a property
> to refer to the physical device just so we know how to update the "iommus".
> The list of device passthrough will still be specified in the configuration
> file. IOW, the partial device-tree is not directly involved in the
> configuration of the guest.
> 
> So far, I don't see a particular issue with this approach because the vMaster
> ID algorithm allocation should be generic. But please let me know if you think
> there are bits I am missing.
>
> For everything else we let the user in control (IPA for mapping, virtual 
> interrupt number) and in this case we switch to a model where we
> automatically generated a vMaster ID.

I think this is a great idea, I only suggest that we move the automatic
generation out of libxl (a separate stand-alone script), in another
place that can be more easily reused by multiple projects and different
use-cases.