Re: Proposal for virtual IOMMU binding b/w vIOMMU and passthrough devices

[Stefano Stabellini <sstabellini@xxxxxxxxxx>]

On Sun, 30 Oct 2022, Julien Grall wrote:
> Hi Stefano,
> 
> On 30/10/2022 14:23, Stefano Stabellini wrote:
> > On Fri, 28 Oct 2022, Julien Grall wrote:
> > > On 28/10/2022 14:13, Bertrand Marquis wrote:
> > > > > On 28 Oct 2022, at 14:06, Julien Grall <julien@xxxxxxx> wrote:
> > > > > 
> > > > > Hi Rahul,
> > > > > 
> > > > > On 28/10/2022 13:54, Rahul Singh wrote:
> > > > > > > > > > > For ACPI, I would have expected the information to be
> > > > > > > > > > > found in
> > > > > > > > > > > the IOREQ.
> > > > > > > > > > > 
> > > > > > > > > > > So can you add more context why this is necessary for
> > > > > > > > > > > everyone?
> > > > > > > > > > We have information for IOMMU and Master-ID but we don’t
> > > > > > > > > > have
> > > > > > > > > > information for linking vMaster-ID to pMaster-ID.
> > > > > > > > > 
> > > > > > > > > I am confused. Below, you are making the virtual master ID
> > > > > > > > > optional. So shouldn't this be mandatory if you really need
> > > > > > > > > the
> > > > > > > > > mapping with the virtual ID?
> > > > > > > > vMasterID is optional if user knows pMasterID is unique on the
> > > > > > > > system. But if pMasterId is not unique then user needs to
> > > > > > > > provide
> > > > > > > > the vMasterID.
> > > > > > > 
> > > > > > > So the expectation is the user will be able to know that the
> > > > > > > pMasterID
> > > > > > > is uniq. This may be easy with a couple of SMMUs, but if you have
> > > > > > > 50+
> > > > > > > (as suggested above). This will become a pain on larger system.
> > > > > > > 
> > > > > > > IHMO, it would be much better if we can detect that in libxl (see
> > > > > > > below).
> > > > > > We can make the vMasterID compulsory to avoid complexity in libxl to
> > > > > > solve this
> > > > > 
> > > > > In general, complexity in libxl is not too much of problem.
> > 
> > I agree with this and also I agree with Julien's other statement:
> > 
> > "I am strongly in favor of libxl to modify it if it greatly improves the
> > user experience."
> > 
> > I am always in favor of reducing complexity for the user as they
> > typically can't deal with tricky details such as MasterIDs. In general,
> > I think we need more automation with our tooling.
> > 
> > However, it might not be as simple as adding support for automatically
> > generating IDs in libxl because we have 2 additional cases to support:
> > 1) dom0less
> > 2) statically built guests
> > 
> > For 1) we would need the same support also in Xen? Which means more
> > complexity in Xen.
> Xen will need to parse the device-tree to find the mapping. So I am not
> entirely convinced there will be more complexity needed other than requiring a
> bitmap to know which vMasterID has been allocated.
> 
> That said, you would still need one to validate the input provided by the
> user. So overall maybe there will be no added complexity?
> 
> > 
> > 2) are guests like Zephyr that consume a device tree at
> > build time instead of runtime. These guests are built specifically for a
> > given environment and it is not a problem to rebuild them for every Xen
> > release.
> > 
> > However I think it is going to be a problem if we have to run libxl to
> > get the device tree needed for the Zephyr build. That is because it
> > means that the Zephyr build system would have to learn how to compile
> > (or crosscompile) libxl in order to retrieve the data needed for its
> > input. Even for systems based on Yocto (Yocto already knows how to build
> > libxl) would cause issues because of internal dependencies this would
> > introduce.
> 
> That would not be very different to how this works today for Zephyr. They need
> libxl to generate the guest DT.
> 
> That said, I agree this is a bit of a pain...

Yeah..


> > So I think the automatic generation might be best done in another tool.
> It sounds like what you want is creating something similar to libacpi but for
> Device-Tree. That should work with some caveats.

Yes, something like that. We have a framework for reading, editing and
generating Device Tree: Lopper https://github.com/devicetree-org/lopper

It is mostly targeted at build time but it could also be invoked on
target at runtime.

 
> > I think we need something like a script that takes a partial device tree
> > as input and provides a more detailed partial device tree as output with
> > the generated IDs.
> 
> AFAICT, having the partial device-tree is not enough. You also need the real
> DT to figure out the pMaster-ID.
> 
> > 
> > If we did it that way, we could call the script from libxl, but also we
> > could call it separately from ImageBuilder for dom0less and Zephyr/Yocto
> > could also call it.
> > 
> > Basically we make it easier for everyone to use it. The only price to
> > pay is that it will be a bit less efficient for xl guests (one more
> > script to fork and exec) but I think is a good compromise.
> 
> We would need an hypercall to retrieve the host Device-Tree. But that would
> not be too difficult to add.

Good point


> > I think this is a great idea, I only suggest that we move the automatic
> > generation out of libxl (a separate stand-alone script), in another
> > place that can be more easily reused by multiple projects and different
> > use-cases.
> 
> If we use the concept of libacpi, we may not need a to have a stand-alone
> script. It could directly linked in libxl or any other tools.
 
I don't feel strongly whether it should be a library, a script or
something else. My only point is that it should be easy to use both at
build time (e.g. Yocto/Zephyr/ImageBuilder/Lopper) and runtime
(xl/libxl).

We have already a partial DTB generator as a Lopper "lop" (a Lopper
plugin). Probably using Lopper would be the easiest way to implement it,
and the "lop" could be under xen.git (it doesn't have to reside under
the lopper repository).

But if we wanted a library that would be OK too. The issue with libxl is
not much that it is a library but that it is complex to build and has
many dependencies (it can only be built from the top level ./configure
and make).

Ideally this would be something quick that can be easily invoked as the
first step of an external third-party build process.