Re: Proposal for virtual IOMMU binding b/w vIOMMU and passthrough devices

[Bertrand Marquis <Bertrand.Marquis@xxxxxxx>]

Hi Julien,

> On 28 Oct 2022, at 16:01, Julien Grall <julien@xxxxxxx> wrote:
> 
> 
> 
> On 28/10/2022 15:37, Bertrand Marquis wrote:
>> Hi Julien,
> 
> Hi Bertrand,
> 
>>> On 28 Oct 2022, at 14:27, Julien Grall <julien@xxxxxxx> wrote:
>>> 
>>> 
>>> 
>>> On 28/10/2022 14:13, Bertrand Marquis wrote:
>>>> Hi Julien,
>>> 
>>> Hi Bertrand,
>>> 
>>>>> On 28 Oct 2022, at 14:06, Julien Grall <julien@xxxxxxx> wrote:
>>>>> 
>>>>> Hi Rahul,
>>>>> 
>>>>> On 28/10/2022 13:54, Rahul Singh wrote:
>>>>>>>>>>> For ACPI, I would have expected the information to be found in the 
>>>>>>>>>>> IOREQ.
>>>>>>>>>>> 
>>>>>>>>>>> So can you add more context why this is necessary for everyone?
>>>>>>>>>> We have information for IOMMU and Master-ID but we don’t have 
>>>>>>>>>> information for linking vMaster-ID to pMaster-ID.
>>>>>>>>> 
>>>>>>>>> I am confused. Below, you are making the virtual master ID optional. 
>>>>>>>>> So shouldn't this be mandatory if you really need the mapping with 
>>>>>>>>> the virtual ID?
>>>>>>>> vMasterID is optional if user knows pMasterID is unique on the system. 
>>>>>>>> But if pMasterId is not unique then user needs to provide the 
>>>>>>>> vMasterID.
>>>>>>> 
>>>>>>> So the expectation is the user will be able to know that the pMasterID 
>>>>>>> is uniq. This may be easy with a couple of SMMUs, but if you have 50+ 
>>>>>>> (as suggested above). This will become a pain on larger system.
>>>>>>> 
>>>>>>> IHMO, it would be much better if we can detect that in libxl (see 
>>>>>>> below).
>>>>>> We can make the vMasterID compulsory to avoid complexity in libxl to 
>>>>>> solve this
>>>>> 
>>>>> In general, complexity in libxl is not too much of problem.
>>>> I am a bit unsure about this strategy.
>>>> Currently xl has one configuration file where you put all Xen parameters. 
>>>> The device tree is only needed by some guests to have a description of the 
>>>> system they run on.
>>>> If we change the model and say that Xen configuration parameters are both 
>>>> in the configuration and the device tree, we somehow enforce to have a 
>>>> device tree even though some guests do not need it at all (for example 
>>>> Zephyr).
>>> 
>>> I think my approach was misunderstood because there is no change in the 
>>> existing model.
>>> 
>>> What I am suggesting is to not introduce iommu_devid_map but instead let 
>>> libxl allocate the virtual Master-ID and create the mapping with the 
>>> physical Master-ID.
>>> 
>>> Libxl would then update the property "iommus" in the device-tree with the 
>>> allocated virtual Master-ID.
>> Ok I understand now.
>>> 
>>> Each node in the partial device-tree would need to have a property
>>> to refer to the physical device just so we know how to update the "iommus". 
>>> The list of device passthrough will still be specified in the configuration 
>>> file. IOW, the partial device-tree is not directly involved in the 
>>> configuration of the guest.
>> But we will generate it. How would something like Zephyr guest work ? Zephyr 
>> is not using the device tree we pass, it has an embedded one.
> 
> In general, guest that don't use the device-tree/ACPI table to detect the 
> layout are already in a bad situation because we don't guarantee that the 
> layout (memory, interrupt...) will be stable across Xen version. Although, 
> there are a implicit agreement that the layout will not change for minor 
> release (i.e. 4.14.x).

Well right now we have no ACPI support.
But I still think that a non dtb guest is definitely a use case we need to keep 
in mind for embedded and safety as most proprietary RTOS are not using a device 
tree.

> 
> But see below for some suggestions how this could be handled.
> 
>>> 
>>> So far, I don't see a particular issue with this approach because the 
>>> vMaster ID algorithm allocation should be generic. But please let me know 
>>> if you think there are bits I am missing.
>> I am a bit afraid of things that are “automatic”.
>> For everything else we let the user in control (IPA for mapping, virtual 
>> interrupt number) and in this case we switch to a model where we 
>> automatically generated a vMaster ID.
> 
> We only let the user control where the device is mapped. But this is quite 
> fragile... I think this should be generated at runtime.
> 
>> With this model, guest not using the device tree will have to guess the 
>> vMaster ID or somehow know how the tools are generating it to use the right 
>> one.
> 
> To be honest, this is already the case today because the layout exposed to 
> the guest is technically not fixed. Yes, so far, we haven't changed it too 
> much. But sooner or later, this is going to bite because we made clear that 
> the layout is not stable.
> 
> Now, if those projects are willing to rebuild for each version, then we could 
> use the following approach:
>  1) Write the xl.cfg
>  2) Ask libxl to generate the device-tree
>  3) Build Zephyr
>  4) Create the domain
> 
> The expectation is for a given Xen version (and compatible), libxl will 
> always generate the same Device-Tree.

This is a good idea yes :-)

Cheers
Bertrand

> 
> Cheers,
> 
> -- 
> Julien Grall