Re: [PATCH v5 7/7] docs: Add device tree overlay documentation

[Stefano Stabellini <sstabellini@xxxxxxxxxx>]

On Fri, 24 May 2024, Julien Grall wrote:
> Hi Stefano,
> 
> On 24/05/2024 03:18, Stefano Stabellini wrote:
> > From: Vikram Garhwal <fnu.vikram@xxxxxxxxxx>
> > 
> > Signed-off-by: Vikram Garhwal <fnu.vikram@xxxxxxxxxx>
> > Signed-off-by: Stefano Stabellini <stefano.stabellini@xxxxxxxxxx>
> > Signed-off-by: Henry Wang <xin.wang2@xxxxxxx>
> > ---
> >   docs/misc/arm/overlay.txt | 82 +++++++++++++++++++++++++++++++++++++++
> >   1 file changed, 82 insertions(+)
> >   create mode 100644 docs/misc/arm/overlay.txt
> > 
> > diff --git a/docs/misc/arm/overlay.txt b/docs/misc/arm/overlay.txt
> > new file mode 100644
> > index 0000000000..0a2dee951a
> > --- /dev/null
> > +++ b/docs/misc/arm/overlay.txt
> > @@ -0,0 +1,82 @@
> > +# Device Tree Overlays support in Xen
> > +
> > +Xen experimentally supports dynamic device assignment to running
> > +domains, i.e. adding/removing nodes (using .dtbo) to/from Xen device
> > +tree, and attaching them to a running domain with given $domid.
> > +
> > +Dynamic node assignment works in two steps:
> > +
> > +## Add/Remove device tree overlay to/from Xen device tree
> > +
> > +1. Xen tools check the dtbo given and parse all other user provided
> > arguments
> > +2. Xen tools pass the dtbo to Xen hypervisor via hypercall.
> > +3. Xen hypervisor applies/removes the dtbo to/from Xen device tree.
> > +
> > +## Attach device from the DT overlay to domain
> > +
> > +1. Xen tools check the dtbo given and parse all other user provided
> > arguments
> > +2. Xen tools pass the dtbo to Xen hypervisor via hypercall.
> > +3. Xen hypervisor attach the device to the user-provided $domid by
> > +   mapping node resources in the DT overlay.
> > +
> > +# Examples
> > +
> > +Here are a few examples on how to use it.
> > +
> > +## Dom0 device add
> > +
> > +For assigning a device tree overlay to Dom0, user should firstly properly
> > +prepare the DT overlay. More information about device tree overlays can be
> > +found in [1]. Then, in Dom0, enter the following:
> > +
> > +    (dom0) xl dt-overlay add overlay.dtbo
> > +
> > +This will allocate the devices mentioned in overlay.dtbo to Xen device
> > tree.
> > +
> > +To assign the newly added device from the dtbo to Dom0:
> > +
> > +    (dom0) xl dt-overlay attach overlay.dtbo 0
> > +
> > +Next, if the user wants to add the same device tree overlay to dom0
> > +Linux, execute the following:
> > +
> > +    (dom0) mkdir -p /sys/kernel/config/device-tree/overlays/new_overlay
> > +    (dom0) cat overlay.dtbo >
> > /sys/kernel/config/device-tree/overlays/new_overlay/dtbo
> > +
> > +Finally if needed, the relevant Linux kernel drive can be loaded using:
> > +
> > +    (dom0) modprobe module_name.ko
> > +
> > +## DomU device add/remove
> > +
> > +All the nodes in dtbo will be assigned to a domain; the user will need
> > +to prepare the dtb for the domU.
> 
> s/dtb/dtbo/? 

Yes, done


> But I am little bit confused with the wording. I think you may
> want to add *different dtbo* so it clarifies from the start (this only becomes
> obvious at the end of the section) that the user is not meant to use the same
> for all the commands.

Yes it was confusing, I tried to clarify it. This is a doc so we could
improve it further after the freeze.


>  For example, the `interrupt-parent`
> > +property of the DomU overlay should be changed to the Xen hardcoded
> > +value `0xfde8`, and the xen,reg property should be added to specify the
> > +address mappings. If xen,reg is not present, it is assumed 1:1 mapping.
> 
> Repeating an earlier comment here. I think xen,reg should be mandatory for
> non-direct mapped domain.

That's OK


> Also, can you clarify what is the expect property layout for xen,reg?

I'll add a pointer


> > +Below assumes the properly written DomU dtbo is `overlay_domu.dtbo`.
> > +
> > +For new domains to be created, the user will need to create the DomU
> > +with below properties properly configured in the xl config file:
> > +- `iomem`
> 
> I looked at your reply in v4 and I am afraid I still don't understand why we
> are mentioning 'iomem'. If we want to use the commands below, then the domain
> needs to be created in advance. So you can't yet know 'iomem'.
> 
> You could avoid "xl dt-overlay attach" but then you need the user to specify
> both "irqs" and "iomem". From a user point of view, it would be easier to add
> a new propery in the configuration file listing the overlays. Something like:
> 
> dt_overlays = [ "overlay.dtbo", ... ]
> 
> Anyway, that somewhat separate. For now, I think we want to drop 'iomem' from
> the list and reword this paragraph to say that the 'passthrough' property
> needs to be set if you plan to use DT overlay and devices requiring the IOMMU.

OK, I did that


> 
> > +- `passthrough` (if IOMMU is needed)
> 
> This property is required at the start because we don't support enabling the
> IOMMU lazily.

Yeah. I think it is much clearer now.


> > +
> > +User will also need to modprobe the relevant drivers. For already
> > +running domains, the user can use the xl dt-overlay attach command,
> > +example:
> > +
> > +    (dom0) xl dt-overlay add overlay.dtbo            # If not executed
> > before
> > +    (dom0) xl dt-overlay attach overlay.dtbo $domid
> > +    (dom0) xl console $domid                         # To access $domid
> > console
> > +
> > +Next, if the user needs to modify/prepare the overlay.dtbo suitable for
> > +the domU:
> > +
> > +    (domU) mkdir -p /sys/kernel/config/device-tree/overlays/new_overlay
> > +    (domU) cat overlay_domu.dtbo >
> > /sys/kernel/config/device-tree/overlays/new_overlay/dtbo
> > +
> > +Finally, if needed, the relevant Linux kernel drive can be probed:
> > +
> > +    (domU) modprobe module_name.ko
> > +
> > +[1] https://www.kernel.org/doc/Documentation/devicetree/overlay-notes.txt