Re: [PATCH V4] Add requirements for Device Passthrough

[Stewart Hildebrand <stewart.hildebrand@xxxxxxx>]

On 4/8/24 09:19, Oleksandr Tyshchenko wrote:
> From: Oleksandr Tyshchenko <oleksandr_tyshchenko@xxxxxxxx>
> 
> Please refer to chapter "Device Passthrough":
> https://groups.io/g/amd-xen-safety/message/1300
> 
> Create corresponding directory and README file.
> 
> Signed-off-by: Oleksandr Tyshchenko <oleksandr_tyshchenko@xxxxxxxx>

Two nits below. With those addressed:

Reviewed-by: Stewart Hildebrand <stewart.hildebrand@xxxxxxx>

> ---
> 
>   V2:
>    - add R-b
>    - update README
>    - lower case for platform, s/simple/non-DMA-capable, other misc
>      updates
>    - add "Allowed for the safe direct mapped VMs only"
>      to reqs for DMA-capable devices without IOMMU protection
>    - add dom0less passthrough details where needed
>    - add reqs for PCI devices discovering
> 
>   V3:
>    - move common reqs "Assign PCI device to domain (without IOMMU)" and
>      "Deassign PCI device from domain (without IOMMU)" to Arm64 only
>    - clarify the DMA-capable device assignment w/o IOMMU,
>      add more details
>    - drop R-b
> 
>   V4:
>    - add the following reqs:
>      - Assign interrupt-less platform device to domain
>      - Deassign interrupt-less platform device from domain
>      - Map platform device MMIO region identity
>      - Map platform device MMIO region non-identity
>    - add more details
>    - repeat the relevant info in all assign reqs
> ---
> ---
>  .../physical_resources/README.rst             |  16 +
>  .../physical_resources/passthrough.rst        | 477 ++++++++++++++++++
>  2 files changed, 493 insertions(+)
>  create mode 100644 domain_creation_and_runtime/physical_resources/README.rst
>  create mode 100644 
> domain_creation_and_runtime/physical_resources/passthrough.rst
> 
> diff --git a/domain_creation_and_runtime/physical_resources/README.rst 
> b/domain_creation_and_runtime/physical_resources/README.rst
> new file mode 100644
> index 0000000..0eb4dd4
> --- /dev/null
> +++ b/domain_creation_and_runtime/physical_resources/README.rst
> @@ -0,0 +1,16 @@
> +Physical resources
> +==================
> +
> +This section lists the requirements related to physical resources directly
> +accessible from the domain as well as physical resources entirely controlled
> +by Xen and invisible to a domain. The later group of resources, although 
> being
> +invisible to a domain, has an impact on it.
> +
> +Examples of domain physical resources:
> +    | 1. PCI device
> +    | 2. Platform device
> +    | 3. MMU stage 1
> +
> +Examples of Xen physical resources:
> +    | 1. IOMMU stage 2
> +    | 2. MMU stage 2
> diff --git a/domain_creation_and_runtime/physical_resources/passthrough.rst 
> b/domain_creation_and_runtime/physical_resources/passthrough.rst
> new file mode 100644
> index 0000000..f619730
> --- /dev/null
> +++ b/domain_creation_and_runtime/physical_resources/passthrough.rst
> @@ -0,0 +1,477 @@
> +Device Passthrough
> +==================
> +
> +The following are the requirements related to a physical device
> +assignment [1], [2] to Arm64 and AMD64 PVH domains.
> +
> +Requirements for both Arm64 and AMD64 PVH
> +=========================================
> +
> +Hide IOMMU from a domain
> +------------------------
> +
> +`XenSSR~hide_iommu_from_domain~1`
> +
> +Description:
> +Xen should not expose the IOMMU device to the domain even if I/O 
> virtualization
> +is disabled. The IOMMU should be under hypervisor control only.
> +
> +Rationale:
> +
> +Covers:
> + - `XenPRQ~device_passthrough~1`
> +
> +Needs:
> + - XenValTestCase
> +
> +Discover PCI devices from hardware domain
> +-----------------------------------------
> +
> +`XenSSR~discover_pci_devices_from_hwdom~1`
> +
> +Description:
> +The hardware domain shall be able to enumerate and discover PCI devices and
> +inform Xen about their appearance and disappearance
> +
> +Rationale:
> +
> +Covers:
> + - `XenPRQ~device_passthrough~1`
> +
> +Needs:
> + - XenValTestCase
> +
> +Discover PCI devices from Xen
> +-----------------------------
> +
> +`XenSSR~discover_pci_devices_from_xen~1`
> +
> +Description:
> +Xen shall be able to discover PCI devices (enumerated by the firmware
> +beforehand) during boot if the hardware domain is not meant to be used
> +
> +Rationale:
> +
> +Covers:
> + - `XenPRQ~device_passthrough~1`
> +
> +Needs:
> + - XenValTestCase
> +
> +Assign PCI device to domain (with IOMMU)
> +----------------------------------------
> +
> +`XenSSR~assign_pci_device_with_iommu~1`
> +
> +Description:
> +Xen shall be able to assign a specified PCI device (always implied as
> +DMA-capable) to a domain during its creation using passthrough (partial)
> +device tree. The physical device to be assigned is protected by the IOMMU.
> +
> +Rationale:
> +
> + - The passthrough device tree is specified using a device tree module node
> +   with compatible ("multiboot,device-tree") in the host device tree
> + - The PCI device to be passed through is specified using device tree 
> property
> +   ("xen,pci-assigned") in the "passthrough" node described in the 
> passthrough
> +   device tree
> +
> +Covers:
> + - `XenPRQ~device_passthrough~1`
> +
> +Needs:
> + - XenValTestCase
> +
> +Deassign PCI device from domain (with IOMMU)
> +--------------------------------------------
> +
> +`XenSSR~deassign_pci_device_with_iommu~1`
> +
> +Description:
> +Xen shall be able to deassign a specified PCI device from a domain during its
> +destruction. The physical device to be deassigned is protected by the IOMMU.
> +
> +Rationale:
> +
> +Covers:
> + - `XenPRQ~device_passthrough~1`
> +
> +Needs:
> + - XenValTestCase
> +
> +Forbid the same PCI device assignment to multiple domains
> +---------------------------------------------------------
> +
> +`XenSSR~forbid_same_pci_device_assignment~1`
> +
> +Description:
> +Xen should not assign the same PCI device to multiple domains by failing
> +to create a new domain if the device to be passed through is already assigned
> +to the existing domain. Also different PCI devices which share any resources
> +(interrupts, IOMMU connections) can be assigned only to the same domain.
> +
> +Rationale:
> +
> +Covers:
> + - `XenPRQ~device_passthrough~1`
> +
> +Needs:
> + - XenValTestCase
> +
> +Requirements for Arm64 only
> +===========================
> +
> +Assign interrupt-less platform device to domain
> +-----------------------------------------------
> +
> +`XenSSR~assign_interrupt_less_platform_device~1`
> +
> +Description:
> +Xen shall be able to assign a specified platform device that has only a MMIO
> +region (does not have any interrupts) to a domain during its creation using
> +passthrough device tree.
> +The example of interrupt-less device is PWM or clock controller.
> +
> +Rationale:
> +
> + - The passthrough device tree is specified using a device tree module node
> +   with compatible ("multiboot,device-tree") in the host device tree
> + - The passthrough device tree should entirely describe the platform device 
> to
> +   be passed through for the domain to be able to discover and use this 
> device
> + - The intention of the platform device usage for the passthrough is 
> specified
> +   using device tree property ("xen,passthrough") in the device node 
> described
> +   in the host device tree
> + - The memory region of the platform device and corresponding guest address 
> for
> +   remapping are specified using device tree property ("xen,reg") in the 
> device
> +   node described in the passthrough device tree
> + - The allowance of the platform device assignment which is not behind an 
> IOMMU
> +   (for both non-DMA-capable and DMA-capable devices) is specified using 
> device
> +   tree property ("xen,force-assign-without-iommu") in the device node
> +   described in the passthrough device tree. The said property also allows
> +   the interrupt-less platform device assignment (a device that has only a 
> MMIO
> +   region) without specifying the corresponding node in the host device via
> +   device tree property ("xen,path").
> +   For the DMA-capable devices this property is only allowed for the direct
> +   mapped VMs with special care to be taken
> +
> +Covers:
> + - `XenPRQ~device_passthrough~1`
> +
> +Needs:
> +
> + - XenValTestCase
> +
> +Deassign interrupt-less platform device from domain
> +---------------------------------------------------
> +
> +`XenSSR~deassign_interrupt_less_platform_device~1`
> +
> +Description:
> +Xen shall be able to deassign a specified platform device that has only a 
> MMIO
> +region (does not have any interrupts) from a domain during its destruction
> +
> +Rationale:
> +
> +Covers:
> + - `XenPRQ~device_passthrough~1`
> +
> +Needs:
> + - XenValTestCase
> +
> +Assign non-DMA-capable platform device to domain
> +------------------------------------------------
> +
> +`XenSSR~assign_non_dma_platform_device~1`
> +
> +Description:
> +Xen shall be able to assign a specified non-DMA-capable platform
> +device to a domain during its creation using passthrough device tree.
> +The example of non-DMA-capable device is Timer.
> +
> +Rationale:
> +
> + - The passthrough device tree is specified using a device tree module node
> +   with compatible ("multiboot,device-tree") in the host device tree
> + - The passthrough device tree should entirely describe the platform device 
> to
> +   be passed through for the domain to be able to discover and use this 
> device
> + - The intention of the platform device usage for the passthrough is 
> specified
> +   using device tree property ("xen,passthrough") in the device node 
> described
> +   in the host device tree
> + - The memory region of the platform device and corresponding guest address 
> for
> +   remapping are specified using device tree property ("xen,reg") in the 
> device
> +   node described in the passthrough device tree
> + - The path of the platform device node in the host device tree is specified
> +   using device tree property ("xen,path") in the device node described
> +   in the passthrough device tree. Both interrupt mappings and IOMMU settings
> +   are based on it
> + - The allowance of the platform device assignment which is not behind an 
> IOMMU
> +   (for both non-DMA-capable and DMA-capable devices) is specified using 
> device
> +   tree property ("xen,force-assign-without-iommu") in the device node
> +   described in the passthrough device tree. The said property also allows
> +   the interrupt-less platform device assignment (a device that has only a 
> MMIO
> +   region) without specifying the corresponding node in the host device via
> +   device tree property ("xen,path").
> +   For the DMA-capable devices this property is only allowed for the direct
> +   mapped VMs with special care to be taken
> +
> +Covers:
> + - `XenPRQ~device_passthrough~1`
> +
> +Needs:
> + - XenValTestCase
> +
> +Deassign non-DMA-capable platform device from domain
> +----------------------------------------------------
> +
> +`XenSSR~deassign_non_dma_platform_device~1`
> +
> +Description:
> +Xen shall be able to deassign a specified non-DMA-capable platform
> +device from a domain during its destruction
> +
> +Rationale:
> +
> +Covers:
> + - `XenPRQ~device_passthrough~1`
> +
> +Needs:
> + - XenValTestCase
> +
> +Assign DMA-capable platform device to domain (with IOMMU)
> +---------------------------------------------------------
> +
> +`XenSSR~assign_dma_platform_device_with_iommu~1`
> +
> +Description:
> +Xen shall be able to assign a specified DMA-capable platform device to a 
> domain
> +during its creation using passthrough device tree. The physical device to be
> +assigned is protected by the IOMMU.
> +
> +Rationale:
> +
> + - The passthrough device tree is specified using a device tree module node
> +   with compatible ("multiboot,device-tree") in the host device tree
> + - The passthrough device tree should entirely describe the platform device 
> to
> +   be passed through for the domain to be able to discover and use this 
> device
> + - The intention of the platform device usage for the passthrough is 
> specified
> +   using device tree property ("xen,passthrough") in the device node 
> described
> +   in the host device tree
> + - The memory region of the platform device and corresponding guest address 
> for
> +   remapping are specified using device tree property ("xen,reg") in the 
> device
> +   node described in the passthrough device tree
> + - The path of the platform device node in the host device tree is specified
> +   using device tree property ("xen,path") in the device node described
> +   in the passthrough device tree. Both interrupt mappings and IOMMU settings
> +   are based on it
> +
> +Covers:
> + - `XenPRQ~device_passthrough~1`
> +
> +Needs:
> + - XenValTestCase
> +
> +Deassign DMA-capable platform device from domain (with IOMMU)
> +-------------------------------------------------------------
> +
> +`XenSSR~deassign_dma_platform_device_with_iommu~1`
> +
> +Description:
> +Xen shall be able to deassign a specified DMA-capable platform device from
> +a domain during its destruction. The physical device to be deassigned is
> +protected by the IOMMU.
> +
> +Rationale:
> +
> +Covers:
> + - `XenPRQ~device_passthrough~1`
> +
> +Needs:
> + - XenValTestCase
> +
> +Assign DMA-capable platform device to domain (without IOMMU)
> +------------------------------------------------------------
> +
> +`XenSSR~assign_dma_platform_device_without_iommu~1`
> +
> +Description:
> +Xen shall be able to assign a specified DMA-capable platform device to a 
> domain
> +during its creation using passthrough device tree. The physical device to be
> +assigned is not protected by the IOMMU.
> +Here and everywhere it applies, the DMA-capable device assignment which is 
> not
> +behind an IOMMU is allowed for the direct mapped VMs only. The direct mapped 
> VM
> +must be either safe or additional security mechanisms for protecting against
> +possible malicious or buggy DMA devices must be in place, e.g. Xilinx memory
> +protection unit (XMPU) and Xilinx peripheral protection unit (XPPU).
> +
> +Rationale:
> +
> + - The IOMMU is absent from the system or it is disabled (status = "disabled"
> +   in the host device tree)
> + - The passthrough device tree is specified using a device tree module node
> +   with compatible ("multiboot,device-tree") in the host device tree
> + - The passthrough device tree should entirely describe the platform device 
> to
> +   be passed through for the domain to be able to discover and use this 
> device
> + - The intention of the platform device usage for the passthrough is 
> specified
> +   using device tree property ("xen,passthrough") in the device node 
> described
> +   in the host device tree
> + - The memory region of the platform device and corresponding guest address 
> for
> +   remapping are specified using device tree property ("xen,reg") in the 
> device
> +   node described in the passthrough device tree
> + - The path of the platform device node in the host device tree is specified
> +   using device tree property ("xen,path") in the device node described
> +   in the passthrough device tree. Both interrupt mappings and IOMMU settings
> +   are based on it
> + - The allowance of the platform device assignment which is not behind an 
> IOMMU
> +   (for both non-DMA-capable and DMA-capable devices) is specified using 
> device
> +   tree property ("xen,force-assign-without-iommu") in the device node
> +   described in the passthrough device tree. The said property also allows
> +   the interrupt-less platform device assignment (a device that has only a 
> MMIO
> +   region) without specifying the corresponding node in the host device via
> +   device tree property ("xen,path").
> +   For the DMA-capable devices this property is only allowed for the direct
> +   mapped VMs with special care to be taken
> +
> +Covers:
> + - `XenPRQ~device_passthrough~1`
> +
> +Needs:
> + - XenValTestCase
> +
> +Deassign DMA-capable platform device from domain (without IOMMU)
> +----------------------------------------------------------------
> +
> +`XenSSR~deassign_dma_platform_device_without_iommu~1`
> +
> +Description:
> +Xen shall be able to deassign a specified DMA-capable platform device from
> +a domain during its destruction. The physical device to be deassigned is not
> +protected by the IOMMU.
> +
> +Rationale:
> +The IOMMU is absent from the system or it is disabled (status = "disabled"
> +in the host device tree)
> +
> +Covers:
> + - `XenPRQ~device_passthrough~1`
> +
> +Needs:
> + - XenValTestCase
> +
> +Map platform device MMIO region identity
> +----------------------------------------
> +
> +`XenSSR~map_platform_device_mmio_region_ident~1`
> +
> +Description:
> +Xen shall be able to map platform device memory region identity
> +(guest address == physical address) when assigning a specified platform
> +device to a domain. The device can be either non-DMA-capable or DMA-capable.
> +
> +Rationale:
> +
> +Covers:
> + - `XenPRQ~device_passthrough~1`
> +
> +Needs:
> + - XenValTestCase
> +
> +Map platform device MMIO region non-identity
> +--------------------------------------------
> +
> +`XenSSR~map_platform_device_mmio_region_non_ident~1`
> +
> +Description:
> +Xen shall be able to map platform device memory region non-identity
> +(guest address != physical address) when assigning a specified platform
> +device to a domain. The device can be either non-DMA-capable or DMA-capable.
> +
> +Rationale:
> +
> +Covers:
> + - `XenPRQ~device_passthrough~1`
> +
> +Needs:
> + - XenValTestCase
> +
> +Assign PCI device to domain (without IOMMU)
> +-------------------------------------------
> +
> +`XenSSR~assign_pci_device_without_iommu~1`
> +
> +Description:
> +Xen shall be able to assign a specified PCI device to a domain during its
> +creation using passthrough device tree. The physical device to be assigned
> +is not protected by the IOMMU.

Nit: The same limitation as the platform device applies here too: it is
allowed for direct mapped VMs only. You do mention "... and everywhere
it applies ..." above, so I guess that covers it here as well.
Regardless, I think it would be clearer to mention it here too, so we
don't have to imply from the context of another requirement.

> +
> +Rationale:
> +
> + - The IOMMU is absent from the system or it is disabled (status = "disabled"
> +   in the host device tree)
> + - The passthrough device tree is specified using a device tree module node
> +   with compatible ("multiboot,device-tree") in the host device tree
> + - The PCI device to be passed through is specified using device tree 
> property
> +   ("xen,pci-assigned") in the "passthrough" node described in the 
> passthrough
> +   device tree
> +
> +Covers:
> + - `XenPRQ~device_passthrough~1`
> +
> +Needs:
> + - XenValTestCase
> +
> +Deassign PCI device from domain (without IOMMU)
> +-----------------------------------------------
> +
> +`XenSSR~deassign_pci_device_without_iommu~1`
> +
> +Description:
> +Xen shall be able to deassign a specified PCI device from a domain during its
> +destruction. The physical device to be deassigned is not protected
> +by the IOMMU.
> +
> +Rationale:
> +The IOMMU is absent from the system or it is disabled (status = "disabled"
> +in the host device tree)
> +
> +Covers:
> + - `XenPRQ~device_passthrough~1`
> +
> +Needs:
> + - XenValTestCase
> +
> +Forbid the same platform device assignment to multiple domains
> +--------------------------------------------------------------
> +
> +`XenSSR~forbid_same_platform_device_assignment~1`
> +
> +Description:
> +Xen should not assign the same platform device to multiple domains by failing
> +to create a new domain if the device to be passed through is already assigned
> +to the existing domain. Also, different platform devices which share any
> +resources (interrupts, IOMMU connections) can be assigned only to the same
> +domain.
> +
> +Rationale:
> +
> +Covers:
> + - `XenPRQ~device_passthrough~1`
> +
> +Needs:
> + - XenValTestCase
> +
> +Notes
> +=====
> +
> +The AMD64 PVH-specific requirements are written under the assumption that 
> once
> +the hyperlaunch feature is completed, Xen shall be able to assign a PCI 
> device
> +to started at boot time domains.

Nit: The grammar here looks odd. How about "... assign a PCI device to
boot time domains." ?

> This is not the case today, where the PCI
> +device can be passed through only to domains launched by a control 
> (toolstack)
> +domain.
> +
> +The Arm64-specific requirements are written under the assumption that once
> +the dom0less PCI Passthrough feature is completed, Xen shall be able to 
> assign
> +a PCI device to started at boot time domains.

Similarly here.

> This is not the case today,
> +where only the platform device Passthrough is supported.
> +
> +| [1] 
> https://xenbits.xenproject.org/gitweb/?p=xen.git;a=blob;f=docs/misc/arm/passthrough.txt;hb=HEAD
> +| [2] 
> https://xenbits.xenproject.org/gitweb/?p=xen.git;a=blob;f=docs/misc/arm/passthrough-noiommu.txt;hb=HEAD