[PATCH v4 22/22] doc/arm: vIOMMU design document

[Milan Djokic <milan_djokic@xxxxxxxx>]

This document outlines the design of the emulated IOMMU,
including supported features, security considerations
and future improvements.

Signed-off-by: Milan Djokic <milan_djokic@xxxxxxxx>
---
 docs/designs/arm-viommu.rst | 502 ++++++++++++++++++++++++++++++++++++
 1 file changed, 502 insertions(+)
 create mode 100644 docs/designs/arm-viommu.rst

diff --git a/docs/designs/arm-viommu.rst b/docs/designs/arm-viommu.rst
new file mode 100644
index 0000000000..c68e771711
--- /dev/null
+++ b/docs/designs/arm-viommu.rst
@@ -0,0 +1,502 @@
+==========================================================
+Design Proposal: Add SMMUv3 Stage-1 Support for XEN Guests
+==========================================================
+
+:Author:     Milan Djokic <milan_djokic@xxxxxxxx>
+:Date:       2026-02-13
+:Status:     Draft
+
+Introduction
+============
+
+The SMMUv3 supports two stages of translation. Each stage of translation 
+can be
+independently enabled. An incoming address is logically translated from 
+VA to
+IPA in stage 1, then the IPA is input to stage 2 which translates the IPA to
+the output PA. Stage 1 translation support is required to provide 
+isolation between different
+devices within OS. XEN already supports Stage 2 translation but there is no
+support for Stage 1 translation.
+This design proposal outlines the introduction of Stage-1 SMMUv3 support 
+in Xen for ARM guests.
+
+Motivation
+==========
+
+ARM systems utilizing SMMUv3 require stage-1 address translation to 
+ensure secure DMA and
+guest managed I/O memory mappings.
+With stage-1 enabled, guest manages IOVA to IPA mappings through its own 
+IOMMU driver.
+
+This feature enables:
+
+- Stage-1 translation for the guest domain
+- Device passthrough with per-device I/O address space
+
+Design Overview
+===============
+
+These changes provide emulated SMMUv3 support:
+
+- **SMMUv3 Stage-1 Translation**: stage-1 and nested translation support
+  in SMMUv3 driver.
+- **vIOMMU Abstraction**: Virtual IOMMU framework for guest stage-1
+  handling.
+- **Register/Command Emulation**: SMMUv3 register emulation and command
+  queue handling.
+- **Device Tree Extensions**: Adds `iommus` and virtual SMMUv3 nodes to
+  device trees for dom0 and dom0less scenarios.
+- **Build Configuration**: Introduces a Kconfig arguments
+  (CONFIG_ARM_VIRTUAL_IOMMU, CONFIG_VIRTUAL_ARM_SMMU_V3) for build-time
+  enablement.
+
+Depending on the domain type, the emulated IOMMU model is the following:
+
+- For the hardware domain, the host layout is re-used. The number of exposed
+  IOMMUs to hardware domain is equal to number of physical IOMMUs
+  present (1-1 mapping).
+- For non-hardware domain, a single vIOMMU device is exposed to the guest and
+  mapped to one or more physical IOMMUs through a Xen-managed translation layer
+
+The vIOMMU feature provides a generic framework together with a backend
+implementation specific to the target IOMMU type. The backend is responsible
+for implementing the hardware-specific data structures and command handling
+logic (currently only SMMUv3 is supported).
+
+This modular design allows the stage-1 support to be reused
+for other IOMMU architectures in the future.
+
+vIOMMU architecture
+===================
+
+Responsibilities:
+
+Guest:
+ - Configures stage-1 via vIOMMU commands.
+ - Handles stage-1 faults received from Xen.
+
+Xen:
+ - Emulates the IOMMU interface (registers, commands, events).
+ - Provides vSID->pSID mappings.
+ - Programs stage-1/stage-2 configuration in the physical IOMMU.
+ - Propagate stage-1 faults to guest.
+
+The stage-1 context is shared between streams assigned to the same device.
+This follows the Linux SMMUv3 driver model, where the IOMMU domain is 
associated
+with the master/device, while the individual SIDs are represented by separate
+STEs pointing to the same translation context. This is considered sufficient 
for
+the current SMMUv3 driver model, although the SMMUv3 architecture does not
+forbid having separate translation contexts for different SIDs.
+
+vIOMMU commands and faults are transmitted between guest and Xen via
+command and event queues (one command/event queue created per guest).
+
+vIOMMU command Flow:
+
+::
+
+    Guest:
+        smmu_cmd(vSID, IOVA -> IPA)
+
+    Xen:
+        trap MMIO read/write
+        translate vSID->pSID
+        store stage-1 state
+        program pIOMMU for (pSID, IPA -> PA)
+
+All hardware programming of the physical IOMMU is performed exclusively by Xen.
+
+vIOMMU Stage-1 fault handling flow:
+
+::
+
+    Xen:
+        receives stage-1 fault
+        triggers vIOMMU callback
+        injects virtual fault
+
+    Guest:
+        receives and handles fault
+
+Register access
+---------------
+
+Guest accesses to SMMUV3 registers are trapped via MMIO regions. vIOMMU 
emulates
+a subset of SMMUv3 registers relevant to currently supported features.
+Unsupported registers are handled as RAZ (Read-As-Zero) and WI (Write-Ignored)
+for the register read and write operations, respectively.
+
+vSMMUv3 is a partially emulated resource for the guest domain. Some
+feature register bits (SMMU_IDR) are reflecting what the underlying HW actually
+support while the rest is preset in the emulation layer to limit guest from
+using features which are not supported by the emulation. Also, implementer
+identification register is fully emulated, providing generic ARM implementer
+ID values, to avoid activating errata/quirk paths in the guest SMMUv3 driver.
+
+Supported SMMUv3 Commands
+-------------------------
+
+The vIOMMU implementation currently supports a subset of the SMMUv3
+architectural command set required for stage-1 translation management.
+
+Commands are submitted by the guest through the virtual Command Queue
+(CMDQ) and consumed by Xen, which translates them into operations on the
+physical SMMUv3 hardware.
+
+Some commands are fully handled by the emulation layer, while others are
+currently treated as architectural no-ops because the required behavior
+is either implicitly guaranteed by Xen or not yet modeled explicitly.
+
+Emulation handled commands:
+
+- CMD_CFGI_STE
+- CMD_TLBI_NH_ASID
+- CMD_TLBI_NSNH_ALL
+- CMD_TLBI_NH_VA
+
+No-op/Implicitly handled commands:
+
+- CMD_SYNC
+- CMD_PREFETCH_CFG
+- CMD_CFGI_CD
+- CMD_CFGI_CD_ALL
+- CMD_CFGI_ALL
+
+Command Handling Model
+----------------------
+
+The guest submits commands through the virtual CMDQ interface.
+
+Xen:
+
+- Validates commands
+- Translates virtual identifiers (vSID->pSID) where required
+- Programs the physical SMMU
+- Performs synchronization
+
+The physical SMMU executes the resulting invalidation or configuration
+operations.
+
+Unsupported Commands
+--------------------
+
+Commands associated with currently unsupported architectural features
+are not implemented.
+
+This includes commands related to:
+
+- PRI
+- ATS
+- Address/Range TLB invalidation (currently, all TLBI commands perform complete
+  domain TLB invalidation)
+- Advanced stall/retry semantics
+- CD runtime update/sync
+
+vSID Mapping Layer
+------------------
+
+Each non-hardware domain-visible Stream ID (vSID) is mapped by Xen to a 
physical
+Stream ID (pSID). The mapping is maintained per-domain. The allocation policy
+guarantees vSID uniqueness within a domain while allowing reuse of pSIDs for
+different pIOMMUs.
+
+* Platform devices receive individually allocated vSIDs.
+* PCI devices receive a contiguous vSID range derived from RID space.
+
+
+Supported Device Model
+======================
+
+Currently, the vIOMMU framework supports only devices described via the
+Device Tree (DT) model. This includes platform devices and basic PCI
+devices support instantiated through the vPCI DT node. ACPI-described
+devices are not supported.
+
+Guest assigned platform devices are mapped via `iommus` property:
+
+::
+
+    <&pIOMMU pSID> -> <&vIOMMU vSID>
+
+PCI devices use RID-based mapping via the root complex `iommu-map`:
+
+::
+
+    <RID-base &viommu vSID-base length>
+
+PCI Topology Assumptions and Constraints:
+
+- RID space must be contiguous
+- Pre-defined continuous pSID space (0-0x1000)
+- No runtime PCI reconfiguration
+- Single root complex assumed
+- Mapping is fixed at guest DT construction
+
+Constraints for PCI devices will be addressed as part of the future work on
+this feature.
+
+Security Considerations
+=======================
+
+Stage-1 translation provides isolation between guest devices by
+enforcing a per-device I/O address space, preventing unauthorized DMA.
+With the introduction of emulated IOMMU, additional protection
+mechanisms are required to minimize security risks.
+
+1. Observation:
+---------------
+Support for Stage-1 translation in SMMUv3 introduces new data structures 
+(`s1_cfg` alongside `s2_cfg`)
+and logic to write both Stage-1 and Stage-2 entries in the Stream Table 
+Entry (STE), including an `abort`
+field to handle partial configuration states.
+
+**Risk:**
+Without proper handling, a partially applied configuration
+might leave guest DMA mappings in an inconsistent state, potentially
+enabling unauthorized access or causing cross-domain interference.
+
+**Mitigation:** *(Handled by design)*
+This feature introduces logic that writes both `s1_cfg` and `s2_cfg` to
+STE and manages the `abort` field - only considering
+configuration if fully attached. This ensures  incomplete or invalid
+device configurations are safely ignored by the hypervisor.
+
+2. Observation:
+---------------
+Guests can now invalidate Stage-1 caches; invalidation needs forwarding
+to SMMUv3 hardware to maintain coherence.
+
+**Risk:**
+Failing to propagate cache invalidation could allow stale mappings,
+enabling access to old mappings and possibly
+data leakage or misrouting between devices assigned to the same guest.
+
+**Mitigation:**
+The guest must issue appropriate invalidation commands whenever
+its stage-1 I/O mappings are modified to ensure that translation caches
+remain coherent.
+
+3. Observation:
+---------------
+Introducing optional per-guest enabled features (`viommu` argument in xl 
+guest config) means some guests
+may opt-out.
+
+**Risk:**
+Guests without vIOMMU enabled (stage-2 only) could potentially dominate
+access to the physical command and event queues, since they bypass the
+emulation layer and processing is faster comparing to vIOMMU-enabled guests.
+
+**Mitigation:**
+Audit the impact of emulation overhead effect on IOMMU processing fairness
+in a multi-guest environment.
+Consider enabling/disabling stage-1 on a system level, instead of per-domain.
+
+4. Observation:
+---------------
+Guests have the ability to issue Stage-1 IOMMU commands like cache 
+invalidation, stream table entries
+configuration, etc. An adversarial guest may issue a high volume of 
+commands in rapid succession.
+
+**Risk:**
+Excessive commands requests can cause high hypervisor CPU consumption 
+and disrupt scheduling,
+leading to degraded system responsiveness and potential 
+denial-of-service scenarios.
+
+**Mitigation:**
+
+- Implement vIOMMU commands execution restart and continuation support:
+
+  - Introduce processing budget with only a limited amount of commands
+    handled per invocation.
+  - If additional commands remain pending after the budget is exhausted,
+    defer further processing and resume it asynchronously, e.g. via a
+    per-domain tasklet.
+
+- Batch multiple commands of same type to reduce emulation overhead:
+
+  - Inspect the command queue and group commands that can be processed
+    together (e.g. multiple successive invalidation requests or STE
+    updates for the same SID).
+  - Execute the entire batch in one go, reducing repeated accesses to
+    guest memory and emulation overhead per command.
+  - This reduces CPU time spent in the vIOMMU command processing loop.
+    The optimization is applicable only when consecutive commands of the
+    same type operate on the same SID/context.
+
+5. Observation:
+---------------
+Some guest commands issued towards vIOMMU are propagated to pIOMMU 
+command queue (e.g. TLB invalidate).
+
+**Risk:**
+Excessive commands requests from abusive guest can cause flooding of 
+physical IOMMU command queue,
+leading to degraded pIOMMU responsiveness on commands issued from other 
+guests.
+
+**Mitigation:**
+
+- Batch commands that are propagated to the pIOMMU command queue and
+  implement batch execution pause/continuation.
+  Rely on the same mechanisms as in the previous observation
+  (command continuation and batching of pIOMMU-related commands of the same
+  type and context).
+- If possible, implement domain penalization by adding a per-domain budget
+  for vIOMMU/pIOMMU usage:
+
+  - Apply per-domain dynamic budgeting of allowed IOMMU commands to
+    execute per invocation, reducing the budget for guests with
+    excessive command requests over a longer period of time
+  - Combine with command continuation mechanism
+
+6. Observation:
+---------------
+The vIOMMU feature includes an event queue used to forward IOMMU events
+to the guest (e.g. translation faults, invalid Stream IDs, permission errors).
+A malicious guest may misconfigure its IOMMU state or intentionally trigger
+faults at a high rate.
+
+**Risk:**
+Occurrence of IOMMU events with high frequency can cause Xen to flood the
+event queue and disrupt scheduling with
+high hypervisor CPU load for events handling.
+
+**Mitigation:**
+
+- Implement fail-safe state by disabling events forwarding when faults 
+  are occurred with high frequency and
+  not processed by guest:
+
+  - Introduce a per-domain pending event counter.
+  - Stop forwarding events to the guest once the number of unprocessed
+    events reaches a predefined threshold.
+
+- Consider disabling the emulated event queue for untrusted guests.
+- Note that this risk is more general and may also apply to stage-2-only
+  guests. This section addresses mitigations in the emulated IOMMU layer
+  only. Mitigation of physical event queue flooding should also be considered
+  in the target pIOMMU driver.
+
+7. Observation:
+---------------
+vIOMMU `CFGI_STE` commands can trigger serialized STE programming on the pIOMMU
+while `xen_domain->lock`, `devices_lock` and `cmd_queue_lock` are held, with
+local interrupts disabled. STE programming may issue multiple polling CMD_SYNC
+operations, each with a one-second timeout.
+
+**Risk:**
+An abusive guest can issue CFGI_STE commands that cause multiple CMD_SYNC
+operations to reach their polling timeout. This can keep a pCPU with interrupts
+disabled for a significant period of time, delaying interrupt handling and
+potentially degrading the responsiveness of Xen and other domains running on
+the same pCPU.
+
+**Mitigation:**
+
+- Implement pinning or otherwise maintaining the lifetime of the required
+  objects before releasing the relevant locks, allowing the
+  serialized STE programming to be performed without holding locks and with
+  local interrupts enabled.
+- Consider redesigning the SMMUv3 command processing and locking model in
+  future improvements to avoid performing potentially long-running pIOMMU
+  operations while holding locks and with local interrupts disabled.
+
+Performance Impact
+==================
+
+With iommu stage-1 and nested translation inclusion, performance 
+overhead is introduced comparing to existing,
+stage-2 only usage in Xen. Once mappings are established, translations 
+should not introduce significant overhead.
+Emulated paths may introduce moderate overhead, primarily affecting 
+device initialization and event/command handling.
+Testing is performed on Renesas R-Car platform.
+Performance is mostly impacted by emulated vIOMMU operations, results 
+shown in the following table.
+
++-------------------------------+---------------------------------+
+| vIOMMU Operation              | Execution time in guest         |
++===============================+=================================+
+| Reg read                      | median: 645ns, worst-case: 2us  |
++-------------------------------+---------------------------------+
+| Reg write                     | median: 630ns, worst-case: 1us  |
++-------------------------------+---------------------------------+
+| Invalidate TLB                | median: 2us, worst-case: 10us   |
++-------------------------------+---------------------------------+
+| Invalidate STE                | median: 5us worst_case: 100us   |
++-------------------------------+---------------------------------+
+
+With vIOMMU exposed to guest, guest OS has to initialize IOMMU device
+and configure stage-1 mappings for the devices
+attached to it.
+Following table shows initialization stages which impact stage-1 enabled 
+guest boot time and compares it with
+stage-1 disabled guest.
+
+NOTE: Device probe execution time varies depending on device complexity.
+A USB host controller was selected as the test device in this case.
+
++---------------------+-----------------------+------------------------+
+| Stage               | Stage-1 Enabled Guest | Stage-1 Disabled Guest |
++=====================+=======================+========================+
+| IOMMU Init          | ~10ms                 | /                      |
++---------------------+-----------------------+------------------------+
+| Dev Attach / Mapping| ~100ms                | ~90ms                  |
++---------------------+-----------------------+------------------------+
+
+For devices configured with dynamic DMA mappings, DMA allocate/map/unmap 
+operations performance is
+also impacted on stage-1 enabled guests.
+Dynamic DMA mapping operation trigger emulated IOMMU functions like mmio 
+write/read and TLB invalidations.
+
++---------------+---------------------------+--------------------------+
+| DMA Op        | Stage-1 Enabled Guest     | Stage-1 Disabled Guest   |
++===============+===========================+==========================+
+| dma_alloc     | median: 20us, worst: 5ms  | median: 8us, worst: 60us |
++---------------+---------------------------+--------------------------+
+| dma_free      | median: 500us, worst: 10ms| median: 6us, worst: 30us |
++---------------+---------------------------+--------------------------+
+| dma_map       | median: 12us, worst: 60us | median: 3us, worst: 20us |
++---------------+---------------------------+--------------------------+
+| dma_unmap     | median: 400us, worst: 5ms | median: 5us, worst: 20us |
++---------------+---------------------------+--------------------------+
+
+Testing
+=======
+
+- QEMU-based ARM system tests for Stage-1 translation.
+- Actual hardware validation to ensure compatibility with real SMMUv3 
+  implementations.
+- Unit/Functional tests validating correct translations (not implemented).
+
+Migration and Compatibility
+===========================
+
+This optional feature defaults to disabled (CONFIG_ARM_VIRTUAL_IOMMU=n,
+CONFIG_VIRTUAL_ARM_SMMU_V3=n).
+
+Future improvements
+===================
+
+- Implement the proposed mitigations to address security risks that are 
+  not covered by the current design
+  (events batching, commands execution continuation)
+- PCI support
+- Support for other IOMMU HW (Renesas, RISC-V, etc.)
+
+References
+==========
+
+- Original feature implemented by Rahul Singh:
+
+https://patchwork.kernel.org/project/xen-devel/cover/cover.1669888522.git.rahul.singh@xxxxxxx/
 
+
+- SMMUv3 architecture documentation
+- Existing vIOMMU code patterns (KVM, QEMU)
\ No newline at end of file
-- 
2.43.0