Re: [PATCH v1] xen: fusa: arch_specs: Definition of the Virtual Machines' roles

["Yann Sionneau" <yann.sionneau@xxxxxxxxxx>]

Hi Ayan, Stefano,

This doc looks nice overall!

However I do think that maybe the role part should be separate from all 
the explanations about virtio.

I would say virtio deserves its own page.

Also, I'm curious what "FFI" is.

Thanks!

Regards,

Yann

On 3/4/25 19:31, Ayan Kumar Halder wrote:
> From: Stefano Stabellini <stefano.stabellini@xxxxxxx>
>
> Define "Hardware Domain", "Control Domain", "DomUs", "SafeVMs" and
> "UnsafeVMs".
>
> Explain the limitations that VirtIO introduces and the way they affect
> the defined roles.
>
> Signed-off-by: Stefano Stabellini <stefano.stabellini@xxxxxxx>
> Signed-off-by: Ayan Kumar Halder <ayan.kumar.halder@xxxxxxx>
> ---
>   docs/fusa/arch_specs/roles.rst | 200 +++++++++++++++++++++++++++++++++
>   docs/fusa/index.rst            |   1 +
>   2 files changed, 201 insertions(+)
>   create mode 100644 docs/fusa/arch_specs/roles.rst
>
> diff --git a/docs/fusa/arch_specs/roles.rst b/docs/fusa/arch_specs/roles.rst
> new file mode 100644
> index 0000000000..0b3fe821c3
> --- /dev/null
> +++ b/docs/fusa/arch_specs/roles.rst
> @@ -0,0 +1,200 @@
> +===============================
> +Xen Domain Roles and Privileges
> +===============================
> +
> +Roles
> +=====
> +
> +We define the following roles that Xen domains can have:
> +- Hardware Domain
> +- Control Domain
> +- DomU
> +
> +In addition, we classify domains based on the criticality of the tasks
> +they perform:
> +- SafeVM
> +- UnSafeVM
> +
> +DomU
> +----
> +
> +A DomU is a regular domain without extra privileges to perform
> +operations that affect other domains or the entire platform. A DomU is
> +therefore unprivileged and can only affect itself.
> +
> +Hardware Domain
> +---------------
> +
> +The Hardware Domain is a domain that has direct access to all hardware
> +peripherals by default, except for those controlled by Xen (CPU, MMU,
> +IOMMU, Interrupt Controller, Timers, UART). Hardware resources can be
> +directly assigned to other domains ("passthrough"), but the default is
> +that they get assigned to the Hardware Domain. As such, there can only
> +be one Hardware Domain in the system. The Hardware Domain is not
> +privileged: it cannot perform operations that affect other domains or
> +the entire platform.
> +
> +In cases where VirtIO is used to share access to physical devices among
> +multiple domains, the Hardware Domain requires some limited privileges
> +to run the VirtIO backends.
> +
> +Control Domain
> +--------------
> +
> +The Control Domain has full privileges over the system. As such, it can
> +perform privileged operations, typically implemented as hypercalls, that
> +affect other domains and even the entire platform, such as platform
> +reboot. There can only be one Control Domain in the system. A
> +non-exhaustive list of these operations follows:
> +
> +- Reboot other domains
> +- Reboot and shutdown the platform
> +- Suspend/resume other domains
> +- Pause/unpause other domains
> +
> +The Control Domain is optional.
> +
> +SafeVMs and UnSafeVMs
> +---------------------
> +
> +A SafeVM is a domain running a safety-critical task. An UnsafeVM is a
> +domain running non-safety-critical software only. No safety-critical
> +tasks are running in the UnsafeVM.
> +
> +The Control Domain must be a SafeVM because the Control Domain has
> +enough privileges to affect the entire system.
> +
> +Without VirtIO (see below), both the Hardware Domain and DomUs can be
> +SafeVMs or UnsafeVMs. There are no restrictions. For example, the
> +following configuration is compatible with safety:
> +
> +- Hardware Domain: UnsafeVM
> +- 2x DomU: SafeVMs
> +- 2x DomU: UnsafeVMs
> +
> +Privileges
> +==========
> +
> +Full Privileges
> +---------------
> +
> +A domain with full privileges has unrestricted access to all services
> +provided by Xen, including the ability to perform operations that impact
> +other VMs and the entire platform. Only one domain holds these full
> +privileges: the Control Domain.
> +
> +Unprivileged
> +------------
> +
> +An Unprivileged domain lacks special access rights, allowing it to
> +perform only operations that affect itself without impacting other
> +domains. For example, an Unprivileged domain cannot map the memory of
> +another domain.
> +
> +VirtIO Privileges
> +-----------------
> +
> +Domains running VirtIO backends, such as QEMU, require specific
> +privileges over domains with VirtIO frontends to enable proper
> +functionality. For instance, they need permission to map the memory of
> +the domain with the VirtIO frontend. These additional privileges are
> +necessary for standard VirtIO operation but can be removed when using
> +the VirtIO extensions described in the VirtIO chapter. This is the list
> +of VirtIO privileges:
> +- mapping memory of VirtIO frontend domains (foreign map hypercall)
> +- device model hypercalls targeting VirtIO frontend domains
> +
> +VirtIO
> +======
> +
> +VirtIO is a specification and set of drivers to share a physical device
> +among multiple domains. For example, assuming the Hardware Domain has
> +access to an Ethernet device, VirtIO provides a way for the Hardware
> +Domain to allow one or more DomUs to access the Ethernet device.
> +
> +VirtIO is based on a frontend/backend architecture: the frontend is a
> +driver similar to a regular Ethernet device driver (using Ethernet as an
> +example), running in the DomU. The backend is a "proxy" driver running
> +in the same domain as the device driver and have direct access to the
> +device. That domain is typically the Hardware Domain because it is the
> +default domain for hardware assignment, but it is also possible to run
> +VirtIO backends in other domains, provided they have been configured with
> +passthrough access to a physical device. These other domains are called
> +"Driver Domains," and the limitations discussed in the context of VirtIO
> +for the Hardware Domain also apply to Driver Domains.
> +
> +There are multiple issues in VirtIO that affect freedom from
> +interference, so deploying VirtIO introduces limits on the configuration
> +regarding what domains can run safety-critical workloads.
> +
> +It is important to note that VirtIO is not the only way for domains to
> +communicate with each other. Specifically, there are other protocols
> +such as Argo and plain shared memory that do not have the same
> +interference concerns.
> +
> +FFI: Shared Memory and Interrupts
> +---------------------------------
> +
> +All VirtIO protocols are based on ring buffers over shared memory,
> +called "VirtQueues." Although there are well-known methods to set up
> +ring buffers over shared memory that are free from interference, in
> +practice, the existing VirtIO specification and current frontend/backend
> +driver implementations do not follow these best practices. Thus, the
> +ring buffers over shared memory can be a source of interference.
> +
> +Given the current state of the VirtIO specification and drivers, it is
> +recommended that both VirtIO frontends and backends run in UnsafeVMs
> +only. This way, SafeVMs will not be affected by interference caused by
> +VirtIO protocols.
> +
> +As ring buffers exist over memory shared between domains, it is the
> +domains' responsibility to ensure they are safe. This is outside of
> +Xen's responsibility, and out of scope for this document.
> +
> +FFI: Memory Mappings
> +--------------------
> +
> +VirtIO expects the backend to access ("map") arbitrary memory addresses
> +passed by the frontend. In other words, the backend needs to map any
> +memory of the frontend, which is a privileged operation in Xen.
> +
> +Xen implements a Xen-specific extension to VirtIO to address this issue:
> +"VirtIO with Grants."
> +
> +Xen has a feature called the "grant table" that guests can use to
> +voluntarily share individual memory pages with other domains. This way,
> +the backend does not need the privilege to map any memory page of the
> +domain running the frontend, as the frontend whitelists pages for the
> +backend to access.
> +
> +Unless "VirtIO with Grants" is used, to enable VirtIO, a domain with
> +VirtIO backends (the Hardware Domain or a Driver Domain) is given the
> +extra privilege of being able to map any memory of domains with VirtIO
> +frontends.  As this is a source of interference, only UnsafeVMs can run
> +VirtIO frontends without "VirtIO with Grants".
> +
> +FFI: Synchronous Emulation
> +--------------------------
> +
> +VirtIO expects certain operations to be synchronous: the frontend
> +virtual CPU requesting the operation is fully blocked until the backend
> +completes the operation. If the backend does nothing, the frontend vCPU
> +remains blocked. This is a source of interference.
> +
> +When VirtIO is enabled, the domain with the backends (Hardware Domain or
> +Driver Domain) implements these synchronous operations for domains
> +running VirtIO frontends.
> +
> +Xen implements a Xen-specific extensions to VirtIO to address this
> +issue: "non-blocking VirtIO." Two extra virtio-pci registers are
> +provided, allowing the frontend to perform non-blocking operations. By
> +using the two Xen-specific non-blocking registers, the frontend virtual
> +CPU doesn't have to block while waiting for the backend to complete the
> +operations.
> +
> +Unless "non-blocking VirtIO" is used, to enable VirtIO, a domain with
> +VirtIO backends (the Hardware Domain or a Driver Domain) is given the
> +extra privilege of being able to implement these synchronous operations
> +for domains running VirtIO frontends. As this is a source of
> +interference, only UnsafeVMs can run VirtIO frontends without
> +"non-blocking VirtIO."
> diff --git a/docs/fusa/index.rst b/docs/fusa/index.rst
> index 5f1e8acfc4..9d989f5105 100644
> --- a/docs/fusa/index.rst
> +++ b/docs/fusa/index.rst
> @@ -7,3 +7,4 @@ Functional Safety documentation
>      :maxdepth: 2
>   
>      reqs/index
> +   arch_specs/roles


Yann Sionneau | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech