Re: Design session notes: GPU acceleration in Xen

[Jan Beulich <jbeulich@xxxxxxxx>]

On 13.06.2024 20:43, Demi Marie Obenour wrote:
> GPU acceleration requires that pageable host memory be able to be mapped
> into a guest.

I'm sure it was explained in the session, which sadly I couldn't attend.
I've been asking Ray and Xenia the same before, but I'm afraid it still
hasn't become clear to me why this is a _requirement_. After all that's
against what we're doing elsewhere (i.e. so far it has always been
guest memory that's mapped in the host). I can appreciate that it might
be more difficult to implement, but avoiding to violate this fundamental
(kind of) rule might be worth the price (and would avoid other
complexities, of which there may be lurking more than what you enumerate
below).

>  This requires changes to all of the Xen hypervisor, Linux
> kernel, and userspace device model.
> 
> ### Goals
> 
>  - Allow any userspace pages to be mapped into a guest.
>  - Support deprivileged operation: this API must not be usable for privilege 
> escalation.
>  - Use MMU notifiers to ensure safety with respect to use-after-free.
> 
> ### Hypervisor changes
> 
> There are at least two Xen changes required:
> 
> 1. Add a new flag to IOREQ that means "retry this instruction".
> 
>    An IOREQ server can set this flag after having successfully handled a
>    page fault.  It is expected that the IOREQ server has successfully
>    mapped a page into the guest at the location of the fault.
>    Otherwise, the same fault will likely happen again.

Were there any thoughts on how to prevent this becoming an infinite loop?
I.e. how to (a) guarantee forward progress in the guest and (b) deal with
misbehaving IOREQ servers?

> 2. Add support for `XEN_DOMCTL_memory_mapping` to use system RAM, not
>    just IOMEM.  Mappings made with `XEN_DOMCTL_memory_mapping` are
>    guaranteed to be able to be successfully revoked with
>    `XEN_DOMCTL_memory_mapping`, so all operations that would create
>    extra references to the mapped memory must be forbidden.  These
>    include, but may not be limited to:
> 
>    1. Granting the pages to the same or other domains.
>    2. Mapping into another domain using `XEN_DOMCTL_memory_mapping`.
>    3. Another domain accessing the pages using the foreign memory APIs,
>       unless it is privileged over the domain that owns the pages.

All of which may call for actually converting the memory to kind-of-MMIO,
with a means to later convert it back.

Jan

>    Open question: what if the other domain goes away?  Ideally,
>    unmapping would (vacuously) succeed in this case.  Qubes OS doesn't
>    care about domid reuse but others might.
> 
> ### Kernel changes
> 
> Linux will add support for mapping userspace memory into an emulated PCI
> BAR.  This requires Linux to automatically revoke access when needed.
> 
> There will be an IOREQ server that handles page faults.  The discussion
> assumed that this handling will happen in kernel mode, but if handling
> in user mode is simpler that is also an option.
> 
> There is no async #PF in Xen (yet), so the entire vCPU will be blocked
> while the fault is handled.  This is not great for performance, but
> correctness comes first.
> 
> There will be a new kernel ioctl to perform the mapping.  A possible C
> prototype (presented at design session, but not discussed there):
> 
>     struct xen_linux_register_memory {
>         uint64_t pointer;
>         uint64_t size;
>         uint64_t gpa;
>         uint32_t id;
>         uint32_t guest_domid;
>     };