Re: AMD GPU problems under Xen

[Demi Marie Obenour <demi@xxxxxxxxxxxxxxxxxxxxxx>]

On Mon, Nov 28, 2022 at 11:18:00AM -0500, Alex Deucher wrote:
> On Mon, Nov 28, 2022 at 2:18 AM Demi Marie Obenour
> <demi@xxxxxxxxxxxxxxxxxxxxxx> wrote:
> >
> > Dear Christian:
> >
> > What is the status of the AMDGPU work for Xen dom0?  That was mentioned in
> > https://lore.kernel.org/dri-devel/b2dec9b3-03a7-e7ac-306e-1da024af8982@xxxxxxx/
> > and there have been bug reports to Qubes OS about problems with AMDGPU
> > under Xen (such as https://github.com/QubesOS/qubes-issues/issues/7648).
> 
> I would say it's a work in progress.  It depends what GPU  you have
> and what type of xen setup you are using (PV vs PVH, etc.).

The current situation is:

- dom0 is PV.
- VMs with assigned PCI devices are HVM and use a Linux-based stubdomain
  QEMU does not run in dom0.
- Everything else is PVH.

In the future, I believe the goal is to move away from PV and HVM in
favor of PVH, though HVM support will remain for compatibility with
guests (such as Windows) that need emulated devices.

> In general, your best bet currently is dGPU add in boards because they
> are largely self contained.

The main problem is that for the trusted GUI to work, there needs to
be at least one GPU attached to a trusted VM, such as the host or a
dedicated GUI VM.  That VM will typically not be running graphics-
intensive workloads, so the compute power of a dGPU is largely wasted.
SR-IOV support would help with that, but the only GPU vendor with open
source SR-IOV support is Intel and it is still not upstream.  I am also
not certain if the support extends to Arc dGPUs.

> APUs and platforms with integrated dGPUs
> are a bit more complicated as they tend to have more platform
> dependencies like ACPI tables and methods in order for the driver to
> be able to initialize the hardware properly.

Is Xen dom0/domU support for such GPUs being worked on?  Is there an
estimate as to when the needed support will be available upstream?  This
is mostly directed at Christian and other people who work for hardware
vendors.

> Additionally, GPUs map a
> lot of system memory so bounce buffers aren't really viable.  You'll
> really need IOMMU,

Qubes OS already needs an IOMMU so that is not a concern.  
-- 
Sincerely,
Demi Marie Obenour (she/her/hers)
Invisible Things Lab

Attachment: signature.asc
Description: PGP signature