Re: Design session notes: GPU acceleration in Xen

[Jan Beulich <jbeulich@xxxxxxxx>]

On 17.06.2024 02:38, Demi Marie Obenour wrote:
> On Fri, Jun 14, 2024 at 10:39:37AM +0200, Roger Pau Monné wrote:
>> On Fri, Jun 14, 2024 at 10:12:40AM +0200, Jan Beulich wrote:
>>> On 14.06.2024 09:21, Roger Pau Monné wrote:
>>>> On Fri, Jun 14, 2024 at 08:38:51AM +0200, Jan Beulich wrote:
>>>>> On 13.06.2024 20:43, Demi Marie Obenour wrote:
>>>>>> GPU acceleration requires that pageable host memory be able to be mapped
>>>>>> into a guest.
>>>>>
>>>>> I'm sure it was explained in the session, which sadly I couldn't attend.
>>>>> I've been asking Ray and Xenia the same before, but I'm afraid it still
>>>>> hasn't become clear to me why this is a _requirement_. After all that's
>>>>> against what we're doing elsewhere (i.e. so far it has always been
>>>>> guest memory that's mapped in the host). I can appreciate that it might
>>>>> be more difficult to implement, but avoiding to violate this fundamental
>>>>> (kind of) rule might be worth the price (and would avoid other
>>>>> complexities, of which there may be lurking more than what you enumerate
>>>>> below).
>>>>
>>>> My limited understanding (please someone correct me if wrong) is that
>>>> the GPU buffer (or context I think it's also called?) is always
>>>> allocated from dom0 (the owner of the GPU).  The underling memory
>>>> addresses of such buffer needs to be mapped into the guest.  The
>>>> buffer backing memory might be GPU MMIO from the device BAR(s) or
>>>> system RAM, and such buffer can be paged by the dom0 kernel at any
>>>> time (iow: changing the backing memory from MMIO to RAM or vice
>>>> versa).  Also, the buffer must be contiguous in physical address
>>>> space.
>>>
>>> This last one in particular would of course be a severe restriction.
>>> Yet: There's an IOMMU involved, isn't there?
>>
>> Yup, IIRC that's why Ray said it was much more easier for them to
>> support VirtIO GPUs from a PVH dom0 rather than classic PV one.
>>
>> It might be easier to implement from a classic PV dom0 if there's
>> pv-iommu support, so that dom0 can create it's own contiguous memory
>> buffers from the device PoV.
> 
> What makes PVH an improvement here?  I thought PV dom0 uses an identity
> mapping for the IOMMU,

True, but see how Roger mentioned PV IOMMU (which would allow a domain
to move away from this identity mapping).

Jan

> while a PVH dom0 uses an IOMMU that mirrors the
> dom0 second-stage page tables.  In both cases, the device physical
> addresses are identical to dom0’s physical addresses.
> 
> PV is terrible for many reasons, so I’m okay with focusing on PVH dom0,
> but I’d like to know why there is a difference.