Re: Mapping memory into a domain

[Demi Marie Obenour <demiobenour@xxxxxxxxx>]

On 5/8/25 3:52 AM, Roger Pau Monné wrote:
> On Wed, May 07, 2025 at 08:36:07PM -0400, Demi Marie Obenour wrote:
>> On 5/7/25 1:39 PM, Roger Pau Monné wrote:
>>> On Tue, May 06, 2025 at 04:56:12PM -0400, Demi Marie Obenour wrote:
>>>> On 5/6/25 9:06 AM, Alejandro Vallejo wrote:
>>>>> On Tue May 6, 2025 at 3:02 AM CEST, Demi Marie Obenour wrote:
>>>>>> On 5/5/25 7:32 AM, Alejandro Vallejo wrote:
>>>>>>> I suppose this is still about multiplexing the GPU driver the way we
>>>>>>> last discussed at Xen Summit?
>>>>>>>
>>>>>>> On Mon May 5, 2025 at 12:51 AM CEST, Demi Marie Obenour wrote:
>>>>>>>> What are the appropriate Xen internal functions for:
>>>>>>>>
>>>>>>>> 1. Turning a PFN into an MFN?
>>>>>>>> 2. Mapping an MFN into a guest?
>>>>>>>> 3. Unmapping that MFN from a guest?
>>>>>>>
>>>>>>> The p2m is the single source of truth about such mappings.
>>>>>>>
>>>>>>> This is all racy business. You want to keep the p2m lock for the full
>>>>>>> duration of whatever operation you wish do, or you risk another CPU
>>>>>>> taking it and pulling the rug under your feet at the most inconvenient
>>>>>>> time.
>>>>>>>
>>>>>>> In general all this faff is hidden under way too many layers beneath
>>>>>>> copy_{to,from}_guest(). Other p2m manipulation high-level constructs
>>>>>>> that might do interesting things worth looking at may be 
>>>>>>> {map,unmap}_mmio_region()
>>>>>>>
>>>>>>> Note that not every pfn has an associated mfn. Not even every valid pfn
>>>>>>> has necessarily an associated mfn (there's pod). And all of this is
>>>>>>> volatile business in the presence of a baloon driver or vPCI placing
>>>>>>> mmio windows over guest memory.
>>>>>>
>>>>>> Can I check that POD is not in use?  
>>>>>
>>>>> Maybe, but now you're reaching exponential complexity considering each
>>>>> individual knob of the p2m into account.
>>>>>
>>>>>>
>>>>>>> In general anything up this alley would need a cohesive pair for
>>>>>>> map/unmap and a credible plan for concurrency and how it's all handled
>>>>>>> in conjunction with other bits that touch the p2m.
>>>>>>
>>>>>> Is taking the p2m lock for the entire operation a reasonable approach
>>>>>> for concurrency?  Will this cause too much lock contention?
>>>>>
>>>>> Maybe. It'd be fine for a page. Likely not so for several GiB if they
>>>>> aren't already superpages.
>>>>>
>>>>>>
>>>>>>>> The first patch I am going to send with this information is a 
>>>>>>>> documentation
>>>>>>>> patch so that others do not need to figure this out for themselves.
>>>>>>>> I remember being unsure even after looking through the source code, 
>>>>>>>> which
>>>>>>>> is why I am asking here.
>>>>>>>
>>>>>>> That's not surprising. There's per-arch stuff, per-p2mtype stuff,
>>>>>>> per-guesttype stuff. Plus madness like on-demand memory. It's no wonder
>>>>>>> such helpers don't exist and the general manipulations are hard to
>>>>>>> explain.
>>>>>>
>>>>>> Is this a task that is only suitable for someone who has several years
>>>>>> experience working on Xen, or is it something that would make sense for
>>>>>> someone who is less experienced?
>>>>>
>>>>> The p2m is a very complex beast that integrates more features than I
>>>>> care to count. It requires a lot of prior knowledge. Whoever does it
>>>>> must know Xen fairly well in many configurations.
>>>>>
>>>>> The real problem is finding the right primitives that do what you want
>>>>> without overcomplicating everything else, preserving system security
>>>>> invariants and have benign (and ideally clear) edge cases.
>>>>>
>>>>> This was the last email you sent (I think?). Has any of the requirements
>>>>> changed in any direction?
>>>>>
>>>>>   https://lore.kernel.org/xen-devel/Z5794ysNE4KDkFuT@itl-email/
>>>>
>>>> Map and Revoke are still needed, with the same requirements as described
>>>> in this email.  Steal and Return were needed for GPU shared virtual memory,
>>>> but it has been decided to not support this with virtio-GPU, so these
>>>> primitives are no longer needed.
>>>>
>>>>> Something I'm missing there is how everything works without Xen. That
>>>>> might help (me, at least) guage what could prove enough to support the
>>>>> usecase. Are there sequence diagrams anywhere about how this whole thing
>>>>> works without Xen? I vaguely remember you showing something last year in
>>>>> Xen Summit in the design session, but my memory isn't that good :)
>>>
>>> Hello,
>>>
>>> Sorry, possibly replying a bit out of context here.
>>>
>>> Since I will mention this in several places: p2m is the second stage
>>> page-tables used by Xen for PVH and HVM guests.  A p2m violation is
>>> the equivalent of a page-fault for guest p2m accesses.
>>>
>>>> A Linux driver that needs access to userspace memory
>>>> pages can get it in two different ways:
>>>>
>>>> 1. It can pin the pages using the pin_user_pages family of APIs.
>>>>    If these functions succeed, the driver is guaranteed to be able
>>>>    to access the pages until it unpins them.  However, this also
>>>>    means that the pages cannot be paged out or migrated.  Furthermore,
>>>>    file-backed pages cannot be safely pinned, and pinning GPU memory
>>>>    isn’t supported.  (At a minimum, it would prevent the pages from
>>>>    migrating from system RAM to VRAM, so all access by a dGPU would
>>>>    cross the PCIe bus, which would be very slow.)
>>>
>>> From a Xen p2m this is all fine - Xen will never remove pages from the
>>> p2m unless it's requested to.  So the pining, while needed on the Linux
>>> side, doesn't need to be propagated to Xen I would think.
>>
>> If pinning were enough things would be simple, but sadly it’s not.
>>
>>>> 2. It can grab the *current* location of the pages and register an
>>>>    MMU notifier.  This works for GPU memory and file-backed memory.
>>>>    However, when the invalidate_range function of this callback, the
>>>>    driver *must* stop all further accesses to the pages.
>>>>
>>>>    The invalidate_range callback is not allowed to block for a long
>>>>    period of time.  My understanding is that things like dirty page
>>>>    writeback are blocked while the callback is in progress.  My
>>>>    understanding is also that the callback is not allowed to fail.
>>>>    I believe it can return a retryable error but I don’t think that
>>>>    it is allowed to keep failing forever.
>>>>
>>>>    Linux’s grant table driver actually had a bug in this area, which
>>>>    led to deadlocks.  I fixed that a while back.
>>>>
>>>> KVM implements the second option: it maps pages into the stage-2
>>>> page tables (or shadow page tables, if that is chosen) and unmaps
>>>> them when the invalidate_range callback is called.
>>>
>>> I assume this map and unmap is done by the host as a result of some
>>> guest action?
>>
>> Unmapping can happen at any time for any or no reason.  Semantically,
>> it would be correct to only map the pages in response to a p2m violation,
>> but for performance it might be better to map the pages eagerly instead.
> 
> That's an implementation detail, you can certainly map the pages
> eagerly, or even map multiple contiguous pages as a result of a single
> p2m violation.
> 
> I would focus on making a functioning prototype first, performance
> comes afterwards.

Makes sense.

>>>> Furthermore,
>>>> if a page fault happens while the page is unmapped, KVM will try
>>>> to bring the pages back into memory so the guest can access it.
>>>
>>> You could likely handle this in Xen in the following way:
>>>
>>>  - A device model will get p2m violations forwarded, as it's the same
>>>    model that's used to handle emulation of device MMIO.  You will
>>>    need to register an ioreq server to request those faults to be
>>>    forwarded, I think the hardware domain kernel will handle those?
>>>
>>>  - Allow ioreqs to signal to Xen that a guest operation must be
>>>    retried.  IOW: resume guest execution without advancing the IP.
>>>
>>> I think this last bit is the one that will require changes to Xen, so
>>> that you can add a type of ioreq reply that implies a retry from the
>>> guest context.
>> I’m not actually sure if this is needed, though it would be nice.  It
>> might be possible for Xen to instead emulate the current instruction and
>> continue, with the ioreq server just returning the current value of the
>> pages.
> 
> You can, indeed, but it's cumbersome?  You might have to map the page
> in the context of the entity that implements the ioreq server to
> access the data.  Allowing retries would be more generic, and reduce
> the code in the ioreq server handler, that would only map the page
> to the guest p2m and request a retry.

Yeah, it is cumbersome indeed.

>> What I’m more concerned about is being able to provide a page
>> into the p2m so that the *next* access doesn’t fault, and being able
>> to remove that page from the p2m so that the next access *does* fault.
> 
> Maybe I'm not getting the question right, all Xen modifications to the
> p2m take immediate effect.  By the time a XEN_DOMCTL_memory_mapping
> hypercall returns the operation would have taken effect.

Ah, that makes sense.  When revoking access, can XEN_DOMCTL_iomem_permission
and XEN_DOMCTL_memory_mapping fail even if the parameters are correct and
the caller has enough permissions, or will they always succeed?

>> Are there any hypercalls that can be used for these operations right
>> now?
> 
> With some trickery you could likely use XEN_DOMCTL_memory_mapping to
> add and remove those pages.  You will need calls to
> XEN_DOMCTL_iomem_permission beforehand so that you grant the receiving
> domain permissions to access those (and of course the granting domain
> needs to have full access to them).
> 
> This is no ideal if mapping RAM pages, AFAICT there are no strict
> checks that the added page is not RAM, but still you will need to
> handle RAM pages as IOMEM so and grant them using
> XEN_DOMCTL_iomem_permission which is not great.  Also note that this
> is a domctl, so not stable.  It might however be enough for a
> prototype.

Unfortunately this won’t work if the backend is a PVH domain, as a PVH
domain doesn’t know its own MFNs.  It also won’t work for deprivileged
backends because XEN_DOMCTL_iomem_permission is subject to XSA-77.

> Long term I think we want to expand XENMEM_add_to_physmap{,_batch} to
> handle this use-case.

That would indeed be better.

>> If not, which Xen functions would one use to implement them?
>> Some notes:
>>
>> - The p2m might need to be made to point to a PCI BAR or system RAM.
>>   The guest kernel and host userspace don’t know which, and in any
>>   case don’t need to care.  The host kernel knows, but I don’t know
>>   if the information is exposed to the Xen driver.
> 
> Hm, as said above, while you could possible handle RAM as IOMEM, it
> has the slight inconvenience of having to add such RAM pages to the
> d->iomem_caps rangeset for XEN_DOMCTL_memory_mapping to succeed.
> 
> From a guest PoV, it doesn't matter if the underlying page is RAM or
> MMIO, as long as it's mapped in the p2m.

Understood, thanks!

>> - If the p2m needs to point to system RAM, the RAM will be memory
>>   that belongs to the backend.
>>
>> - If the p2m needs to point to a PCI BAR, it will initially need
>>   to point to a real PCI device that is owned by the backend.
> 
> As long as you give the destination domain access to the page using
> XEN_DOMCTL_iomem_permission prior to the XEN_DOMCTL_memory_mapping
> call it should work.
> 
> How does this work for device DMA accesses?  If the device is assigned
> to the backend domain (and thus using the backend domain IOMMU context
> entry and page-tables) DMA accesses cannot be done against guest
> provided addresses, there needs to be some kind of translation layer
> that filters commands?

Thankfully, this is handled by the backend.

> My initial recommendation would be to look into what you can do with
> the existing XEN_DOMCTL_iomem_permission and XEN_DOMCTL_memory_mapping
> hypercalls.

I think this would be suitable for a prototype but not for production.

>> - The switch from “emulated MMIO” to “MMIO or real RAM” needs to
>>   be atomic from the guest’s perspective.
> 
> Updates of p2m PTEs are always atomic.
That’s good.

Xenia, would it be possible for AMD to post whatever has been
implemented so far?  I think this would help a lot, even if it
is incomplete.
-- 
Sincerely,
Demi Marie Obenour (she/her/hers)

Attachment: OpenPGP_0xB288B55FFF9C22C1.asc
Description: OpenPGP public key

Attachment: OpenPGP_signature.asc
Description: OpenPGP digital signature