Re: [PATCH] memory: arrange to conserve on DMA reservation

[Roger Pau Monné <roger.pau@xxxxxxxxxx>]

On Wed, Jun 18, 2025 at 02:04:12PM +0200, Jan Beulich wrote:
> On 16.06.2025 19:23, Roger Pau Monné wrote:
> > On Mon, Jun 16, 2025 at 06:02:07PM +0200, Jan Beulich wrote:
> >> On 16.06.2025 17:41, Roger Pau Monné wrote:
> >>> On Mon, Jun 16, 2025 at 05:20:45PM +0200, Jan Beulich wrote:
> >>>> On 16.06.2025 16:46, Roger Pau Monné wrote:
> >>>>> One question I have though, on systems with a low amount of memory
> >>>>> (let's say 8GB), does this lead to an increase in domain construction
> >>>>> time due to having to fallback to order 0 allocations when running out
> >>>>> of non-DMA memory?
> >>>>
> >>>> It'll likely be slower, yes, but I can't guesstimate by how much.
> >>>
> >>> Should there be some way to control this behavior then?  I'm mostly
> >>> thinking about client systems like Qubes where memory is likely
> >>> limited, and the extra slowness to create VMs could become
> >>> noticeable?
> >>
> >> What kind of control would you be thinking of here? Yet another command
> >> line option?
> > 
> > I guess that would be enough.  I think we need a way to resort to the
> > previous behavior if required,
> 
> Thinking about it, there already is "dma_bits=". Simply setting this low
> enough would have largely the same effect as yet another new command line
> option. Thoughts?
> 
> > and likely a CHANGELOG entry to notice the change.
> 
> Hmm, not sure here. This is too small imo, and really an implementation
> detail.

I think it's possible to have a noticeable effect for some use-cases,
so it should be listed on CHANGELOG in case users see divergence from
previous Xen versions.  That can given them a hint at what needs
adjusting.

It's not only the possible domain creation overhead: if the guest
being created is HVM with p2m, the resulting p2m will also be
shattered leading to worse guest performance.

> > Overall, would it be possible to only include the flag if we know
> > there's non-DMA memory available to allocate?  Otherwise we are
> > crippling allocation performance when there's only DMA memory left.
> 
> Imo trying to determine this would only make sense if the result can
> then be relied upon. To determine we'd need to obtain the heap lock,
> and we'd need to not drop it until after the allocation(s) were done.
> I think that's far away from being a realistic option.

Couldn't Xen do best effort?

A mismatch won't be fatal anyway.  Error cases: you set MEMF_no_dma and
there's no non-DMA memory to allocate from, in which case it's the
same that's unconditionally done.  Other possible failure is you don't
set non-DMA and there's still non-DMA memory that's not consumed,
which is the current behavior.  IMO it would be safer in terms of
behavior change to do:

if ( a->extent_order && <non-DMA memory available> )
   a->memflags |= MEMF_no_dma;

I think that's likely to be less intrusive and won't lead to system
malfunctions, even if strictly it's (kind of?) a TOCTOU race.

> > That also raises the question whether it's an acceptable trade-off to
> > possibly shatter p2m super pages (that could be used if allocating
> > from the DMA pool) at the expense of not allocating from the DMA pool
> > until there's non-DMA memory left.
> 
> This being an acceptable tradeoff is imo an implicit pre-condition of
> adding such a heuristic. For the system as a whole, exhausting special
> purpose memory is likely worse than some loss of performance. Plus as
> said above, people valuing performance more can reduce the "DMA pool".

That's an acceptable workaround, but it might be helpful to note this
in the commit message, as otherwise it will get lost in xen-devel.

Thanks, Roger.