Re: Excluding init_on_free for pages for initial balloon down (Xen)

[Jan Beulich <jbeulich@xxxxxxxx>]

On 02.03.2026 12:01, Marek Marczykowski-Górecki wrote:
> On Mon, Mar 02, 2026 at 09:40:29AM +0100, David Hildenbrand (Arm) wrote:
>> On 3/2/26 07:36, Jürgen Groß wrote:
>>> On 01.03.26 16:04, Marek Marczykowski-Górecki wrote:
>>>> Hi,
>>>>
>>>> Some time ago I made a change to disable scrubbing pages that are
>>>> ballooned out during system boot. I'll paste the whole commit message as
>>>> it's relevant here:
>>>>
>>>>      197ecb3802c0 xen/balloon: add runtime control for scrubbing
>>>> ballooned out pages
>>>>
>>>>      Scrubbing pages on initial balloon down can take some time,
>>>> especially
>>>>      in nested virtualization case (nested EPT is slow). When HVM/PVH
>>>> guest is
>>>>      started with memory= significantly lower than maxmem=, all the extra
>>>>      pages will be scrubbed before returning to Xen. But since most of
>>>> them
>>>>      weren't used at all at that point, Xen needs to populate them first
>>>>      (from populate-on-demand pool). In nested virt case (Xen inside KVM)
>>>>      this slows down the guest boot by 15-30s with just 1.5GB needed
>>>> to be
>>>>      returned to Xen.
>>>>           Add runtime parameter to enable/disable it, to allow
>>>> initially disabling
>>>>      scrubbing, then enable it back during boot (for example in
>>>> initramfs).
>>>>      Such usage relies on assumption that a) most pages ballooned out
>>>> during
>>>>      initial boot weren't used at all, and b) even if they were, very few
>>>>      secrets are in the guest at that time (before any serious userspace
>>>>      kicks in).
>>>>      Convert CONFIG_XEN_SCRUB_PAGES to CONFIG_XEN_SCRUB_PAGES_DEFAULT
>>>> (also
>>>>      enabled by default), controlling default value for the new runtime
>>>>      switch.
>>>>
>>>> Now, I face the same issue with init_on_free/init_on_alloc (not sure
>>>> which one applies here, probably the latter one), which several
>>>> distributions enable by default. The result is (see timestamps):
>>>>
>>>>      [2026-02-24 01:12:55] [    7.485151] xen:balloon: Waiting for
>>>> initial ballooning down having finished.
>>>>      [2026-02-24 01:14:14] [   86.581510] xen:balloon: Initial
>>>> ballooning down finished.
>>>>
>>>> But here the situation is a bit more complicated:
>>>> init_on_free/init_on_alloc applies to any pages, not just those for
>>>> balloon driver. I see two approaches to solve the issue:
>>>> 1. Similar to xen_scrub_pages=, add a runtime switch for
>>>>     init_on_free/init_on_alloc, then force them off during boot, and
>>>>     re-enable early in initramfs.
>>>> 2. Somehow adjust balloon driver to bypass init_on_alloc when ballooning
>>>>     a page out.
>>>>
>>>> The first approach is likely easier to implement, but also has some
>>>> drawbacks: it may result in some kernel structures that are allocated
>>>> early to remain with garbage data in uninitialized places. While it may
>>>> not matter during early boot, such structures may survive for quite some
>>>> time, and maybe attacker can use them later on to exploit some other
>>>> bug. This wasn't really a concern with xen_scrub_pages, as those pages
>>>> were immediately ballooned out.
>>>>
>>>> The second approach sounds architecturally better, and maybe
>>>> init_on_alloc could be always bypassed during balloon out? The balloon
>>>> driver can scrub the page on its own already (which is enabled by
>>>> default). That of course assumes the issue is only about init_on_alloc,
>>>> not init_on_free (or both) - which I haven't really confirmed yet...
>>>> If going this way, I see the balloon driver does basically
>>>> alloc_page(GFP_BALLOON), where GFP_BALLOON is:
>>>>
>>>>      /* When ballooning out (allocating memory to return to Xen) we
>>>> don't really
>>>>         want the kernel to try too hard since that can trigger the oom
>>>> killer. */
>>>>      #define GFP_BALLOON \
>>>>          (GFP_HIGHUSER | __GFP_NOWARN | __GFP_NORETRY | __GFP_NOMEMALLOC)
>>>>
>>>> Would that be about adding some new flag here? Or maybe there is already
>>>> one for this purpose?
>>>
>>> There doesn't seem to be a flag for that.
>>>
>>> But I think adding a new flag __GFP_NO_INIT and testing that in
>>> want_init_on_alloc() _before_ checking CONFIG_INIT_ON_ALLOC_DEFAULT_ON
>>> would be a sensible approach.
>>
>> People argued against such flags in the past, because it will simply get
>> abused by arbitrary drivers that want to be smart.
> 
> Could it be named differently to discourage such usage? Maybe
> __GFP_BALLOON_OUT ?
> 
>> Whatever leaves the buddy shall be zeroed out. If there is a
>> double-zeroing happen, the latter could get optimized out by checking
>> something like user_alloc_needs_zeroing().
>>
>> See mm/huge_memory.c:vma_alloc_anon_folio_pmd() as an example where we
>> avoid double-zeroing.
> 
> It isn't just reducing double-zeroing to single zeroing. It's about
> avoiding zeroing such pages at all. If a domU is started with
> populate-on-demand, many (sometimes most) of its pages are populated in
> EPT.

ITYM "unpopulated in EPT"?

Jan

> The idea of PoD is to start guest with high static memory size, but
> low actual allocation and fake it until balloon driver kicks in and make
> the domU really not use more pages than it has. When balloon driver try
> to return those pages to the hypervisor, normally it would just take
> unallocated page one by one and made Linux not use them. But if _any_
> zeroing is happening, each page first needs to be mapped to the guest by
> the hypervisor (one trip through EPT), just to be removed from them a
> moment later...
> 
>>>> Any opinions?
>>>
>>> You are aware of the "init_on_alloc" boot parameter? So if this is fine
>>> for you, you could just use approach 1 above without any kernel patches
>>> needed.
>>
>> I don't think init_on_alloc can be enabled after boot. IIUC, 1) would
>> require a runtime switch.
> 
> Indeed.
>