Re: Excluding init_on_free for pages for initial balloon down (Xen)

[Jürgen Groß <jgross@xxxxxxxx>]

On 01.03.26 16:04, Marek Marczykowski-Górecki wrote:
> Hi,
>
> Some time ago I made a change to disable scrubbing pages that are
> ballooned out during system boot. I'll paste the whole commit message as
> it's relevant here:
>
>      197ecb3802c0 xen/balloon: add runtime control for scrubbing ballooned out 
> pages
>
>      Scrubbing pages on initial balloon down can take some time, especially
>      in nested virtualization case (nested EPT is slow). When HVM/PVH guest is
>      started with memory= significantly lower than maxmem=, all the extra
>      pages will be scrubbed before returning to Xen. But since most of them
>      weren't used at all at that point, Xen needs to populate them first
>      (from populate-on-demand pool). In nested virt case (Xen inside KVM)
>      this slows down the guest boot by 15-30s with just 1.5GB needed to be
>      returned to Xen.
>      
>      Add runtime parameter to enable/disable it, to allow initially disabling
>      scrubbing, then enable it back during boot (for example in initramfs).
>      Such usage relies on assumption that a) most pages ballooned out during
>      initial boot weren't used at all, and b) even if they were, very few
>      secrets are in the guest at that time (before any serious userspace
>      kicks in).
>      Convert CONFIG_XEN_SCRUB_PAGES to CONFIG_XEN_SCRUB_PAGES_DEFAULT (also
>      enabled by default), controlling default value for the new runtime
>      switch.
>
> Now, I face the same issue with init_on_free/init_on_alloc (not sure
> which one applies here, probably the latter one), which several
> distributions enable by default. The result is (see timestamps):
>
>      [2026-02-24 01:12:55] [    7.485151] xen:balloon: Waiting for initial 
> ballooning down having finished.
>      [2026-02-24 01:14:14] [   86.581510] xen:balloon: Initial ballooning down 
> finished.
>
> But here the situation is a bit more complicated:
> init_on_free/init_on_alloc applies to any pages, not just those for
> balloon driver. I see two approaches to solve the issue:
> 1. Similar to xen_scrub_pages=, add a runtime switch for
>     init_on_free/init_on_alloc, then force them off during boot, and
>     re-enable early in initramfs.
> 2. Somehow adjust balloon driver to bypass init_on_alloc when ballooning
>     a page out.
>
> The first approach is likely easier to implement, but also has some
> drawbacks: it may result in some kernel structures that are allocated
> early to remain with garbage data in uninitialized places. While it may
> not matter during early boot, such structures may survive for quite some
> time, and maybe attacker can use them later on to exploit some other
> bug. This wasn't really a concern with xen_scrub_pages, as those pages
> were immediately ballooned out.
>
> The second approach sounds architecturally better, and maybe
> init_on_alloc could be always bypassed during balloon out? The balloon
> driver can scrub the page on its own already (which is enabled by
> default). That of course assumes the issue is only about init_on_alloc,
> not init_on_free (or both) - which I haven't really confirmed yet...
> If going this way, I see the balloon driver does basically
> alloc_page(GFP_BALLOON), where GFP_BALLOON is:
>
>      /* When ballooning out (allocating memory to return to Xen) we don't really
>         want the kernel to try too hard since that can trigger the oom killer. 
> */
>      #define GFP_BALLOON \
>          (GFP_HIGHUSER | __GFP_NOWARN | __GFP_NORETRY | __GFP_NOMEMALLOC)
>
> Would that be about adding some new flag here? Or maybe there is already
> one for this purpose?
There doesn't seem to be a flag for that.

But I think adding a new flag __GFP_NO_INIT and testing that in
want_init_on_alloc() _before_ checking CONFIG_INIT_ON_ALLOC_DEFAULT_ON
would be a sensible approach.

> Any opinions?
You are aware of the "init_on_alloc" boot parameter? So if this is fine
for you, you could just use approach 1 above without any kernel patches
needed.


Juergen

Attachment: OpenPGP_0xB0DE9DD628BF132F.asc
Description: OpenPGP public key

Attachment: OpenPGP_signature.asc
Description: OpenPGP digital signature