Re: [PATCH] xen/x86: Change stub page freeing to fix smt=0

[Andrew Cooper <andrew.cooper3@xxxxxxxxxx>]

On 26/05/2026 9:31 pm, Jason Andryuk wrote:
> A single stubs page is initialized with 0xcc and re-used, with multiple
> CPUs each using a portion of the shared page.  In cpu_smpboot_free(),
> each stubs area is checked against 0xcc.  When all are set to 0xcc, the
> page is freed.
>
> Booting a system with smt=0, CPU0 is initially setup, allocating the
> stubs page and initializing to 0xcc.  When more CPUs are brought up,
> CPU1 is initialized and then immediately brough offline as it is the
> sibling of CPU0.  Since the page was initially memset with 0xcc,
> cpu_smpboot_free() finds all stubs as 0xcc and frees the page.
> However, the page is still assigned to CPU0 and continues to be assigned
> to other CPUs.

It's more complicated than this.

With CONFIG_PV (and !opt_fred in 4.22 which is perhaps newer than you're
testing), the LSTAR and CSTAR stubs guarantee that the 0xcc's are
overwritten with real instructions.

In !CONFIG_PV, the 0xcc's only get overwritten by the exception recovery
selftests (CPU0 only, and gated on CONFIG_SELF_TESTS), and "complicated"
instructions in the emulator (which in your safety environment, you
likely have compiled out).

So, in your environment, I think you probably can exclude the stubs
entirely and trim even more LoC.

>
> Meanwhile the page can be reallocated, which can lead to misbehavior.
> The particular instance was the stubs page re-used as a page table which
> later faulted when the entry was all 0xcc.
>
> Change to initializing the page as 0xd6/STUB_BUF_FREE, and initializing
> individual stubs as 0xcc/STUB_BUF_USED.  0xd6 now indicates unused, and
> 0xcc indicates used/assigned.  When freeing a CPU, the stub is set to
> 0xd6, and the page is freed if all stubs are 0xd6.  Initializing with
> STUB_BUF_FREE lets cpu_smpboot_free() a page that was only ever
> partially used.
>
> 0xd6/UDB is a 1 byte invalid opcode, which is similar to the existing
> use of 0xcc.  0xd6 is used to identify bug frames, but the stub addr
> (e.g. 0xffff82d07fffe000) fails the is_active_kernel_text() check.  It
> should be okay to use here.
>
> Fixes: 7a66ac8d1633 ("x86: move syscall trampolines off the stack")
> Signed-off-by: Jason Andryuk <jason.andryuk@xxxxxxx>
> ---
> It would be nice to use get_page()/put_page() to let count_info handle
> reference counting, but they require an owning domain.
>
> The listed Fixes introduced the use of 0xcc, but the smt commit may have
> made it more problematic.
> Fixes: d8f974f1a646 ("x86: command line option to avoid use of secondary 
> hyper-threads")

Honestly, I dislike all of this "try to free data for going-offline
CPUs".  It is both complex and a non-stop source of bugs for tantamount
to 0 benefit.

On x86, we must boot all CPUs we find in the MADT.  You're seeing this
behaviour already.  This is because if an #MC hits any group of CPUs
where any CR4.MCE=0, it's an instant reset.

For this reason, firmware doesn't hand APs over to the OS in the
Wait-for-SIPI state (which resets CR4 to 0); they're in MWAIT or IO-wait
typically these days, using firmware provided stacks.  But firmware
cannot handle an #MC intended for the OS, so the OS must set up stacks
and at least an NMI and #MC handler even for those CPUs not wanting to run.

This what park_offline_cpus is trying to do, and while it's set for
Intel and clear for AMD, I'm pretty sure this is a bug on AMD because
you can still get MCEs with core-scope groups.


Beyond that,  smt=0 is an emergency bodge for speculation safety, which
is always better done by changing SMT settings in the firmware. 
xen-hptool is useful for testing but it's not a thing anyone uses in a
production system.

ACPI CPU hot-add does exist in virtual environments, but hot-remove is
theoretical at best.  I've not seen any evidence of ACPI hotplug
actually working on Xen, and I think the chances that it does are slim;
it requires AML execution, and is right in the middle of the split-brain
problem with physical vs virtual details that dom0 suffers.


So, lets just allocate the stubs and "leak" them in testing scenarios. 
It removes bugs and removes code, and has no effect on well-configured
systems (where cpu offline is not used in practice).

~Andrew