Re: [PATCH 05/16] xen/percpu: don't initialize percpu on resume

[Julien Grall <julien@xxxxxxx>]

(+ Juergen)

Hi Mykola,

On 05/03/2025 09:11, Mykola Kvach wrote:
> Invocation of the CPU_UP_PREPARE notification
> on ARM64 during resume causes a crash:
>
> (XEN) [  315.807606] Error bringing CPU1 up: -16
> (XEN) [  315.811926] Xen BUG at common/cpu.c:258
> [...]
> (XEN) [  316.142765] Xen call trace:
> (XEN) [  316.146048]    [<00000a0000202264>] enable_nonboot_cpus+0x128/0x1ac 
> (PC)
> (XEN) [  316.153219]    [<00000a000020225c>] enable_nonboot_cpus+0x120/0x1ac 
> (LR)
> (XEN) [  316.160391]    [<00000a0000278180>] suspend.c#system_suspend+0x4c/0x1a0
> (XEN) [  316.167476]    [<00000a0000206b70>] 
> domain.c#continue_hypercall_tasklet_handler+0x54/0xd0
> (XEN) [  316.176117]    [<00000a0000226538>] 
> tasklet.c#do_tasklet_work+0xb8/0x100
> (XEN) [  316.183288]    [<00000a0000226920>] do_tasklet+0x68/0xb0
> (XEN) [  316.189077]    [<00000a000026e120>] domain.c#idle_loop+0x7c/0x194
> (XEN) [  316.195644]    [<00000a0000277638>] shutdown.c#halt_this_cpu+0/0x14
> (XEN) [  316.202383]    [<0000000000000008>] 0000000000000008
>
> Freeing per-CPU areas and setting __per_cpu_offset to INVALID_PERCPU_AREA
> only occur when !park_offline_cpus and system_state is not SYS_STATE_suspend.
> On ARM64, park_offline_cpus is always false, so setting __per_cpu_offset to
> INVALID_PERCPU_AREA depends solely on the system state.
>
> If the system is suspended, this area is not freed, and during resume, an error
> occurs in init_percpu_area, causing a crash because INVALID_PERCPU_AREA is not
> set and park_offline_cpus remains 0:
>
>      if ( __per_cpu_offset[cpu] != INVALID_PERCPU_AREA )
>          return park_offline_cpus ? 0 : -EBUSY;
>
> It appears that the same crash can occur on x86 if park_offline_cpus is set
> to 0 during Xen suspend.
I am rather confused. Looking at the x86 code, it seems 
park_offline_cpus is cleared for AMD platforms. So are you saying the 
suspend/resume doesn't work on AMD?
I am also CCing Juergen because he originally add the check to the 
system_state in 2019. Maybe he will remember why CPU_UP_PREPARE was not 
changed.
> Signed-off-by: Mykyta Poturai <mykyta_poturai@xxxxxxxx>
> Signed-off-by: Mykola Kvach <mykola_kvach@xxxxxxxx>
> ---
> These changes were introduced in V2 inside
> "xen: don't free percpu areas during suspend" patch.
> ---
>   xen/common/percpu.c | 3 ++-
>   1 file changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/xen/common/percpu.c b/xen/common/percpu.c
> index e4e8b7bcab..83dca7edd6 100644
> --- a/xen/common/percpu.c
> +++ b/xen/common/percpu.c
> @@ -74,7 +74,8 @@ static int cf_check cpu_percpu_callback(
>       switch ( action )
>       {
>       case CPU_UP_PREPARE:
> -        rc = init_percpu_area(cpu);
> +        if ( system_state != SYS_STATE_resume )
> +            rc = init_percpu_area(cpu);
>           break;
>   
>       case CPU_UP_CANCELED:
Cheers,

--
Julien Grall