Re: [Xen-devel] [RFC v2] xen/arm: Suspend to RAM Support in Xen for ARM

[Peng Fan <peng.fan@xxxxxxx>]

Hi Mirela,

Good to know that you are working suspend/resume support. Currently we are also 
trying
to support this on i.MX8, just wonder do you have any open source available to
support suspend to ram?

> +
> +Suspend to RAM (in the following text 'suspend') for ARM in Xen should
> +be coordinated using ARM PSCI standard [1].
> +
> +Ideally, EL1/2 should suspend in the following order:
> +1) Unprivileged guests (DomUs) suspend
> +2) Privileged guest (Dom0) suspends
> +3) Xen suspends
> +
> +However, suspending unprivileged guests is not mandatory for suspending
> +Dom0 and Xen. System suspend initiated by Dom0 (step 2) is considered
> +to be an ultimate decision to suspend the physical machine. Suspending
> +of Xen (step 3) is triggered whenever Dom0 completes suspend. Xen
> +suspend leads to the full suspend of EL2.
> +
> +If an unprivileged guest is not suspended at the moment when Dom0
> +initiates its own suspend, the guest will be paused on Xen's suspend
> +and unpaused on Xen's resume. That way, a guest which doesn't have
> +power management support cannot prevent the physical system from
> +suspending when the decision to suspend is made by privileged software
> (Dom0).
> +
> +Each guest in the system is responsible for suspending the devices it owns.
> +If a guest does not suspend a device, the device will continue to
> +operate as it is configured at the moment when the system suspends. If
> +a device triggers an interrupt while the physical system is suspended, the
> system will resume.
> +
> +It is recommended for an unprivileged guest to participate in power
> +management in the following scenario:
> +Assume unprivileged guest owns a device which will trigger interrupt at
> +some point. This interrupt will wake-up the system. If such a behavior
> +is not wanted, coordination between Dom0 and the guest is required in
> +order to inform the guest about the intended physical system suspend.
> +Then, the guest will have a chance to suspend the device or respond to the
> request in an abort fashion.
> +
> +Since this proposal is focused on implementing PSCI-based suspend
> +mechanisms in Xen, communication with or among the guests is not covered by
> this document.
> +The order of suspending the guests is assumed to be guaranteed by the
> +software running in EL1.
> +
> +This document covers the following:
> +1) Mechanism for suspending/resuming a guest:
> +     1.1) Suspend is initiated by the guest
> +     1.2) Resume is initiated by a device interrupt
> +2) Mechanism for pausing/unpausing running guests when Dom0 suspends

Will this take care of passthroughed devices for DomU?

Thanks,
Peng.

> +3) Mechanism for suspending/resuming Xen when Dom0 completes suspend
> +4) Resuming from any state on a wake-up event (device interrupt):
> +     4.1) Resume DomU on wake-up event when Dom0 is still running
> +     4.2) Resume DomU on wake-up event when Xen is suspended
> +     4.3) Resume Dom0 on wake-up event
> +
> +Mechanisms enumerated above will allow different kind of policies and
> +coordination among guests to be implemented in EL1. That is out of the
> +scope of this document.
> +
> +-----------------
> +Suspending Guests
> +-----------------
> +
> +Suspend procedure for a guest consists of the following:
> +1) Suspending devices
> +2) Suspending non-boot CPUs (based on hotplug/PSCI)
> +3) System suspend, performed by the boot CPU
> +
> +Each guest should suspend the devices it owns just like it would when
> +running without Xen.
> +
> +Guests should suspend their non-boot vCPUs using the hotplug mechanism.
> +Virtual CPUs should be put offline using the already implemented PSCI
> +vCPU_OFF call (prefix 'v' is added to distinguish PSCI calls made by
> +guests to Xen, which affect virtual machines; as opposed to PSCI calls
> +made by Xen to the EL3, which can affect power state of the physical
> machine).
> +
> +After suspending its non-boot vCPUs a guest should finalize the suspend
> +by making the vSYSTEM_SUSPEND PSCI call. The resume address is
> +specified by the guest via the vSYSTEM_SUSPEND entry_point_address
> +argument. The vSYSTEM_SUSPEND call is currently not implemented in Xen.
> +
> +It is expected that a guest leaves enabled all interrupts that should
> +wake it up. Other interrupts should be disabled by the guest prior to
> +calling vSYSTEM_SUSPEND.
> +
> +After an unprivileged guest suspends, Xen will not suspend. Xen would
> +suspend only after the Dom0 completes the system suspend.
> +
> +--------------
> +Suspending Xen
> +--------------
> +
> +Xen should start suspending itself upon receiving the vSYSTEM_SUSPEND
> +call from the last running guest (Dom0). At that moment all physical
> +CPUs are still online (taking offline a vCPU or suspending a VM does not 
> affect
> physical CPUs).
> +Xen shall now put offline the non-boot pCPUs by making the CPU_OFF PSCI
> +call to EL3. The CPU_OFF PSCI function is currently not implemented in Xen.
> +
> +After putting offline the non-boot cores Xen must save the context and
> +finalize suspend by invoking SYSTEM_SUSPEND PSCI call, which is passed to 
> EL3.
> +The resume point of Xen is specified by the entry_point_address
> +argument of the SYSTEM_SUSPEND call. The SYSTEM_SUSPEND function and
> +context saving is not implemented in Xen for ARM today.
> +
> +------------
> +Resuming Xen
> +------------
> +
> +Xen must be resumed prior to any software running in EL1. Starting from
> +the resume point, Xen should restore the context and resume Dom0. Dom0
> +shall always be resumed whenever Xen resumes.
> +
> +---------------
> +Resuming Guests
> +---------------
> +
> +Resume of the privileged guest (Dom0) is always following the Xen resume.
> +
> +An unprivileged guest shall resume once a device it owns triggers a
> +wake-up interrupt, regardless of whether Xen was suspended when the
> +wake-up interrupt was triggered. If Xen was suspended, it is assumed
> +that Dom0 will be running before the DomU guest starts to resume. The
> +synchronization mechanism to enforce the assumed condition is TBD.
> +
> +If the ARM's GIC was powered down after the ARM subsystem suspended, it
> +is assumed that Xen needs to restore the GIC interface for a VM prior
> +to handing over control to the guest. However, the guest should restore
> +its own context upon entering the resume point, just like it would when
> running without Xen.
> +
> +===============
> +Implementation
> +===============
> +
> +--------
> +Overview
> +--------
> +
> +In order to enable the suspend/resume of VMs and Xen itself, the
> +following PSCI calls have to be implemented and integrated in Xen:
> +1) vSYSTEM_SUSPEND
> +2) CPU_OFF
> +3) SYSTEM_SUSPEND
> +
> +In addition, the following have to be implemented:
> +* Suspend/resume vCPU (triggered by vSYSTEM_SUSPEND call)
> +* Suspend/resume Xen (triggered by vSYSTEM_SUSPEND called by Dom0),
> including:
> +     * Disable wathdog on suspend, enable it on resume
> +     * Pause domains on suspend, unpause them on resume
> +     * Disable non-boot pCPUs on suspend, enable them on resume
> +     * Save/restore of GIC configuration
> +     * Suspend/resume timer
> +     * Save/restore of EL2 context
> +     * Implement resume entry point in Xen, including MMU configuration
> +
> +Implementation details are provided in the sections below. Function
> +names and paths used below are consistent within the document but may
> +not always match the names used in future implementation. Existing
> +functions and paths are named as in Xen source tree.
> +
> +-------------------------------------
> +Suspend/Resume Implementation Details
> +-------------------------------------
> +
> +PSCI Implementation and Integration
> +-----------------------------------
> +vSYSTEM_SUSPEND
> +---------------
> +vSYSTEM_SUSPEND shall be implemented in
> +* do_psci_system_suspend() in arch/arm/vpsci.c
> +* Code independent from PSCI interface will be added in
> +arch/arm/suspend.c
> +
> +The implementation shall include the following steps:
> +* Suspend the current (calling) vCPU. Consists of 2 major steps:
> +1) Reset context of vCPU and save entry point into PC and context ID
> +into X0 (entry point and context ID are provided via vSYSTEM_SUSPEND
> +arguments)
> +2) Block vCPU to ensure that it is not scheduled until it is unblocked
> +by an interrupt.
> +In step 1) above, the context is reset in order to prepare the vCPU for
> +resume, i.e. to save vCPU context that matches reset values as expected
> +by software on resume. This doesn't hold for PC and X0, since the PC
> +contains resume entry point and X0 contains context ID, as defined by PSCI.
> +* If the hardware domain made the call trigger Xen suspend, i.e.
> +  call machine_suspend() which will be implemented in
> +arch/arm/suspend.c  (similar as the machine_restart() is implemented in
> +arch/arm/shutdown.c)
> +
> +The function do_psci_system_suspend() shall be called from
> +* do_trap_psci() in arch/arm/traps.c
> +
> +CPU_OFF (physical CPUs)
> +-----------------------
> +The CPU_OFF function shall be implemented in
> +* call_psci_cpu_off() in arch/arm/psci.c
> +
> +The implementation shall consist just of making the SMC call to EL3.
> +
> +This function needs to be called when Xen generic code disables a non-boot
> CPU.
> +When a CPU is disabled it will loop forever in while loop (stop_cpu()
> +function which is already implemented in xen/arch/arm/smpboot.c). Call
> +to
> +call_psci_cpu_off() shall be made before the CPU enters infinite loop.
> +
> +SYSTEM_SUSPEND (physical)
> +-------------------------
> +The SYSTEM_SUSPEND function shall be implemented in
> +* call_psci_system_suspend() in arch/arm/psci.c
> +
> +The implementation shall consist just of making the SMC call to EL3.
> +The entry_point_address argument of the SMC call needs to be an ARM
> +architecture resume address, which shall be implemented, e.g. as
> +hyp_resume() in arch/arm/arm64/entry.S. The call_psci_system_suspend()
> function does not return.
> +On the resume, the execution flow continues from hyp_resume.
> +
> +The function needs to be called from machine_suspend() to finalize the
> +suspend procedure.
> +
> +------------------
> +Additional Changes
> +------------------
> +
> +Suspend Flow
> +------------
> +The suspend procedure shall be implemented in
> +* machine_suspend() in arch/arm/suspend.c
> +
> +The implementation shall include the following steps:
> +* Move the execution to boot pCPU
> +* Set the system_state variable to SYS_STATE_suspend
> +* Disable watchdog
> +* Freeze domains by calling domain_pause() for each domain
> +* Disable non-boot CPUs by calling disable_nonboot_cpus()
> +* Disable interrupts
> +* Suspend timer
> +* Save GIC context. Shall be implemented in arch/arm/gic.c,
> +  include/asm-arm/gic.h and arch/arm/gic-v2.c (only GICv2 will be supported).
> +* Save CPU context. This shall be implemented in assembly, in
> +hyp_suspend()
> +  in arch/arm/arm64/entry.S. The context consists of callee-saved
> +general
> +  purpose registers, as well as few system registers. Context of
> +registers shall
> +  be saved in a statically allocated structure.
> +* Finalize the suspend by calling call_psci_system_suspend()
> +
> +Resume Flow
> +------------
> +The resume entry point shall be implemented in
> +* hyp_resume() in arch/arm/arm64/entry.S The very beginning of the
> +resume procedure has to be implemented in assembly.
> +It shall contain the following:
> +* Enable the MMU so that the structure containing CPU context which was
> +saved on suspend can be accessed
> +* Restore CPU context (to match the values saved on suspend) and return
> +into C
> +* Set the system_state variable to SYS_STATE_resume
> +* Restore GIC context
> +* Resume timer
> +* Enable interrupts
> +* Enable non-boot CPUs by calling enable_nonboot_cpus()
> +* Thaw domains by calling domain_unpause() for each domain
> +* Enable watchdog
> +* Set the system_state variable to SYS_STATE_active
> +* Resume Dom0
> +
> +==========
> +References
> +==========
> +
> +[1] Power State Coordination Interface (ARM):
> +https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Finfoc
> +enter.arm.com%2Fhelp%2Ftopic%2Fcom.arm.doc.den0022d%2FPower_State
> _Coord
> +ination_Interface_PDD_v1_1_DEN0022D.pdf&data=02%7C01%7Cpeng.fan%4
> 0nxp.c
> +om%7Cb343d128930d44c90f5d08d54963807b%7C686ea1d3bc2b4c6fa92cd99
> c5c30163
> +5%7C0%7C1%7C636495614074885940&sdata=3ycqEZR9XgcqdvrmJKY86aukt
> %2BQv%2BS
> +BSZMxbCrpraEY%3D&reserved=0
> --
> 2.13.0
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@xxxxxxxxxxxxxxxxxxxx
> https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.xe
> nproject.org%2Fmailman%2Flistinfo%2Fxen-devel&data=02%7C01%7Cpeng.fan
> %40nxp.com%7Cb343d128930d44c90f5d08d54963807b%7C686ea1d3bc2b4c6f
> a92cd99c5c301635%7C0%7C0%7C636495614074885940&sdata=YLuJhbx%2B1
> tDvblYbgtOZZBhsG36%2BUhpRc4VpSpHHM%2FU%3D&reserved=0
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel