Xen project Mailing List

On Tue, May 15, 2018 at 1:44 PM, Mirela Simonovic <mirela.simonovic@xxxxxxxxxx> wrote:

This patch set contains fixes that are required as precondition for suspend to
RAM support, including the CPU hotplug which is required to suspend non-boot
CPUs.
The first two patches in this series:
1) xen/arm64: Added handling of the trapped access to OSLSR register
2) xen/arm: Ignore write to GICD_ISACTIVERn registers (vgic-v2)
are required to avoid Dom0 crashes when Dom0 performs its own suspend. This
patch set does not include the implementation of virtual PSCI system suspend
call that would allow guests to finalize their suspend procedures. This will
be submitted in the following series.

Remaining of the patches are related to enabling CPU hotplug for non-boot
CPUs is resume scenario. CPU hotplug of non-boot CPUs will be used for suspend
to RAM support for ARM. In suspend procedure, the hot-unplug of non-boot CPUs
will be triggered with disable_nonboot_cpus(), while the hotplug is triggered
with enable_nonboot_cpus(). Using these calls, the physical non-boot CPUs could
be powered down/up on suspend/resume, respectively, if the underlying firmware
allows so. Calls to enable/disable_nonboot_cpus() functions currently do not
exist in Xen ARM code. This will be added with the suspend to RAM support for
ARM.

When non-boot pCPUs are hot-unplugged their interrupts are migrated to the boot
pCPU. This series also includes a fix that would restore the interrupts affinity
once non-boot pCPUs are hotplugged. Here only SPIs used by guests are covered.
Migration of Xen internal SPIs is not covered. According to my understanding
Xen internal SPIs are routed to the boot CPU which initializes the respective
devices. Therefore, there is no need to migrate Xen internal SPIs.

The code is tested on Xilinx Zynq UltraScale+ MPSoC/ZCU102 board (includes
physical power down/up of non-boot CPUs). The testing requires additional
patches for issuing system suspend. These patches and instructions for testing
will be submitted later, when we get closer to the final version of the series.

---
Changes in v2:
-Rename cover-letter title and emphasize that 2 patches from this series are not
specific to CPU hotplug (my initial fault, splitting it now could be confusing)
-Fix cover-letter explanations
-Address all the issues and comments as discussed on mailing list for v1
-Add 3 patches to ensure that suspend/resume does not cause any memory leaks.
All the memory allocated when a CPU was hotplugged is now freed when the CPU is
hot-unplugged.
-Remove from the v1 series the patch which incorrectly dealt with an issue:
[PATCH 4/7] xen/arm: When CPU dies, free percpu area immediatelly
One solution to the issue addressed by the patch above is to add rcu_barrier()
prior to calling enable_nonboot_cpus() during the suspend. This is how it is
done in x86 suspend implementation. Until the discussion here
https://lists.xenproject.org/archives/html/xen-devel/2018-04/msg01199.html
doesn't conclude differently, I need to assume that adding rcu_barrier() prior
to calling enable_nonboot_cpus() as it is done for x86 is the right way to go.
Therefore, the fix to the issue will be part of the suspend to RAM series.

Changes in v3:
-Add acked-by where needed
-Fix CPU_OFF PSCI implementation (physical interface)
-Use notifiers to implement freeing memory and releasing interrupts on CPU
hotplug
-Use notifier to trigger setup of virtual paging for non-boot CPUs on CPU
hotplug
-Add enabling errata workarounds on CPU hotplug, also based on a notifier
-Remove patch:
[PATCH v2 10/10] xen/arm: Call check_local_cpu_errata for secondary CPU only on boot

Changes in v4:
-Add acked-by/reviewed-by where needed
-Cleanup: use smp_processor_id() instead of get_processor_id(), fixed
indentation, add includes alphabetically, add newline before return, etc.
-Disable timers prior to releasing timer interrupts
-Initialize cpu_smpboot notifier at presmp_init rather than init phase
-In the last patch of the series errata notifier now returns an error

---
CC: Stefano Stabellini <sstabellini@xxxxxxxxxx>
CC: Julien Grall <julien.grall@xxxxxxx>
CC: George Dunlap <george.dunlap@xxxxxxxxxxxxx>
CC: Dario Faggioli <dfaggioli@xxxxxxxx>
---

Mirela Simonovic (10):
xen/arm64: Added handling of the trapped access to OSLSR register
xen/arm: Ignore write to GICD_ISACTIVERn registers (vgic-v2)
xen/arm: Implement CPU_OFF PSCI call (physical interface)
xen/arm: Remove __initdata and __init to enable CPU hotplug
xen/arm: Setup virtual paging for non-boot CPUs on hotplug/resume
xen/common: Restore IRQ affinity when hotplugging a pCPU
xen/arm: Release maintenance interrupt when CPU is hot-unplugged
xen/arm: Disable timers and release their interrupts on CPU hot-unplug
xen/arm: Free memory allocated for sibling/core maps on CPU hot-unplug
xen/arm: Enable errata for secondary CPU on hotplug after the boot

xen/arch/arm/arm64/smpboot.c | 2 +-
xen/arch/arm/arm64/vsysreg.c | 3 ++-
xen/arch/arm/cpuerrata.c | 49 +++++++++++++++++++++++++++++++++++++
xen/arch/arm/cpufeature.c | 29 ++++++++++++++++++++++
xen/arch/arm/gic.c | 31 +++++++++++++++++++++++
xen/arch/arm/irq.c | 2 +-
xen/arch/arm/p2m.c | 53 ++++++++++++++++++++++++++++++++++++----
xen/arch/arm/processor.c | 2 +-
xen/arch/arm/psci.c | 13 ++++++++++
xen/arch/arm/smpboot.c | 42 +++++++++++++++++++++++++++++--
xen/arch/arm/time.c | 45 ++++++++++++++++++++++++++++++++++
xen/arch/arm/vgic-v2.c | 2 ++
xen/common/schedule.c | 4 +++
xen/include/asm-arm/cpufeature.h | 1 +
xen/include/asm-arm/procinfo.h | 4 +--
xen/include/asm-arm/psci.h | 1 +
16 files changed, 270 insertions(+), 13 deletions(-)

--
2.13.0

Re: [Xen-devel] [PATCH v4 00/10] xen/arm64: Suspend preconditions and CPU hotplug fixes