[PATCH v2 00/10] Device tree based NUMA support for Arm - Part#1

[Wei Chen <wei.chen@xxxxxxx>]

(The Arm device tree based NUMA support patch set contains 35
patches. In order to make stuff easier for reviewers, I split
them into 3 parts:
1. Preparation. I have re-sorted the patch series. And moved
   independent patches to the head of the series.
2. Move generically usable code from x86 to common.
3. Add new code to support Arm.

This series only contains the first part patches.
The full patch list can be found in this cover letter.)

Xen memory allocation and scheduler modules are NUMA aware.
But actually, on x86 has implemented the architecture APIs
to support NUMA. Arm was providing a set of fake architecture
APIs to make it compatible with NUMA awared memory allocation
and scheduler.

Arm system was working well as a single node NUMA system with
these fake APIs, because we didn't have multiple nodes NUMA
system on Arm. But in recent years, more and more Arm devices
support multiple nodes NUMA system.

So now we have a new problem. When Xen is running on these Arm
devices, Xen still treat them as single node SMP systems. The
NUMA affinity capability of Xen memory allocation and scheduler
becomes meaningless. Because they rely on input data that does
not reflect real NUMA layout.

Xen still think the access time for all of the memory is the
same for all CPUs. However, Xen may allocate memory to a VM
from different NUMA nodes with different access speeds. This
difference can be amplified in workloads inside VM, causing
performance instability and timeouts. 

So in this patch series, we implement a set of NUMA API to use
device tree to describe the NUMA layout. We reuse most of the
code of x86 NUMA to create and maintain the mapping between
memory and CPU, create the matrix between any two NUMA nodes.
Except ACPI and some x86 specified code, we have moved other
code to common. In next stage, when we implement ACPI based
NUMA for Arm64, we may move the ACPI NUMA code to common too,
but in current stage, we keep it as x86 only.

This patch serires has been tested and booted well on one
Arm64 NUMA machine and one HPE x86 NUMA machine.

---
Part1 v1->v2:
1. Move independent patches from later to early of this series.
2. Drop the copy of EFI stub.c from Arm. Share common codes of
   x86 EFI stub for Arm.
3. Use CONFIG_ARM_EFI to replace CONFIG_EFI and remove help text
   and make CONFIG_ARM_EFI invisible.
4. Use ASSERT to replace VIRTUAL_BUG_ON in phys_to_nid.
5. Move MAX_NUMNODES from xen/numa.h to asm/numa.h for x86.
6. Extend the description of Arm's workaround for reserve DMA
   allocations to avoid the same discussion every time for
   arch_have_default_dmazone.
7. Update commit messages.

Wei Chen (10):
  xen/arm: Print a 64-bit number in hex from early uart
  xen/x86: move reusable EFI stub functions from x86 to common
  xen/arm: add CONFIG_ARM_EFI to stub EFI API
  xen/arm: Keep memory nodes in device tree when Xen boots from EFI
  xen/x86: Use ASSERT instead of VIRTUAL_BUG_ON for phys_to_nid
  xen: introduce an arch helper for default dma zone status
  xen: decouple NUMA from ACPI in Kconfig
  xen/arm: use !CONFIG_NUMA to keep fake NUMA API
  xen/x86: use paddr_t for addresses in NUMA node structure
  xen/x86: add detection of memory interleaves for different nodes

PART2:
  xen/x86: introduce a helper to update memory hotplug end
  xen/x86: Use enumerations to indicate NUMA status
  xen/x86: move generically usable NUMA code from x86 to common
  xen/x86: use arch_get_memory_map to get information from E820 map
  xen/x86: move NUMA scan nodes codes from x86 to common
  xen/x86: rename bad_srat to numa_bad

PART3:
  xen: rename acpi_scan_nodes to numa_scan_nodes
  xen: introduce a Kconfig option to configure NUMA nodes number
  xen/arm: use NR_MEM_BANKS to override default NR_NODE_MEMBLKS
  xen/arm: implement helpers to get and update NUMA status
  xen/arm: implement node distance helpers for Arm
  xen/arm: use arch_get_memory_map to memory bank from bootinfo
  xen/arm: build NUMA cpu_to_node map in dt_smp_init_cpus
  xen/arm: Add boot and secondary CPU to NUMA system
  xen/arm: stub mem_hotplug_update_boundary for Arm
  xen/arm: introduce a helper to parse device tree processor node
  xen/arm: introduce a helper to parse device tree memory node
  xen/arm: introduce a helper to parse device tree NUMA distance map
  xen/arm: unified entry to parse all NUMA data from device tree
  xen/arm: keep guest still be NUMA unware
  xen/arm: enable device tree based NUMA in system init
  xen/arm: implement a dummy 1:1 node to pxm mapping
  xen/arm: use CONFIG_NUMA to gate node_online_map in smpboot
  xen/arm: Provide Kconfig options for Arm to enable NUMA
  docs: update numa command line to support Arm

 xen/arch/arm/Kconfig                    |  5 ++
 xen/arch/arm/Makefile                   |  2 +-
 xen/arch/arm/arm64/head.S               | 12 ++--
 xen/arch/arm/bootfdt.c                  |  8 ++-
 xen/arch/arm/efi/Makefile               |  5 ++
 xen/arch/arm/efi/efi-boot.h             | 25 ---------
 xen/arch/arm/include/asm/numa.h         |  6 ++
 xen/arch/x86/Kconfig                    |  2 +-
 xen/arch/x86/efi/Makefile               |  4 +-
 xen/arch/x86/efi/{stub.c => stub-x86.c} | 37 -------------
 xen/arch/x86/include/asm/config.h       |  1 -
 xen/arch/x86/include/asm/numa.h         | 15 ++---
 xen/arch/x86/numa.c                     | 30 +++++-----
 xen/arch/x86/srat.c                     | 74 +++++++++++++++++++------
 xen/common/Kconfig                      |  3 +
 xen/common/efi/stub.c                   | 38 +++++++++++++
 xen/common/page_alloc.c                 |  2 +-
 xen/drivers/acpi/Kconfig                |  3 +-
 xen/drivers/acpi/Makefile               |  2 +-
 xen/include/xen/numa.h                  |  2 +
 20 files changed, 162 insertions(+), 114 deletions(-)
 rename xen/arch/x86/efi/{stub.c => stub-x86.c} (71%)
 create mode 100644 xen/common/efi/stub.c

-- 
2.25.1