[RFC PATCH 1/1] static-memory: allow skipping the cache flush

[Jan Setje-Eilers <Jan.SetjeEilers@xxxxxxxxxx>]

Static memory acquisition follows the same cache-coherency policy as
ordinary heap allocation. Xen therefore cleans and invalidates every page
in each xen,static-mem bank before assigning it to a domain.

Static-memory support is currently enabled only on Arm, but its
implementation and this acquisition path are in common code.

This cache maintenance is independent of boot scrubbing, which clears
memory. With bootscrub=1 or the default bootscrub=idle, create_domUs()
acquires static banks before heap_init_late() starts boot scrubbing. The
pages are in-use by then, while boot scrubbing processes only free pages.
Neither mode therefore clears an assigned static bank.

Walking a large static bank can add several seconds to boot, including time
spent flushing pages Xen does not touch during domain construction.

If a platform permits those untouched pages to be cleared before guest
start, the guest cannot depend on their incoming contents. This is a
platform policy assumption, not a property established by Xen's boot
scrubbing. Xen need not make that initial data coherent with RAM. Xen still
cleans every page it writes while loading the guest kernel, initrd, and
device tree.

Add the default-on staticmem-cache-flush option. Specifying
no-staticmem-cache-flush passes MEMF_no_cache_flush when acquiring a
xen,static-mem bank. The existing behavior remains the default, and static
shared memory is unaffected.

Skipping the flush is safe only if the platform also guarantees that the
untouched pages are absent from all caches and no other agent writes them
before guest start. Such agents include firmware, non-coherent DMA, and
EL3 runtime services.

Assisted-by: Codex:GPT-5
Signed-off-by: Jan Setje-Eilers <Jan.SetjeEilers@xxxxxxxxxx>
---
 docs/misc/xen-command-line.pandoc      | 26 ++++++++++++++++++++++++++
 xen/common/device-tree/static-memory.c |  9 ++++++++-
 xen/common/page_alloc.c                |  6 ++++--
 xen/include/xen/mm.h                   |  2 ++
 4 files changed, 40 insertions(+), 3 deletions(-)

diff --git a/docs/misc/xen-command-line.pandoc 
b/docs/misc/xen-command-line.pandoc
index 1c711fa980..3f5c27858c 100644
--- a/docs/misc/xen-command-line.pandoc
+++ b/docs/misc/xen-command-line.pandoc
@@ -2647,6 +2647,32 @@ On Sappire and Emerald Rapids CPUs with May 2025 
microcode or later, the
 `ibpb-alt=` option can be used to switch to the alternative mitigation for
 Intel SA-00982.  Intel suggest that some workloads will benefit from this.
 
+### staticmem-cache-flush (arm)
+> `= <boolean>`
+
+> Default: `true`
+
+Control the up-front per-page cache flush over dom0less `xen,static-mem`
+banks. Specify `no-staticmem-cache-flush` to skip it. Xen still cleans the
+guest kernel, initrd, and device tree pages it writes to guest RAM.
+
+Static-memory support is currently available only on Arm. Its implementation
+and this option live in common code.
+
+Here, cleaning means cache maintenance, not boot scrubbing. With
+`bootscrub=1` or the default `bootscrub=idle`, Xen assigns dom0less static
+banks before boot scrubbing starts. Assigned pages are no longer free, so
+boot scrubbing does not clear them. This option is therefore independent of
+`bootscrub`.
+
+Only disable this flush when the guest does not depend on the initial contents
+of pages Xen leaves untouched. For example, this may be true when the platform
+permits those pages to be cleared before guest start.
+
+The platform must also ensure that those pages are absent from all caches and
+rule out other writers before guest start. Such writers may include
+non-coherent DMA or EL3 runtime services active during domain construction.
+
 ### sync_console
 > `= <boolean>`
 
diff --git a/xen/common/device-tree/static-memory.c 
b/xen/common/device-tree/static-memory.c
index ffbc12aa24..ed155d162d 100644
--- a/xen/common/device-tree/static-memory.c
+++ b/xen/common/device-tree/static-memory.c
@@ -1,10 +1,15 @@
 /* SPDX-License-Identifier: GPL-2.0-only */
 
+#include <xen/init.h>
+#include <xen/param.h>
 #include <xen/sched.h>
 #include <xen/static-memory.h>
 
 #include <asm/setup.h>
 
+static bool __initdata opt_staticmem_cache_flush = true;
+boolean_param("staticmem-cache-flush", opt_staticmem_cache_flush);
+
 static bool __init append_static_memory_to_bank(struct domain *d,
                                                 struct membank *bank,
                                                 mfn_t smfn,
@@ -54,7 +59,9 @@ static mfn_t __init acquire_static_memory_bank(struct domain 
*d,
     }
 
     smfn = maddr_to_mfn(*pbase);
-    res = acquire_domstatic_pages(d, smfn, PFN_DOWN(*psize), 0);
+    res = acquire_domstatic_pages(d, smfn, PFN_DOWN(*psize),
+                                  opt_staticmem_cache_flush ? 0 :
+                                  MEMF_no_cache_flush);
     if ( res )
     {
         printk(XENLOG_ERR
diff --git a/xen/common/page_alloc.c b/xen/common/page_alloc.c
index 1e47f38721..b2ede17bd2 100644
--- a/xen/common/page_alloc.c
+++ b/xen/common/page_alloc.c
@@ -3094,8 +3094,10 @@ static struct page_info * __init 
acquire_staticmem_pages(mfn_t smfn,
      * Ensure cache and RAM are consistent for platforms where the guest
      * can control its own visibility of/through the cache.
      */
-    for ( i = 0; i < nr_mfns; i++ )
-        flush_page_to_ram(mfn_x(smfn) + i, !(memflags & MEMF_no_icache_flush));
+    if ( !(memflags & MEMF_no_cache_flush) )
+        for ( i = 0; i < nr_mfns; i++ )
+            flush_page_to_ram(mfn_x(smfn) + i,
+                              !(memflags & MEMF_no_icache_flush));
 
     return pg;
 }
diff --git a/xen/include/xen/mm.h b/xen/include/xen/mm.h
index fd8b0ba3f5..511bbc56e0 100644
--- a/xen/include/xen/mm.h
+++ b/xen/include/xen/mm.h
@@ -228,6 +228,8 @@ struct npfec {
 #define  MEMF_no_icache_flush (1U<<_MEMF_no_icache_flush)
 #define _MEMF_no_scrub    8
 #define  MEMF_no_scrub    (1U<<_MEMF_no_scrub)
+#define _MEMF_no_cache_flush 9
+#define  MEMF_no_cache_flush (1U<<_MEMF_no_cache_flush)
 #define _MEMF_node        16
 #define  MEMF_node_mask   ((1U << (8 * sizeof(nodeid_t))) - 1)
 #define  MEMF_node(n)     ((((n) + 1) & MEMF_node_mask) << _MEMF_node)
-- 
2.47.3