Re: [RFC PATCH 0/1] static-memory: allow skipping the cache flush

["Grall, Julien" <julien@xxxxxxx>]

Hi Jan,

Sorry for the late answer. The topic is a bit tricky.

On 29/08/2026 07:06, Jan Setje-Eilers wrote:
> Hi all,
>
> I would like feedback on whether Xen needs to flush every page in a
> dom0less xen,static-mem bank before assigning it to a guest.
>
> Static-memory support is currently enabled only on Arm, where I am using
> and testing this change. Its implementation and acquisition path are in
> common code, so this RFC is about the generic static-memory path.
>
> The current code cleans and invalidates every page in the bank. This is a
> safe default, but walking a large bank can add several seconds to boot even
> though Xen only writes the pages used for the guest kernel, initrd, and
> device tree.
>
> Here, cleaning refers to cache maintenance, not boot scrubbing. With
> bootscrub=1 or the default bootscrub=idle, Xen assigns dom0less static
> banks before boot scrubbing starts. The pages are no longer free, so boot
> scrubbing does not clear them.
>
> My starting point is that if a platform permits the untouched pages to be
> cleared before guest start, the guest cannot depend on their initial
> contents. This is a platform policy assumption, not something Xen's boot
> scrubbing establishes. Xen already cleans each page it writes during
> domain construction.
>
> Would it therefore be reasonable to let a platform skip the full-bank
> flush, provided it can also guarantee that the untouched pages are absent
> from all caches and that firmware, DMA, EL3 services, and other CPUs do not
> write them before guest start?
IIUC, the cache flush was mainly added to prevent memory leaking between 
two guests if they are the MMU off (see XSA-364). I think what you wrote 
makes sense, but I would prefer Arm to confirm this is fine. For Xen ...
> I am using no-staticmem-cache-flush with bootscrub=0 on a 64-bit Arm
> system with two dom0less guests. Both guests boot normally. Skipping the
> flush reduced Xen startup from about five seconds to about one second.
... I think the slow down you describe could also happen when a dynamic 
VM is created (e.g. via xl).
A few years ago, Arm introduced FEAT_FWB to rework how stage-2 + stage-1 
memory attributes works. Currently in Xen, we are using the "old" 
approach where the used attributes is the strongest of the two. For 
instance, the guest has the MMU off, then the access will bypass the 
cache. With FEAT_FWB, you can force the memory attributes to be 
cacheable even when the MMU is off.
I believe FEAT_FWB should solve the problem you have (assume your HW 
supports it). In term of effort, it is mostly updating the stage-2 
memory attributes and then getting rid of some cache operations (include 
set/way emulation) when the feature is enabled.
BTW, how much memory do you give to the domain? 4 seconds difference 
seems quite a lot.
Cheers,

--
Julien Grall