Re: [RFC PATCH 0/1] static-memory: allow skipping the cache flush

[Jan Setje-Eilers <Jan.SetjeEilers@xxxxxxxxxx>]

On 9/17/26 04:06, Grall, Julien wrote:

Hi!

> Sorry for the late answer. The topic is a bit tricky.

Thanks for replying regardless. FWIW I tried to get some feedback
internally as well, and it seemed like no one wanted to get involved. :)


> On 29/08/2026 07:06, Jan Setje-Eilers wrote:
>> Hi all,
>> 
>> I would like feedback on whether Xen needs to flush every page in a
>> dom0less xen,static-mem bank before assigning it to a guest.
>> 
>> Static-memory support is currently enabled only on Arm, where I am using
>> and testing this change. Its implementation and acquisition path are in
>> common code, so this RFC is about the generic static-memory path.
>> 
>> The current code cleans and invalidates every page in the bank. This is a
>> safe default, but walking a large bank can add several seconds to boot even
>> though Xen only writes the pages used for the guest kernel, initrd, and
>> device tree.
>> 
>> Here, cleaning refers to cache maintenance, not boot scrubbing. With
>> bootscrub=1 or the default bootscrub=idle, Xen assigns dom0less static
>> banks before boot scrubbing starts. The pages are no longer free, so boot
>> scrubbing does not clear them.
>> 
>> My starting point is that if a platform permits the untouched pages to be
>> cleared before guest start, the guest cannot depend on their initial
>> contents. This is a platform policy assumption, not something Xen's boot
>> scrubbing establishes. Xen already cleans each page it writes during
>> domain construction.
>> 
>> Would it therefore be reasonable to let a platform skip the full-bank
>> flush, provided it can also guarantee that the untouched pages are absent
>> from all caches and that firmware, DMA, EL3 services, and other CPUs do not
>> write them before guest start?
> 
> IIUC, the cache flush was mainly added to prevent memory leaking between 
> two guests if they are the MMU off (see XSA-364). I think what you wrote 
> makes sense, but I would prefer Arm to confirm this is fine. For Xen ...

 Yes, I think in a world where guests are coming and going, and smaller,
this flush seems relatively sensible.

>> I am using no-staticmem-cache-flush with bootscrub=0 on a 64-bit Arm
>> system with two dom0less guests. Both guests boot normally. Skipping the
>> flush reduced Xen startup from about five seconds to about one second.
> 
> ... I think the slow down you describe could also happen when a dynamic 
> VM is created (e.g. via xl).
> 
> A few years ago, Arm introduced FEAT_FWB to rework how stage-2 + stage-1 
> memory attributes works. Currently in Xen, we are using the "old" 
> approach where the used attributes is the strongest of the two. For 
> instance, the guest has the MMU off, then the access will bypass the 
> cache. With FEAT_FWB, you can force the memory attributes to be 
> cacheable even when the MMU is off.
> 
> I believe FEAT_FWB should solve the problem you have (assume your HW 
> supports it). In term of effort, it is mostly updating the stage-2 
> memory attributes and then getting rid of some cache operations (include 
> set/way emulation) when the feature is enabled.
 I found that as well, but at least some of my hardware is from just
before that shows up.


> BTW, how much memory do you give to the domain? 4 seconds difference 
> seems quite a lot.

 This is two domains: 15 1/4 G to one and 4 1/4 G to the other.

 It really stood out when we did a boot time reduction exercise.
-jan