Re: RFC: arm64: Handling reserved memory nodes

[Julien Grall <julien@xxxxxxx>]

On 14/09/2023 09:16, Leo Yan wrote:
> Hi all,
Hi Leo,

> I'd like to discuss for how to handle the reserved memory nodes in DT
> binding on Xen / Arm64.  Note, now I am using DTB when boot Xen but
> not UEFI/ACPI (ACPI is disabled in this case).
>
> ## Failure
>
> I ported Xen on a platform, after the kernel booting, the Xen hypervisor
> reports error:
>
>    (XEN) arch/arm/p2m.c:2202: d0v0: Failing to acquire the MFN 0x1a02dc
>
> This error is caused by kernel using an invalid memory frame number
> 0x1a02dc, we can convert it to the address:
>
>    0x1a02dc << PAGE_SHIFT = 0x1_a02d_c000
This error is coming from get_page_from_gva(). The use of the function 
is usually an indication that Xen is trying to access the page. Can you 
use WARN() to provide a full trace?
> ## Reason
>
> Two important things we need to check.  One is what's the DT binding
> passed from the bootloader to Xen hypervisor, and the second thing is
> what's the DT binding passed from Xen hypervisor to kernel.
>
> We can see the bootloader passes below memory nodes to Xen hypervisor:
>
>    (XEN) RAM: 0000000020000000 - 00000000bfffffff
>    (XEN) RAM: 00000001a0000000 - 00000002ffffffff
>    (XEN)
>    (XEN) MODULE[0]: 0000000020100000 - 0000000020265000 Xen
>    (XEN) MODULE[1]: 0000000023000000 - 0000000023024000 Device Tree
>    (XEN) MODULE[2]: 0000000024000000 - 0000000028000000 Kernel
>    (XEN)  RESVD[0]: 0000000020000000 - 000000002000ffff
>    (XEN)  RESVD[1]: 0000000040000000 - 000000005fffffff
>    (XEN)  RESVD[2]: 00000001a0000000 - 00000001bfffffff
>    (XEN)  RESVD[3]: 000000002e000000 - 000000002fffffff
>
> We can see the second DDR section is:
>
>    [0x0000_0001_a000_0000 .. 0x0000_0002_ffff_ffff]
>
> And there have reserved memory section is:
>
>    [0x0000_0001_a000_0000 .. 0x0000_0001_bfff_ffff]
>
> When register the boot memory sections, dt_unreserved_regions() will
> remove all reserved memory sections, which means the section
> [0x0000_0001_a000_0000 .. 0x0000_0001_bfff_ffff] is not managed by Xen
> hypervisor at all.  If later kernel uses any pages in this range, Xen
> will report the error.
It is perfectly fine for dom0 to access a page that are not managed by 
Xen (a good example is MMIO regions are not managed). Now, some problems 
can occur if the page then needs to be access by Xen (e.g. if they 
contain hypercall buffers).
[...]

> ## Fixes
>
> I think it's wrong to add the reserved memory regions into the DT
> binding as normal memory nodes for Dom0 kernel.  On the other hand, we
> cannot simply remove these reserved memory regions and don't pass to
> Dom0 kernel - we might reserve memory for specific purpose, for example,
> ramoops [1] for kernel debugging.
>
> The right thing to do is to keep these reserved memory nodes to Dom0
> kernel.  So one task is to record properties for these reserved memory
> node name and properties and pass to Dom0 kernel.
>
> The difficulty is how we can avoid allocate these reserved memory
> regions in Xen hypervisor.  We cannot register the reserved memory
> into the boot pages, otherwise, the reserved memory might be allocated
> in the early phase.  But we need to register these pages into the
> frame management framework and reserve them in the first place, so
> that we can allow Dom0 kernel to use them.  (I checked a bit the static
> memory mechanism, seems to me we cannot use it to resolve this issue).
From my understanding reserved region are normal RAM which have been 
carved out for specific purpose. They may expect different caching 
policy (e.g. non-cachable). AFAIK, Xen doesn't have the capability to 
know the memory attribute (the DT binding only tell whether the region 
should not mapped. See the property "no-map"), hence why they were 
excluded from the memory management.
It would be good to understand why Xen may try to get a reference on the 
page.
Also, can you find the associated reserved-region and content of the 
Device-Tree?
Cheers,

--
Julien Grall