Re: [PATCH] xen/dom0less: Increase guest DTB size for high-vCPU guests

[Oleksii Kurochko <oleksii.kurochko@xxxxxxxxx>]

Hello Oleksandr,

On 12/3/25 12:03 PM, Oleksandr Tyshchenko wrote:
> On 02.12.25 23:33, Grygorii Strashko wrote:
> > On 02.12.25 21:32, Oleksandr Tyshchenko wrote:
> > > Creating a guest with a high vCPU count (e.g., >32) fails because
> > > the guest's device tree buffer (DOMU_DTB_SIZE) overflows during creation.
> > > The FDT nodes for each vCPU quickly exhaust the 4KiB buffer,
> > > causing a guest creation failure.
> > >
> > > Increase the buffer size to 16KiB to support guests up to
> > > the MAX_VIRT_CPUS limit (128).
> > >
> > > Signed-off-by: Oleksandr Tyshchenko <oleksandr_tyshchenko@xxxxxxxx>
> > > ---
> > > Noticed when testing the boundary conditions for dom0less guest
> > > creation on Arm64.
> > >
> > > Domain configuration:
> > > fdt mknod /chosen domU0
> > > fdt set /chosen/domU0 compatible "xen,domain"
> > > fdt set /chosen/domU0 \#address-cells <0x2>
> > > fdt set /chosen/domU0 \#size-cells <0x2>
> > > fdt set /chosen/domU0 memory <0x0 0x10000 >
> > > fdt set /chosen/domU0 cpus <33>
> > > fdt set /chosen/domU0 vpl011
> > > fdt mknod /chosen/domU0 module@40400000
> > > fdt set /chosen/domU0/module@40400000 compatible  "multiboot,kernel"
> > > "multiboot,module"
> > > fdt set /chosen/domU0/module@40400000 reg <0x0 0x40400000 0x0 0x16000 >
> > > fdt set /chosen/domU0/module@40400000 bootargs "console=ttyAMA0"
> > >
> > > Failure log:
> > > (XEN) Xen dom0less mode detected
> > > (XEN) *** LOADING DOMU cpus=33 memory=0x10000KB ***
> > > (XEN) Loading d1 kernel from boot module @ 0000000040400000
> > > (XEN) Allocating mappings totalling 64MB for d1:
> > > (XEN) d1 BANK[0] 0x00000040000000-0x00000044000000 (64MB)
> > > (XEN) Device tree generation failed (-22).
> > > (XEN)
> > > (XEN) ****************************************
> > > (XEN) Panic on CPU 0:
> > > (XEN) Could not set up domain domU0 (rc = -22)
> > > (XEN) ****************************************
> > > ---
> > > ---
> > >    xen/common/device-tree/dom0less-build.c | 8 +++++---
> > >    1 file changed, 5 insertions(+), 3 deletions(-)
> > >
> > > diff --git a/xen/common/device-tree/dom0less-build.c b/xen/common/
> > > device-tree/dom0less-build.c
> > > index 3f5b987ed8..d7d0a47b97 100644
> > > --- a/xen/common/device-tree/dom0less-build.c
> > > +++ b/xen/common/device-tree/dom0less-build.c
> > > @@ -461,10 +461,12 @@ static int __init
> > > domain_handle_dtb_boot_module(struct domain *d,
> > >    /*
> > >     * The max size for DT is 2MB. However, the generated DT is small
> > > (not including
> > > - * domU passthrough DT nodes whose size we account separately), 4KB
> > > are enough
> > > - * for now, but we might have to increase it in the future.
> > > + * domU passthrough DT nodes whose size we account separately). The
> > > size is
> > > + * primarily driven by the number of vCPU nodes. The previous 4KiB
> > > buffer was
> > > + * insufficient for guests with high vCPU counts, so it has been
> > > increased
> > > + * to support up to the MAX_VIRT_CPUS limit (128).
> > >     */
> > > -#define DOMU_DTB_SIZE 4096
> > > +#define DOMU_DTB_SIZE (4096 * 4)
> > May be It wants Kconfig?
> > Or some formula which accounts MAX_VIRT_CPUS?
> I agree that using a formula that accounts for MAX_VIRT_CPUS is the most
> robust approach.
One option could be to detect the size at runtime, essentially, try to allocate
it, and if an error occurs, increase the fdtsize and try again. I don’t really
like this approach, but I wanted to mention it in case someone finds it useful.
The benefit of this approach is that if, in the future, something else such
as a CPU node contributes to the final FDT size, we won’t need to update the
formula again.

> Here is the empirical data (by testing with the maximum number of device
> tree nodes (e.g., hypervisor and reserved-memory nodes) and enabling all
> optional CPU properties (e.g., clock-frequency)):
>
> cpus=1
> (XEN) Final compacted FDT size is: 1586 bytes
>
> cpus=2
> (XEN) Final compacted FDT size is: 1698 bytes
>
> cpus=32
> (XEN) Final compacted FDT size is: 5058 bytes
>
> cpus=128
> (XEN) Final compacted FDT size is: 15810 bytes
>
>
> static int __init prepare_dtb_domU(struct domain *d, struct kernel_info
> *kinfo)
>    {
>        int addrcells, sizecells;
> @@ -569,6 +569,8 @@ static int __init prepare_dtb_domU(struct domain *d,
> struct kernel_info *kinfo)
>        if ( ret < 0 )
>            goto err;
>
> +    printk("Final compacted FDT size is: %d bytes\n",
> fdt_totalsize(kinfo->fdt));
> +
>        return 0;
>
>      err:
>
> This data shows (assuming my testing/calculations are correct):
>
> - A marginal cost of 112 bytes per vCPU in the final, compacted device tree.
> - A fixed base size of 1474 bytes for all non-vCPU content.
>
> Based on that I would propose the following formula with the justification:
>
> /*
>    * The size is calculated from a fixed baseline plus a scalable
>    * portion for each potential vCPU node up to the system limit
>    * (MAX_VIRT_CPUS), as the vCPU nodes are the primary consumer
>    * of space.
>    *
>    * The baseline of 2KiB is a safe buffer for all non-vCPU FDT
>    * content. The 128 bytes per vCPU is derived from a worst-case
>    * analysis of the FDT construction-time size for a single
>    * vCPU node.
>    */
> #define DOMU_DTB_SIZE (2048 + (MAX_VIRT_CPUS * 128))
>
> **********************************************
>
> Please tell me would you be happy with that?
I would also like to note that we probably want to add a BUILD_BUG_ON() check
to ensure that DOMU_DTB_SIZE is not larger than SZ_2M. Otherwise, we would get
a runtime error instead of a build-time failure, since there is code that limits
fdtsize to SZ_2M:

    /* Cap to max DT size if needed */
    fdt_size = min(fdt_size, SZ_2M);

Thanks.

~ Oleksii