Re: [PATCH] xen/dom0less: Increase guest DTB size for high-vCPU guests

[Oleksandr Tyshchenko <Oleksandr_Tyshchenko@xxxxxxxx>]

On 02.12.25 23:33, Grygorii Strashko wrote:


Hello Grygorii

> 
> 
> On 02.12.25 21:32, Oleksandr Tyshchenko wrote:
>> Creating a guest with a high vCPU count (e.g., >32) fails because
>> the guest's device tree buffer (DOMU_DTB_SIZE) overflows during creation.
>> The FDT nodes for each vCPU quickly exhaust the 4KiB buffer,
>> causing a guest creation failure.
>>
>> Increase the buffer size to 16KiB to support guests up to
>> the MAX_VIRT_CPUS limit (128).
>>
>> Signed-off-by: Oleksandr Tyshchenko <oleksandr_tyshchenko@xxxxxxxx>
>> ---
>> Noticed when testing the boundary conditions for dom0less guest
>> creation on Arm64.
>>
>> Domain configuration:
>> fdt mknod /chosen domU0
>> fdt set /chosen/domU0 compatible "xen,domain"
>> fdt set /chosen/domU0 \#address-cells <0x2>
>> fdt set /chosen/domU0 \#size-cells <0x2>
>> fdt set /chosen/domU0 memory <0x0 0x10000 >
>> fdt set /chosen/domU0 cpus <33>
>> fdt set /chosen/domU0 vpl011
>> fdt mknod /chosen/domU0 module@40400000
>> fdt set /chosen/domU0/module@40400000 compatible  "multiboot,kernel" 
>> "multiboot,module"
>> fdt set /chosen/domU0/module@40400000 reg <0x0 0x40400000 0x0 0x16000 >
>> fdt set /chosen/domU0/module@40400000 bootargs "console=ttyAMA0"
>>
>> Failure log:
>> (XEN) Xen dom0less mode detected
>> (XEN) *** LOADING DOMU cpus=33 memory=0x10000KB ***
>> (XEN) Loading d1 kernel from boot module @ 0000000040400000
>> (XEN) Allocating mappings totalling 64MB for d1:
>> (XEN) d1 BANK[0] 0x00000040000000-0x00000044000000 (64MB)
>> (XEN) Device tree generation failed (-22).
>> (XEN)
>> (XEN) ****************************************
>> (XEN) Panic on CPU 0:
>> (XEN) Could not set up domain domU0 (rc = -22)
>> (XEN) ****************************************
>> ---
>> ---
>>   xen/common/device-tree/dom0less-build.c | 8 +++++---
>>   1 file changed, 5 insertions(+), 3 deletions(-)
>>
>> diff --git a/xen/common/device-tree/dom0less-build.c b/xen/common/ 
>> device-tree/dom0less-build.c
>> index 3f5b987ed8..d7d0a47b97 100644
>> --- a/xen/common/device-tree/dom0less-build.c
>> +++ b/xen/common/device-tree/dom0less-build.c
>> @@ -461,10 +461,12 @@ static int __init 
>> domain_handle_dtb_boot_module(struct domain *d,
>>   /*
>>    * The max size for DT is 2MB. However, the generated DT is small 
>> (not including
>> - * domU passthrough DT nodes whose size we account separately), 4KB 
>> are enough
>> - * for now, but we might have to increase it in the future.
>> + * domU passthrough DT nodes whose size we account separately). The 
>> size is
>> + * primarily driven by the number of vCPU nodes. The previous 4KiB 
>> buffer was
>> + * insufficient for guests with high vCPU counts, so it has been 
>> increased
>> + * to support up to the MAX_VIRT_CPUS limit (128).
>>    */
>> -#define DOMU_DTB_SIZE 4096
>> +#define DOMU_DTB_SIZE (4096 * 4)
> 
> May be It wants Kconfig?
> Or some formula which accounts MAX_VIRT_CPUS?


I agree that using a formula that accounts for MAX_VIRT_CPUS is the most 
robust approach.

Here is the empirical data (by testing with the maximum number of device 
tree nodes (e.g., hypervisor and reserved-memory nodes) and enabling all 
optional CPU properties (e.g., clock-frequency)):

cpus=1
(XEN) Final compacted FDT size is: 1586 bytes

cpus=2
(XEN) Final compacted FDT size is: 1698 bytes

cpus=32
(XEN) Final compacted FDT size is: 5058 bytes

cpus=128
(XEN) Final compacted FDT size is: 15810 bytes


static int __init prepare_dtb_domU(struct domain *d, struct kernel_info 
*kinfo)
  {
      int addrcells, sizecells;
@@ -569,6 +569,8 @@ static int __init prepare_dtb_domU(struct domain *d, 
struct kernel_info *kinfo)
      if ( ret < 0 )
          goto err;

+    printk("Final compacted FDT size is: %d bytes\n", 
fdt_totalsize(kinfo->fdt));
+
      return 0;

    err:

This data shows (assuming my testing/calculations are correct):

- A marginal cost of 112 bytes per vCPU in the final, compacted device tree.
- A fixed base size of 1474 bytes for all non-vCPU content.

Based on that I would propose the following formula with the justification:

/*
  * The size is calculated from a fixed baseline plus a scalable
  * portion for each potential vCPU node up to the system limit
  * (MAX_VIRT_CPUS), as the vCPU nodes are the primary consumer
  * of space.
  *
  * The baseline of 2KiB is a safe buffer for all non-vCPU FDT
  * content. The 128 bytes per vCPU is derived from a worst-case
  * analysis of the FDT construction-time size for a single
  * vCPU node.
  */
#define DOMU_DTB_SIZE (2048 + (MAX_VIRT_CPUS * 128))

**********************************************

Please tell me would you be happy with that?


> 
>>   static int __init prepare_dtb_domU(struct domain *d, struct 
>> kernel_info *kinfo)
>>   {
>>       int addrcells, sizecells;
>