Can't create domain with AWS metal instances (latest Xen/Linux revs)

[Jean-Yves Migeon <jym@xxxxxxxxxx>]

Dear list,

While attempting to run Xen on AWS metal instances, I am facing an issue 
that ressembles the one Tamas (K Lengyel) faced months ago [1].
I have tested this on various variations of CPU revisions and kernels 
(Linux 5.4 to 11, Xen 4.15), even latest Linux revisions are susceptible 
to this behavior; as it is a bit cumbersome to trace down xl stack, I 
have not found yet what happens.
Creating new domains seem to fail due to timeouts waiting for block/vbd 
to switch to later "running" states within xenstore. In my setup it 
seems specific to block/vbd (I cannot reproduce a similar issue with vif 
devices). I have not tested with other revisions of Xen, although I 
doubt the hypervisor has anything to deal with here. It properly 
allocated the pages and wait to unpause domain when VMM is ready.
Does that ring a bell to someone? FWIW the patches mentioned in the 
original mail thread are applied (or have been backported).
Verbose output of a (failed) xl create is attached below, with the 
xl.cfg file and xenstore-ls /libxl/5 (the domid of the failed domain). 
Nothing specific logged on dmesg, xl dmesg, or xenstored. The 
xenstore-ls dump looks like typical entry nodes for a vbd device that 
stays in init state.
Any advice/pointers appreciated; keep me CC please.

Thanks

[1] 
https://lore.kernel.org/xen-devel/CABfawh=fNCxQs+QBsYw5StA+vNwqzyQUHyiib3_rujDS1W8S9A@xxxxxxxxxxxxxx/
--
jym@xxxxxxxxxx


====== xl create verbose output ======

# xl -v create debian.cfg
Parsing config from debian.cfg
libxl: detail: libxl_dom.c:182:numa_place_domain: NUMA placement 
candidate with 1 nodes, 36 cpus and 256763 KB free selected
domainbuilder: detail: xc_dom_allocate: cmdline="(null)", features="(null)"
domainbuilder: detail: xc_dom_kernel_file: 
filename="/usr/lib/xen-4.15/boot/hvmloader"
domainbuilder: detail: xc_dom_malloc_filemap    : 174 kB
domainbuilder: detail: xc_dom_boot_xen_init: ver 4.15, caps 
xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
domainbuilder: detail: xc_dom_parse_image: called
domainbuilder: detail: xc_dom_find_loader: trying multiboot-binary 
loader ...
domainbuilder: detail: loader probe failed
domainbuilder: detail: xc_dom_find_loader: trying HVM-generic loader ...
domainbuilder: detail: loader probe OK
xc: detail: ELF: phdr: paddr=0x100000 memsz=0x350a4
xc: detail: ELF: memory: 0x100000 -> 0x1350a4
domainbuilder: detail: xc_dom_mem_init: mem 2040 MB, pages 0x7f800 
pages, 4k each
domainbuilder: detail: xc_dom_mem_init: 0x7f800 pages
domainbuilder: detail: xc_dom_boot_mem_init: called
domainbuilder: detail: xc_dom_malloc            : 4080 kB
xc: detail: PHYSICAL MEMORY ALLOCATION:
xc: detail:   4KB PAGES: 0x0000000000000200
xc: detail:   2MB PAGES: 0x00000000000003fb
xc: detail:   1GB PAGES: 0x0000000000000000
domainbuilder: detail: xc_dom_build_image: called
domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 
0x100+0x36 at 0x7f74a9265000
domainbuilder: detail: xc_dom_alloc_segment:   kernel       : 0x100000 
-> 0x136000  (pfn 0x100 + 0x36 pages)
xc: detail: ELF: phdr 0 at 0x7f74a922f000 -> 0x7f74a925a620
domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 
0x136+0x40 at 0x7f74a9225000
domainbuilder: detail: xc_dom_alloc_segment:   System Firmware module : 
0x136000 -> 0x176000  (pfn 0x136 + 0x40 pages)
domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 
0x176+0x1 at 0x7f74a9318000
domainbuilder: detail: xc_dom_alloc_segment:   HVM start info : 0x176000 
-> 0x177000  (pfn 0x176 + 0x1 pages)
domainbuilder: detail: alloc_pgtables_hvm: doing nothing
domainbuilder: detail: xc_dom_build_image  : virt_alloc_end : 0x177000
domainbuilder: detail: xc_dom_build_image  : virt_pgtab_end : 0x0
domainbuilder: detail: xc_dom_boot_image: called
domainbuilder: detail: xc_dom_compat_check: supported guest type: 
xen-3.0-x86_64
domainbuilder: detail: xc_dom_compat_check: supported guest type: 
xen-3.0-x86_32p
domainbuilder: detail: xc_dom_compat_check: supported guest type: 
hvm-3.0-x86_32 <= matches
domainbuilder: detail: xc_dom_compat_check: supported guest type: 
hvm-3.0-x86_32p
domainbuilder: detail: xc_dom_compat_check: supported guest type: 
hvm-3.0-x86_64
domainbuilder: detail: clear_page: pfn 0xfefff, mfn 0xfefff
domainbuilder: detail: clear_page: pfn 0xfeffc, mfn 0xfeffc
domainbuilder: detail: domain builder memory footprint
domainbuilder: detail:    allocated
domainbuilder: detail:       malloc             : 4085 kB
domainbuilder: detail:       anon mmap          : 0 bytes
domainbuilder: detail:    mapped
domainbuilder: detail:       file mmap          : 174 kB
domainbuilder: detail:       domU mmap          : 476 kB
domainbuilder: detail: vcpu_hvm: called
domainbuilder: detail: xc_dom_gnttab_hvm_seed: called, pfn=0xff000
domainbuilder: detail: xc_dom_gnttab_hvm_seed: called, pfn=0xff001
domainbuilder: detail: xc_dom_release: called
libxl: error: libxl_device.c:1075:device_backend_callback: Domain 
5:unable to add device with path /local/domain/0/backend/vbd/5/51712
libxl: error: libxl_create.c:1249:domcreate_launch_dm: Domain 5:unable 
to add disk devices
libxl: error: libxl_device.c:1075:device_backend_callback: Domain 
5:unable to remove device with path /local/domain/0/backend/vbd/5/51712
libxl: error: libxl_domain.c:1075:devices_destroy_cb: Domain 
5:libxl__devices_destroy failed
libxl: error: libxl_domain.c:1003:libxl__destroy_domid: Domain 
5:Non-existant domain
libxl: error: libxl_domain.c:962:domain_destroy_callback: Domain 
5:Unable to destroy guest
libxl: error: libxl_domain.c:889:domain_destroy_cb: Domain 5:Destruction 
of domain failed
#

====== debian.cfg ======

name = "debian"
builder = "hvm"
vcpus = 1
maxvcpus = 1
memory = 2048
maxmem = 2048
hap = 1
boot = "cd"
serial='pty'
vif = ['bridge=bridge0']
#vfb = ['vnc=1,vncdisplay=0,vncpasswd=foobar,vnclisten=127.0.0.1']
usb = 1
#usbdevice = ['tablet']
nomigrate = 1
#disk = ['file:/home/ubuntu/VMs/debian/debian11.vmdisk,xvda,w']
disk = 
['file:/home/ubuntu/VMs/debian/debian.vmdisk,xvda','file:/home/ubuntu/VMs/debian/debian.iso,xvdc,cdrom']
====== xenstore-ls /libxl/5 ======

# xenstore-ls /libxl/5
device = ""
 vbd = ""
  51712 = ""
   frontend = "/local/domain/5/device/vbd/51712"
   backend = "/local/domain/0/backend/vbd/5/51712"
   params = "/home/ubuntu/VMs/debian/debian.vmdisk"
   script = "/etc/xen/scripts/block"
   frontend-id = "5"
   online = "1"
   removable = "0"
   bootable = "1"
   state = "1"
   dev = "xvda"
   type = "phy"
   mode = "w"
   device-type = "disk"
   discard-enable = "1"
  51744 = ""
   frontend = "/local/domain/5/device/vbd/51744"
   backend = "/local/domain/0/backend/qdisk/5/51744"
   params = "aio:/home/ubuntu/VMs/debian/debian.iso"
   frontend-id = "5"
   online = "1"
   removable = "1"
   bootable = "1"
   state = "1"
   dev = "xvdc"
   type = "qdisk"
   mode = "r"
   device-type = "cdrom"
   discard-enable = "0"
dm-version = "qemu_xen"