[BUG] Xen Dom0 fails to boot with "nvme.poll_queues=24" and "dom0_max_vcpus=6 dom0_vcpus_pin"

[Egon Ojamaa <egon.ojamaa@xxxxxxxxxxxxxx>]

Hello, Real human writing, not AI here ;)

#### Problem:

Booting Xen hypervisor with Linux Dom0.

GRUB loads, Xen kernel messages load up to "Using 6 CPU"

# From this point:

Next screen goes black and stays blank for 10+minutes.

Have to force reboot and choose non XEN kernel booting option via GRUB.

# From this point expected behaviour without bug:

Next screen goes black and stays so for 30-70sec and Dom0 kernel messages appear,

Next login is possible and system is running, "xl info" and other xl commands function.

#### Cause:

I spent 2 days rebooting to narrow this down..

This happens if we set both settings same time:

NVME io polling and XEN kernel CPU limits, via default grub configs:

GRUB_CMDLINE_LINUX_DEFAULT="nvme.poll_queues=24"

# Can also be set via /mnt/oldroot/etc/modprobe.d/nvme.conf

# For proper polling setup nvme.io_poll=1 should also be set,

# but symptom already appears if we set only nvme.poll_queues=24

and

GRUB_CMDLINE_XEN_DEFAULT="dom0_max_vcpus=6 dom0_vcpus_pin"

#### Affected software:

24.04.4 LTS (Noble Numbat)

Kernel:   6.8.0-136-generic x86_64  

And HWE:  7.0.0-38-generic x86_64

xen-hypervisor-4.17-amd64 4.17.3+10-g091466ba55-1.1ubuntu3

## updating was attempted, but same symptoms still:

26.04.1 LTS (Resolute Raccoon)

Kernel:   7.0.0-38-generic x86_64

xen-hypervisor-4.20-amd64 4.20.2+7-g1badcf5035-2build2

#### Hardware:

Supermicro MB,

AMD EPYC Zen5 CPU with 48threads,

(horrendeus 2+min boot time to GRUB, server MB with BMC/IPMI)

BIOS (not sure how relevant):

 ACPI SRAT L3 Cache As NUMA Domain - Enabled

 IOMMU - enabled

 DMA Protection - enabled

 DMAr Support - disabled (if enabled we get +50sec to boot time)

 DRTM Virtual Device Support - enabeld

 DRTM Memory Reservation - enabeld

 SEV-SNP Support - enabled

 ASPM Support - enabled for 24.04 testing later set it to disabled for 26.04 testing.

Gen5 NVME disks connected via bifurcation cards:

5pieces: Crucial T700 / PN: CT2000T700SSD5

4pieces: FireCuda 540 / PN: 3JT301-300

1piece: Lexar NM1090 / PN: LNM1090002T-RNANG

Sadly not my personal system, so I can not do further heavy testing on it.

io polling does give very good performance boost to Gen5 NVME, and even Gen4.

Sad we cant use it, as CPU pinning is more important for XEN hypervizor system.

This has been also reported to launchpad.net:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2170270

Sincerely,
Egon O.