[BUG] xen-swiotlb: 32-bit coherent DMA mask rejected on PV dom0 since 6.6 (CONFIG_SWIOTLB_DYNAMIC), aacraid probe fails

[Anton Markov <akmarkov45@xxxxxxxxx>]

Hello,

I'm hitting a regression on a PV dom0 where a driver that requests a 32-bit coherent DMA mask fails to probe. Bare metal with the same kernel is fine.

Environment:

Xen: 4.21.2-pre

dom0: Ubuntu 24.04, linux-image-6.8.0-139-generic (Ubuntu stock, CONFIG_SWIOTLB_DYNAMIC=y)

dom0 mode: PV

Hardware: Supermicro X10DRW-i (BIOS 2.0, 12/17/2015), 2x Xeon E5-2670 v3, 128 GB RAM

Controller: 81:00.0 RAID bus controller [0104]: Adaptec Series 8 12G SAS/PCIe 3 [9005:028d] (rev 01)

Xen command line (no dom0_mem, so dom0 gets all host memory):

placeholder ucode=scan dom0_max_vcpus=4-48 dom0_vcpus_pin=0 force-ept=1 ept=no-ad,no-pml hap_1gb=0 hap_2mb=0 altp2m=1 hpet=legacy-replacement smt=1 spec-ctrl=no gnttab_max_frames=512 cpufreq=xen:performance max_cstate=1 sched=credit sched-gran=cpu apicv=0 sched_credit2_max_cpus_runqueue=48 credit2_runqueue=cpu credit2_load_window_shift=33 credit2_load_precision_shift=18 sched_credit_tslice_ms=5 sched_ratelimit_us=500 hvm-freeze-on-pause=adaptive no-real-mode edd=off

Symptom (dmesg):

aacraid 0000:81:00.0: PCI 32 B consistent dma mask set failed

aacraid: probe of 0000:81:00.0 failed with error -5

The controller is present in lspci, the driver loads, but dma_set_coherent_mask(DMA_BIT_MASK(32)) is refused, so the disks never appear. Booting the same kernel without Xen works.

What probably goes wrong:

Since v6.6, xen_swiotlb_dma_supported() checks

xen_phys_to_dma(hwdev, default_swiotlb_limit()) <= mask

With CONFIG_SWIOTLB_DYNAMIC=y, default_swiotlb_limit() returns io_tlb_default_mem.phys_limit, which swiotlb_init_remap() sets to virt_to_phys(high_memory - 1) because Xen passes SWIOTLB_ANY (ad96ce3252db, used by swiotlb-xen since 05ee774122bd). On a PV dom0 that pseudo-physical address translates to a machine frame that, on any host with RAM above 4 GB, is above the 32-bit mask, so every 32-bit coherent mask request is rejected.

Before v6.6 the check used the end of the actual default pool, which xen_swiotlb_fixup() places below 4 GB, so 32-bit masks always passed. The coherent allocation itself would still succeed via xen_swiotlb_alloc_coherent() / xen_create_contiguous_region(); only the capability check is wrong.

This looks like the same root cause as the unanswered report from July 2024 (megaraid_sas, 63-bit mask, "Failed to set DMA mask"):

https://lkml.iu.edu/hypermail/linux/kernel/2407.3/08254.html

That report already confirmed CONFIG_SWIOTLB_DYNAMIC=n makes the problem go away. The 32-bit-mask case here is deterministic on any >4 GB host, so it should be easy to reproduce.

A minimal change that restores the previous behaviour, e.g.

return xen_phys_to_dma(hwdev, io_tlb_default_mem.defpool.end - 1) <= mask;

probably isn't the right fix for dynamic pools in general, but I wanted to flag where the check went wrong. Happy to test patches on this hardware.

Thanks.