Test report: IGD (RPL) SR-IOV on Xen - Linux 7.2.5, xe

[Marek Marczykowski-Górecki <marmarek@xxxxxxxxxxxxxxxxxxxxxx>]

Hi,

I was talking with Teddy about SR-IOV of Intel GPU and decided to repeat
the test.

My setup is:
Hardware:
Intel Raptor Lake system with:
00:02.0 Display controller [0380]: Intel Corporation Raptor Lake-S GT1 [UHD 
Graphics 770] [8086:a780] (rev 04)

Software:
Qubes 4.3 with Xen 4.19.5
(PV) Dom0 Linux 6.18.51, with "xe.force_probe=a780 i915.force_probe=!a780" on
cmdline
DomU Linux 7.2.5, also with "xe.force_probe=a780 i915.force_probe=!a780"
on cmdline

Hardware-wise, this platform should support SR-IOV:
https://www.intel.com/content/www/us/en/support/articles/000093216/graphics/processor-graphics.html

But software-wise, the "xe" driver doesn't officially support this
platform - this one is officially still on i915 - which in upstream
version doesn't support SR-IOV at all. There is a i915 branch from
Intel with SR-IOV support added, but it isn't maintained anymore, and I
had worse results with it (although some people report it worked for
them).

I tried also LNL, but the one I have does not advertise SR-IOV at all
(not sure if hardware lacking, or disabled at firmware level).
I have also PTL that should theoretically be better, but userspace side
on the current Qubes OS version is too old for that, so it's more
complicated for me to test.

Anyway, back to RPL:

Dom0 steps:
echo 7 > /sys/bus/pci/devices/0000:00:02.0/sriov_numvfs

This makes the VFs to appear, now they can be assigned to a domU (I used
00:02.1).

Dom0 messages on boot:
[    6.063363] xe 0000:00:02.0: [drm] Running in SR-IOV PF mode
[    6.063373] xe 0000:00:02.0: [drm] Found alderlake_s/raptorlake_s (device ID 
a780) integrated display version 12.00 stepping D0
[    6.119874] xe 0000:00:02.0: [drm] Finished loading DMC firmware 
i915/adls_dmc_ver2_01.bin (v2.1)
[    6.129583] xe 0000:00:02.0: [drm] Tile0: GT0: Using GuC firmware from 
i915/tgl_guc_70.bin version 70.49.4
[    6.134506] xe 0000:00:02.0: [drm] Tile0: GT0: Using HuC firmware from 
i915/tgl_huc.bin version 7.9.3
[    6.224163] xe 0000:00:02.0: [drm] Tile0: GT0: vcs1 fused off
[    6.224237] xe 0000:00:02.0: [drm] Tile0: GT0: vcs3 fused off
[    6.224240] xe 0000:00:02.0: [drm] Tile0: GT0: vcs4 fused off
[    6.224244] xe 0000:00:02.0: [drm] Tile0: GT0: vcs5 fused off
[    6.224248] xe 0000:00:02.0: [drm] Tile0: GT0: vcs6 fused off
[    6.224251] xe 0000:00:02.0: [drm] Tile0: GT0: vcs7 fused off
[    6.224255] xe 0000:00:02.0: [drm] Tile0: GT0: vecs1 fused off
[    6.224259] xe 0000:00:02.0: [drm] Tile0: GT0: vecs2 fused off
[    6.224262] xe 0000:00:02.0: [drm] Tile0: GT0: vecs3 fused off
[    6.242973] xe 0000:00:02.0: [drm] Registered 4 planes with drm panic
[    6.242979] [drm] Initialized xe 1.1.0 for 0000:00:02.0 on minor 0
[    6.323673] xe 0000:00:02.0: [drm] Allocated fbdev into stolen failed: -12
[    6.326272] Console: switching to colour frame buffer device 128x48
[    6.343560] xe 0000:00:02.0: [drm] fb1: xedrmfb frame buffer device

And then on adding VFs:
[  274.829735] xe 0000:00:02.0: [drm] Tile0: GT0: PF: VF1..VF7 provisioned with 
598138880 (570 MiB) GGTT
[  274.830257] xe 0000:00:02.0: [drm] Tile0: GT0: PF: VF1..VF7 provisioned with 
9324 GuC context IDs
[  274.831069] xe 0000:00:02.0: [drm] Tile0: GT0: PF: VF1..VF7 provisioned with 
36 GuC doorbell IDs
[  274.935797] pci 0000:00:02.1: [8086:a780] type 00 class 0x038000 PCIe Root 
Complex Integrated Endpoint
[  274.935947] pci 0000:00:02.1: DMAR: Skip IOMMU disabling for graphics
[  274.936569] i915 0000:00:02.1: I915 probe blocked for Device ID a780.
[  274.936607] xe 0000:00:02.1: enabling device (0000 -> 0002)
[  274.940247] xe 0000:00:02.1: [drm] Running in SR-IOV VF mode
[  274.940272] xe 0000:00:02.1: [drm] VF: migration disabled: experimental 
feature not available on production builds
[  274.958658] xe 0000:00:02.1: [drm] Tile0: GT0: vcs1 fused off
[  274.958664] xe 0000:00:02.1: [drm] Tile0: GT0: vcs3 fused off
[  274.958668] xe 0000:00:02.1: [drm] Tile0: GT0: vcs4 fused off
[  274.958671] xe 0000:00:02.1: [drm] Tile0: GT0: vcs5 fused off
[  274.958674] xe 0000:00:02.1: [drm] Tile0: GT0: vcs6 fused off
[  274.958677] xe 0000:00:02.1: [drm] Tile0: GT0: vcs7 fused off
[  274.958680] xe 0000:00:02.1: [drm] Tile0: GT0: vecs1 fused off
[  274.958683] xe 0000:00:02.1: [drm] Tile0: GT0: vecs2 fused off
[  274.958686] xe 0000:00:02.1: [drm] Tile0: GT0: vecs3 fused off
[  274.964105] [drm] Initialized xe 1.1.0 for 0000:00:02.1 on minor 1
[  274.964276] pci 0000:00:02.2: [8086:a780] type 00 class 0x038000 PCIe Root 
Complex Integrated Endpoint
[  274.964297] pci 0000:00:02.2: DMAR: Skip IOMMU disabling for graphics
[  274.964426] i915 0000:00:02.2: I915 probe blocked for Device ID a780.
[  274.964440] xe 0000:00:02.2: enabling device (0000 -> 0002)
[  274.965163] xe 0000:00:02.2: [drm] Running in SR-IOV VF mode
[  274.965167] xe 0000:00:02.2: [drm] VF: migration disabled: experimental 
feature not available on production builds
[  274.981117] xe 0000:00:02.2: [drm] Tile0: GT0: vcs1 fused off
[  274.981124] xe 0000:00:02.2: [drm] Tile0: GT0: vcs3 fused off
[  274.981127] xe 0000:00:02.2: [drm] Tile0: GT0: vcs4 fused off
[  274.981130] xe 0000:00:02.2: [drm] Tile0: GT0: vcs5 fused off
[  274.981134] xe 0000:00:02.2: [drm] Tile0: GT0: vcs6 fused off
[  274.981137] xe 0000:00:02.2: [drm] Tile0: GT0: vcs7 fused off
[  274.981140] xe 0000:00:02.2: [drm] Tile0: GT0: vecs1 fused off
[  274.981143] xe 0000:00:02.2: [drm] Tile0: GT0: vecs2 fused off
[  274.981146] xe 0000:00:02.2: [drm] Tile0: GT0: vecs3 fused off
[  274.986586] [drm] Initialized xe 1.1.0 for 0000:00:02.2 on minor 2
[  274.986666] pci 0000:00:02.3: [8086:a780] type 00 class 0x038000 PCIe Root 
Complex Integrated Endpoint
[  274.986690] pci 0000:00:02.3: DMAR: Skip IOMMU disabling for graphics
[  274.986873] i915 0000:00:02.3: I915 probe blocked for Device ID a780.
[  274.986886] xe 0000:00:02.3: enabling device (0000 -> 0002)
[  274.987857] xe 0000:00:02.3: [drm] Running in SR-IOV VF mode
[  274.987949] xe 0000:00:02.3: [drm] VF: migration disabled: experimental 
feature not available on production builds
[  275.000724] xe 0000:00:02.3: [drm] Tile0: GT0: vcs1 fused off
[  275.000730] xe 0000:00:02.3: [drm] Tile0: GT0: vcs3 fused off
[  275.000733] xe 0000:00:02.3: [drm] Tile0: GT0: vcs4 fused off
[  275.000736] xe 0000:00:02.3: [drm] Tile0: GT0: vcs5 fused off
[  275.000739] xe 0000:00:02.3: [drm] Tile0: GT0: vcs6 fused off
[  275.000742] xe 0000:00:02.3: [drm] Tile0: GT0: vcs7 fused off
[  275.000745] xe 0000:00:02.3: [drm] Tile0: GT0: vecs1 fused off
[  275.000748] xe 0000:00:02.3: [drm] Tile0: GT0: vecs2 fused off
[  275.000751] xe 0000:00:02.3: [drm] Tile0: GT0: vecs3 fused off
[  275.005750] [drm] Initialized xe 1.1.0 for 0000:00:02.3 on minor 3
[  275.005819] pci 0000:00:02.4: [8086:a780] type 00 class 0x038000 PCIe Root 
Complex Integrated Endpoint
[  275.005844] pci 0000:00:02.4: DMAR: Skip IOMMU disabling for graphics
[  275.006222] i915 0000:00:02.4: I915 probe blocked for Device ID a780.
[  275.006236] xe 0000:00:02.4: enabling device (0000 -> 0002)
[  275.007191] xe 0000:00:02.4: [drm] Running in SR-IOV VF mode
[  275.007197] xe 0000:00:02.4: [drm] VF: migration disabled: experimental 
feature not available on production builds
[  275.023297] xe 0000:00:02.4: [drm] Tile0: GT0: vcs1 fused off
[  275.023303] xe 0000:00:02.4: [drm] Tile0: GT0: vcs3 fused off
[  275.023306] xe 0000:00:02.4: [drm] Tile0: GT0: vcs4 fused off
[  275.023309] xe 0000:00:02.4: [drm] Tile0: GT0: vcs5 fused off
[  275.023312] xe 0000:00:02.4: [drm] Tile0: GT0: vcs6 fused off
[  275.023315] xe 0000:00:02.4: [drm] Tile0: GT0: vcs7 fused off
[  275.023318] xe 0000:00:02.4: [drm] Tile0: GT0: vecs1 fused off
[  275.023321] xe 0000:00:02.4: [drm] Tile0: GT0: vecs2 fused off
[  275.023324] xe 0000:00:02.4: [drm] Tile0: GT0: vecs3 fused off
[  275.028862] [drm] Initialized xe 1.1.0 for 0000:00:02.4 on minor 4
[  275.028936] pci 0000:00:02.5: [8086:a780] type 00 class 0x038000 PCIe Root 
Complex Integrated Endpoint
[  275.028958] pci 0000:00:02.5: DMAR: Skip IOMMU disabling for graphics
[  275.029139] i915 0000:00:02.5: I915 probe blocked for Device ID a780.
[  275.029151] xe 0000:00:02.5: enabling device (0000 -> 0002)
[  275.030260] xe 0000:00:02.5: [drm] Running in SR-IOV VF mode
[  275.030266] xe 0000:00:02.5: [drm] VF: migration disabled: experimental 
feature not available on production builds
[  275.042968] xe 0000:00:02.5: [drm] Tile0: GT0: vcs1 fused off
[  275.042974] xe 0000:00:02.5: [drm] Tile0: GT0: vcs3 fused off
[  275.042977] xe 0000:00:02.5: [drm] Tile0: GT0: vcs4 fused off
[  275.042980] xe 0000:00:02.5: [drm] Tile0: GT0: vcs5 fused off
[  275.042983] xe 0000:00:02.5: [drm] Tile0: GT0: vcs6 fused off
[  275.042986] xe 0000:00:02.5: [drm] Tile0: GT0: vcs7 fused off
[  275.042989] xe 0000:00:02.5: [drm] Tile0: GT0: vecs1 fused off
[  275.042992] xe 0000:00:02.5: [drm] Tile0: GT0: vecs2 fused off
[  275.042995] xe 0000:00:02.5: [drm] Tile0: GT0: vecs3 fused off
[  275.047932] [drm] Initialized xe 1.1.0 for 0000:00:02.5 on minor 5
[  275.048093] pci 0000:00:02.6: [8086:a780] type 00 class 0x038000 PCIe Root 
Complex Integrated Endpoint
[  275.048125] pci 0000:00:02.6: DMAR: Skip IOMMU disabling for graphics
[  275.048443] i915 0000:00:02.6: I915 probe blocked for Device ID a780.
[  275.048456] xe 0000:00:02.6: enabling device (0000 -> 0002)
[  275.049613] xe 0000:00:02.6: [drm] Running in SR-IOV VF mode
[  275.049622] xe 0000:00:02.6: [drm] VF: migration disabled: experimental 
feature not available on production builds
[  275.068094] xe 0000:00:02.6: [drm] Tile0: GT0: vcs1 fused off
[  275.068100] xe 0000:00:02.6: [drm] Tile0: GT0: vcs3 fused off
[  275.068103] xe 0000:00:02.6: [drm] Tile0: GT0: vcs4 fused off
[  275.068107] xe 0000:00:02.6: [drm] Tile0: GT0: vcs5 fused off
[  275.068110] xe 0000:00:02.6: [drm] Tile0: GT0: vcs6 fused off
[  275.068113] xe 0000:00:02.6: [drm] Tile0: GT0: vcs7 fused off
[  275.068116] xe 0000:00:02.6: [drm] Tile0: GT0: vecs1 fused off
[  275.068119] xe 0000:00:02.6: [drm] Tile0: GT0: vecs2 fused off
[  275.068122] xe 0000:00:02.6: [drm] Tile0: GT0: vecs3 fused off
[  275.073227] [drm] Initialized xe 1.1.0 for 0000:00:02.6 on minor 6
[  275.073299] pci 0000:00:02.7: [8086:a780] type 00 class 0x038000 PCIe Root 
Complex Integrated Endpoint
[  275.073323] pci 0000:00:02.7: DMAR: Skip IOMMU disabling for graphics
[  275.073512] i915 0000:00:02.7: I915 probe blocked for Device ID a780.
[  275.073525] xe 0000:00:02.7: enabling device (0000 -> 0002)
[  275.074485] xe 0000:00:02.7: [drm] Running in SR-IOV VF mode
[  275.074581] xe 0000:00:02.7: [drm] VF: migration disabled: experimental 
feature not available on production builds
[  275.088133] xe 0000:00:02.7: [drm] Tile0: GT0: vcs1 fused off
[  275.088139] xe 0000:00:02.7: [drm] Tile0: GT0: vcs3 fused off
[  275.088142] xe 0000:00:02.7: [drm] Tile0: GT0: vcs4 fused off
[  275.088146] xe 0000:00:02.7: [drm] Tile0: GT0: vcs5 fused off
[  275.088149] xe 0000:00:02.7: [drm] Tile0: GT0: vcs6 fused off
[  275.088152] xe 0000:00:02.7: [drm] Tile0: GT0: vcs7 fused off
[  275.088155] xe 0000:00:02.7: [drm] Tile0: GT0: vecs1 fused off
[  275.088159] xe 0000:00:02.7: [drm] Tile0: GT0: vecs2 fused off
[  275.088162] xe 0000:00:02.7: [drm] Tile0: GT0: vecs3 fused off
[  275.093163] [drm] Initialized xe 1.1.0 for 0000:00:02.7 on minor 7
[  275.093338] xe 0000:00:02.0: [drm] PF: Enabled 7 of 7 VFs


And then domU kernel messages:
[  101.601460] xe 0000:00:07.0: [drm] Running in SR-IOV VF mode
[  101.601479] xe 0000:00:07.0: [drm] VF: migration disabled: experimental 
feature not available on production builds
[  101.616800] xe 0000:00:07.0: [drm] Tile0: GT0: vcs1 fused off
[  101.616818] xe 0000:00:07.0: [drm] Tile0: GT0: vcs3 fused off
[  101.616831] xe 0000:00:07.0: [drm] Tile0: GT0: vcs4 fused off
[  101.616844] xe 0000:00:07.0: [drm] Tile0: GT0: vcs5 fused off
[  101.616856] xe 0000:00:07.0: [drm] Tile0: GT0: vcs6 fused off
[  101.616872] xe 0000:00:07.0: [drm] Tile0: GT0: vcs7 fused off
[  101.616885] xe 0000:00:07.0: [drm] Tile0: GT0: vecs1 fused off
[  101.616899] xe 0000:00:07.0: [drm] Tile0: GT0: vecs2 fused off
[  101.616912] xe 0000:00:07.0: [drm] Tile0: GT0: vecs3 fused off
[  101.623546] [drm] Initialized xe 1.1.0 for 0000:00:07.0 on minor 0

but when actually trying to use the device, it fails:
[  177.031375] xe 0000:00:07.0: [drm] Tile0: GT0: Schedule disable failed to 
respond, guc_id=2
[  177.031520] xe 0000:00:07.0: [drm] Xe device coredump has been created
[  177.031558] xe 0000:00:07.0: [drm] Check your 
/sys/class/drm/card0/device/devcoredump/data
[  177.031598] xe 0000:00:07.0: [drm] Tile0: GT0: trying reset from 
guc_exec_queue_timedout_job [xe]
[  177.031934] xe 0000:00:07.0: [drm] Tile0: GT0: reset queued
[  177.031985] xe 0000:00:07.0: [drm] Tile0: GT0: reset started
[  177.092882] xe 0000:00:07.0: [drm] *ERROR* Tile0: GT0: GuC mmio request 
0x5507: no reply 0xffff
[  177.143396] xe 0000:00:07.0: [drm] *ERROR* Tile0: GT0: GuC mmio request 
0x5507: no reply 0x5507
[  177.211682] xe 0000:00:07.0: [drm] *ERROR* Tile0: GT0: GuC mmio request 
0x5507: no reply 0x5507
[  177.270458] xe 0000:00:07.0: [drm] *ERROR* Tile0: GT0: GuC mmio request 
0x5507: no reply 0x5507
[  177.333223] xe 0000:00:07.0: [drm] *ERROR* Tile0: GT0: GuC mmio request 
0x5507: no reply 0x5507
[  177.392473] xe 0000:00:07.0: [drm] *ERROR* Tile0: GT0: GuC mmio request 
0x5507: no reply 0x5507
[  177.447398] xe 0000:00:07.0: [drm] *ERROR* Tile0: GT0: GuC mmio request 
0x5507: no reply 0x5507
[  177.515673] xe 0000:00:07.0: [drm] *ERROR* Tile0: GT0: GuC mmio request 
0x5507: no reply 0x5507
[  177.574551] xe 0000:00:07.0: [drm] *ERROR* Tile0: GT0: GuC mmio request 
0x5507: no reply 0x5507
[  177.637391] xe 0000:00:07.0: [drm] *ERROR* Tile0: GT0: GuC mmio request 
0x5507: no reply 0x5507
[  177.637455] xe 0000:00:07.0: [drm] *ERROR* Tile0: GT0: VF: Failed to reset 
GuC state (-ETIMEDOUT)
[  177.637537] xe 0000:00:07.0: [drm] *ERROR* Tile0: GT0: reset failed 
(-ETIMEDOUT)
[  177.637610] xe 0000:00:07.0: [drm] *ERROR* CRITICAL: Xe has declared device 
0000:00:07.0 as wedged.
[  177.637610] IOCTLs and executions are blocked. Only a rebind may clear the 
failure
[  177.637610] Please file a _new_ bug report at 
https://gitlab.freedesktop.org/drm/xe/kernel/issues/new
[  177.637744] xe 0000:00:07.0: [drm] device wedged, needs recovery
[  177.637842] xe 0000:00:07.0: [drm] Tile0: GT0: Timedout job: 
seqno=4294967175, lrc_seqno=4294967175, guc_id=2, flags=0x0 in Xorg [1388]

Full logs, note multiple files there (unfortunately start of Xen log is cut):
https://gist.github.com/marmarek/fe5cdd3a82efc75a02f5b3050e6b5c89

I kinda suspect getting similar result on PTL, but I'll hold off with
the proper bug report until I can actually confirm it.
I'll repeat the test on Xen 4.22 later.

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

Attachment: signature.asc
Description: PGP signature