Re: [XEN RFC PATCH v4 0/5] IOMMU subsystem redesign and PV-IOMMU interface

["Teddy Astie" <teddy.astie@xxxxxxxxxx>]

Hello,

>>> 3. Xen complains on boot about missing endbr64 (surprisingly, it didn't
>>>      exploded):
>>>
>>>       (XEN) alt table ffff82d0404234d8 -> ffff82d040432d82
>>>       (XEN) altcall iommu_get_max_iova+0x11/0x30 dest 
>>> iommu.c#intel_iommu_get_max_iova has no endbr64
>>>       (XEN) altcall context.c#iommu_reattach_phantom+0x30/0x50 dest 
>>> iommu.c#intel_iommu_add_devfn has no endbr64
>>>       (XEN) altcall context.c#iommu_detach_phantom+0x25/0x40 dest 
>>> iommu.c#intel_iommu_remove_devfn has no endbr64
>>>       (XEN) altcall iommu_context_init+0x27/0x40 dest 
>>> iommu.c#intel_iommu_context_init has no endbr64
>>>       (XEN) altcall iommu_attach_context+0x3c/0xd0 dest 
>>> iommu.c#intel_iommu_attach has no endbr64
>>>       (XEN) altcall context.c#iommu_attach_context.cold+0x1d/0x53 dest 
>>> iommu.c#intel_iommu_detach has no endbr64
>>>       (XEN) altcall iommu_detach_context+0x37/0xa0 dest 
>>> iommu.c#intel_iommu_detach has no endbr64
>>>       (XEN) altcall iommu_reattach_context+0x95/0x240 dest 
>>> iommu.c#intel_iommu_reattach has no endbr64
>>>       (XEN) altcall context.c#iommu_reattach_context.cold+0x29/0x110 dest 
>>> iommu.c#intel_iommu_reattach has no endbr64
>>>       (XEN) altcall iommu_context_teardown+0x3f/0xa0 dest 
>>> iommu.c#intel_iommu_context_teardown has no endbr64
>>>       (XEN) altcall pci.c#deassign_device+0x99/0x270 dest 
>>> iommu.c#intel_iommu_add_devfn has no endbr64
>>>
>>
>> I also see that, but I am not sure what I need to do to fix it.
> 
> I guess add "cf_check" annotation to functions that are called
> indirectly.
> 

Will add them for v5.

>>> 4. Starting a HVM domU with PCI device fails with:
>>>
>>>       libxl: libxl_pci.c:1552:pci_add_dm_done: Domain 1:xc_assign_device 
>>> failed: No space left on device
>>>       libxl: libxl_pci.c:1875:device_pci_add_done: Domain 
>>> 1:libxl__device_pci_add failed for PCI device 0:aa:0.0 (rc -3)
>>>       libxl: libxl_create.c:2061:domcreate_attach_devices: Domain 1:unable 
>>> to add pci devices
>>>> I didn't change anything in the toolstack - maybe default context needs
>>> to be initialized somehow? But the docs suggest the default context
>>> should work out of the box. On the other hand, changelog for v4 says
>>> some parts are moved to the toolstack, but I don't see any changes in
>>> tools/ in this series...
>>>
>>
>> I only tried stuff inside Dom0, but I haven't really tried passing
>> through a device. I think I missed some step regarding quarantine domain
>> initialization, which is probably why you have "-ENOSPC" here. You can
>> try in the meantime to set "quarantine=0" to disable this part to see if
>> it progresses further.
> 
> That helped a bit. Now domU starts. But device doesn't work - qemu
> complains:
> 
> [2025-01-09 06:52:45] [00:08.0] xen_pt_realize: Real physical device 00:0d.3 
> registered successfully
> ...
> [2025-01-09 06:52:45] [00:09.0] xen_pt_realize: Real physical device 00:0d.2 
> registered successfully
> ...
> [2025-01-09 06:52:45] [00:0a.0] xen_pt_realize: Real physical device 00:0d.0 
> registered successfully
> ...
> [2025-01-09 06:52:59] [00:0a.0] xen_pt_msgctrl_reg_write: setup MSI 
> (register: 87).
> [2025-01-09 06:52:59] [00:0a.0] msi_msix_setup: Error: Mapping of MSI (err: 
> 19, vec: 0x25, entry 0[2025-01-09 06:52:59] x0)
> [2025-01-09 06:52:59] [00:0a.0] xen_pt_msgctrl_reg_write: Warning: Can not 
> map MSI (register: 86)!
> [2025-01-09 06:54:21] [00:08.0] msix_set_enable: disabling MSI-X.
> [2025-01-09 06:54:21] [00:08.0] xen_pt_msixctrl_reg_write: disable MSI-X
> [2025-01-09 06:54:21] [00:09.0] xen_pt_msixctrl_reg_write: enable MSI-X
> [2025-01-09 06:54:21] [00:09.0] msi_msix_setup: Error: Mapping of MSI-X (err: 
> 19, vec: 0xef, entry 0x0)
> [2025-01-09 06:54:21] [00:09.0] msi_msix_setup: Error: Mapping of MSI-X (err: 
> 19, vec: 0xef, entry 0x1)
> [2025-01-09 06:54:21] [00:09.0] msi_msix_setup: Error: Mapping of MSI-X (err: 
> 19, vec: 0xef, entry 0x2)
> [2025-01-09 06:54:21] [00:09.0] msi_msix_setup: Error: Mapping of MSI-X (err: 
> 19, vec: 0xef, entry 0x3)
> [2025-01-09 06:54:21] [00:09.0] msi_msix_setup: Error: Mapping of MSI-X (err: 
> 19, vec: 0xef, entry 0x4)
> [2025-01-09 06:54:21] [00:09.0] msi_msix_setup: Error: Mapping of MSI-X (err: 
> 19, vec: 0xef, entry 0x5)
> [2025-01-09 06:54:21] [00:09.0] msi_msix_setup: Error: Mapping of MSI-X (err: 
> 19, vec: 0xef, entry 0x6)
> [2025-01-09 06:54:21] [00:09.0] msi_msix_setup: Error: Mapping of MSI-X (err: 
> 19, vec: 0xef, entry 0x7)
> [2025-01-09 06:54:21] [00:09.0] msi_msix_setup: Error: Mapping of MSI-X (err: 
> 19, vec: 0xef, entry 0x8)
> [2025-01-09 06:54:21] [00:09.0] msi_msix_setup: Error: Mapping of MSI-X (err: 
> 19, vec: 0xef, entry 0x9)
> [2025-01-09 06:54:21] [00:09.0] msi_msix_setup: Error: Mapping of MSI-X (err: 
> 19, vec: 0xef, entry 0xa)
> [2025-01-09 06:54:21] [00:09.0] msi_msix_setup: Error: Mapping of MSI-X (err: 
> 19, vec: 0xef, entry 0xb)
> 
> and interestingly, Xen says all devices are still in dom0:
> 
> [2025-01-09 06:53:39] (XEN) ==== PCI devices ====
> [2025-01-09 06:53:39] (XEN) ==== segment 0000 ====
> [2025-01-09 06:53:39] (XEN) 0000:aa:00.0 - d0 - node -1
> [2025-01-09 06:53:39] (XEN) 0000:01:00.0 - d0 - node -1  - MSIs < 132 133 134 
> 135 136 >
> [2025-01-09 06:53:39] (XEN) 0000:00:1f.5 - d0 - node -1
> [2025-01-09 06:53:39] (XEN) 0000:00:1f.4 - d0 - node -1
> [2025-01-09 06:53:39] (XEN) 0000:00:1f.3 - d0 - node -1  - MSIs < 139 >
> [2025-01-09 06:53:39] (XEN) 0000:00:1f.0 - d0 - node -1
> [2025-01-09 06:53:39] (XEN) 0000:00:1d.0 - d0 - node -1  - MSIs < 131 >
> [2025-01-09 06:53:39] (XEN) 0000:00:16.0 - d0 - node -1  - MSIs < 138 >
> [2025-01-09 06:53:39] (XEN) 0000:00:15.3 - d0 - node -1
> [2025-01-09 06:53:39] (XEN) 0000:00:15.1 - d0 - node -1
> [2025-01-09 06:53:39] (XEN) 0000:00:15.0 - d0 - node -1
> [2025-01-09 06:53:39] (XEN) 0000:00:14.2 - d0 - node -1
> [2025-01-09 06:53:39] (XEN) 0000:00:14.0 - d0 - node -1
> [2025-01-09 06:53:39] (XEN) 0000:00:12.0 - d0 - node -1
> [2025-01-09 06:53:39] (XEN) 0000:00:0d.3 - d0 - node -1
> [2025-01-09 06:53:39] (XEN) 0000:00:0d.2 - d0 - node -1
> [2025-01-09 06:53:39] (XEN) 0000:00:0d.0 - d0 - node -1
> [2025-01-09 06:53:39] (XEN) 0000:00:0a.0 - d0 - node -1
> [2025-01-09 06:53:39] (XEN) 0000:00:08.0 - d0 - node -1
> [2025-01-09 06:53:39] (XEN) 0000:00:07.3 - d0 - node -1  - MSIs < 130 >
> [2025-01-09 06:53:39] (XEN) 0000:00:07.2 - d0 - node -1  - MSIs < 129 >
> [2025-01-09 06:53:39] (XEN) 0000:00:07.1 - d0 - node -1  - MSIs < 128 >
> [2025-01-09 06:53:39] (XEN) 0000:00:07.0 - d0 - node -1  - MSIs < 127 >
> [2025-01-09 06:53:39] (XEN) 0000:00:06.0 - d0 - node -1  - MSIs < 126 >
> [2025-01-09 06:53:39] (XEN) 0000:00:04.0 - d0 - node -1
> [2025-01-09 06:53:39] (XEN) 0000:00:02.0 - d0 - node -1  - MSIs < 137 >
> [2025-01-09 06:53:39] (XEN) 0000:00:00.0 - d0 - node -1
> 

I checked the PCI passthrough logic, and it looks like some bits are 
missing in my code. While they seem to be set up properly for the IOMMU 
subsystem point of view (DMA remapping at least), their domains 
(pdev->domain) are not updated. I suppose it is what confuses the 
intremap code.

Will do some additional testing with PCI passthrough and plan to fix it 
for v5.

Teddy


Teddy Astie | Vates XCP-ng Developer

XCP-ng & Xen Orchestra - Vates solutions

web: https://vates.tech