Re: [XEN RFC PATCH v4 0/5] IOMMU subsystem redesign and PV-IOMMU interface

[Marek Marczykowski-Górecki <marmarek@xxxxxxxxxxxxxxxxxxxxxx>]

On Mon, Nov 04, 2024 at 02:28:38PM +0000, Teddy Astie wrote:
> This work has been presented at Xen Summit 2024 during the
>   IOMMU paravirtualization and Xen IOMMU subsystem rework
> design session.
> 
> Operating systems may want to have access to a IOMMU in order to do DMA
> protection or implement certain features (e.g VFIO on Linux).
> 
> VFIO support is mandatory for framework such as SPDK, which can be useful to
> implement an alternative storage backend for virtual machines [1].
> 
> In this patch series, we introduce in Xen the ability to manage several
> contexts per domain and provide a new hypercall interface to allow guests
> to manage IOMMU contexts.
> 
> The VT-d driver is updated to support these new features.
> 
> [1] Using SPDK with the Xen hypervisor - FOSDEM 2023
> ---
> Changed in v2 :
> * fixed Xen crash when dumping IOMMU contexts (using X debug key)
> with DomUs without IOMMU
> * s/dettach/detach/
> * removed some unused includes
> * fix dangling devices in contexts with detach
> 
> Changed in v3 :
> * lock entirely map/unmap in hypercall
> * prevent IOMMU operations on dying contexts (fix race condition)
> * iommu_check_context+iommu_get_context -> iommu_get_context and check for 
> NULL
> 
> Changed in v4 :
> * Part of initialization logic is moved to domain or toolstack (IOMMU_init)
>   + domain/toolstack now decides on "context count" and "pagetable pool size"
>   + for now, all domains are able to initialize PV-IOMMU
> * introduce "dom0-iommu=no-dma" to make default context block all DMA
>   (disables HAP and sync-pt), enforcing usage of PV-IOMMU for DMA
>   Can be used to expose properly "Pre-boot DMA protection"
> * redesigned locking logic for contexts
>   + contexts are accessed using iommu_get_context and released with 
> iommu_put_context
> 
> TODO:
> * add stub implementations for bissecting needs and non-ported IOMMU 
> implementations
> * fix some issues with no-dma+PV and grants
> * complete "no-dma" mode (expose to toolstack, add documentation, ...)
> * properly define nested mode and PASID support

Hi,

I finally got time to try this revision (sorry it took so long!). My
goal was to test it this time with some HVM domU too. I didn't get very
far...

Issues I hit:

1. AMD IOMMU driver is not converted (fails to build), for now disabled
   CONFIG_AMD_IOMMU.
2. PV shim build fails (linker fails to find p2m_add_identity_entry
   symbol referenced from iommu.c)
3. Xen complains on boot about missing endbr64 (surprisingly, it didn't
   exploded):

    (XEN) alt table ffff82d0404234d8 -> ffff82d040432d82
    (XEN) altcall iommu_get_max_iova+0x11/0x30 dest 
iommu.c#intel_iommu_get_max_iova has no endbr64
    (XEN) altcall context.c#iommu_reattach_phantom+0x30/0x50 dest 
iommu.c#intel_iommu_add_devfn has no endbr64
    (XEN) altcall context.c#iommu_detach_phantom+0x25/0x40 dest 
iommu.c#intel_iommu_remove_devfn has no endbr64
    (XEN) altcall iommu_context_init+0x27/0x40 dest 
iommu.c#intel_iommu_context_init has no endbr64
    (XEN) altcall iommu_attach_context+0x3c/0xd0 dest 
iommu.c#intel_iommu_attach has no endbr64
    (XEN) altcall context.c#iommu_attach_context.cold+0x1d/0x53 dest 
iommu.c#intel_iommu_detach has no endbr64
    (XEN) altcall iommu_detach_context+0x37/0xa0 dest 
iommu.c#intel_iommu_detach has no endbr64
    (XEN) altcall iommu_reattach_context+0x95/0x240 dest 
iommu.c#intel_iommu_reattach has no endbr64
    (XEN) altcall context.c#iommu_reattach_context.cold+0x29/0x110 dest 
iommu.c#intel_iommu_reattach has no endbr64
    (XEN) altcall iommu_context_teardown+0x3f/0xa0 dest 
iommu.c#intel_iommu_context_teardown has no endbr64
    (XEN) altcall pci.c#deassign_device+0x99/0x270 dest 
iommu.c#intel_iommu_add_devfn has no endbr64

4. Starting a HVM domU with PCI device fails with:

    libxl: libxl_pci.c:1552:pci_add_dm_done: Domain 1:xc_assign_device failed: 
No space left on device
    libxl: libxl_pci.c:1875:device_pci_add_done: Domain 1:libxl__device_pci_add 
failed for PCI device 0:aa:0.0 (rc -3)
    libxl: libxl_create.c:2061:domcreate_attach_devices: Domain 1:unable to add 
pci devices

I didn't change anything in the toolstack - maybe default context needs
to be initialized somehow? But the docs suggest the default context
should work out of the box. On the other hand, changelog for v4 says
some parts are moved to the toolstack, but I don't see any changes in
tools/ in this series...

FWIW The exact version I tried is this (this series, on top of staging +
qubes patches):
https://github.com/QubesOS/qubes-vmm-xen/pull/200
At this stage, dom0 kernel didn't have PV-IOMMU driver included yet.

Full Xen log, with some debug info collected:
https://gist.github.com/marmarek/e7ac2571df033c7181bf03f21aa5f9ab

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

Attachment: signature.asc
Description: PGP signature