Re: [XEN RFC PATCH v5 0/5] IOMMU subsystem redesign and PV-IOMMU interface

[Marek Marczykowski-Górecki <marmarek@xxxxxxxxxxxxxxxxxxxxxx>]

On Thu, Jan 23, 2025 at 01:45:40PM +0100, Marek Marczykowski-Górecki wrote:
> On Tue, Jan 21, 2025 at 04:13:20PM +0000, Teddy Astie wrote:
> > This work has been presented at Xen Summit 2024 during the
> >   IOMMU paravirtualization and Xen IOMMU subsystem rework
> > design session.
> > 
> > Operating systems may want to have access to a IOMMU in order to do DMA
> > protection or implement certain features (e.g VFIO on Linux).
> > 
> > VFIO support is mandatory for framework such as SPDK, which can be useful to
> > implement an alternative storage backend for virtual machines [1].
> > 
> > In this patch series, we introduce in Xen the ability to manage several
> > contexts per domain and provide a new hypercall interface to allow guests
> > to manage IOMMU contexts.
> > 
> > The VT-d driver is updated to support these new features.
> > 
> > [1] Using SPDK with the Xen hypervisor - FOSDEM 2023
> > ---
> > Cc: Marek Marczykowski-Górecki <marmarek@xxxxxxxxxxxxxxxxxxxxxx>
> > 
> > PCI Passthrough now work on my side, but things are still feels quite 
> > brittle.
> > 
> > Changed in v2 :
> > * fixed Xen crash when dumping IOMMU contexts (using X debug key)
> > with DomUs without IOMMU
> > * s/dettach/detach/
> > * removed some unused includes
> > * fix dangling devices in contexts with detach
> > 
> > Changed in v3 :
> > * lock entirely map/unmap in hypercall
> > * prevent IOMMU operations on dying contexts (fix race condition)
> > * iommu_check_context+iommu_get_context -> iommu_get_context and check for 
> > NULL
> > 
> > Changed in v4 :
> > * Part of initialization logic is moved to domain or toolstack (IOMMU_init)
> >   + domain/toolstack now decides on "context count" and "pagetable pool 
> > size"
> >   + for now, all domains are able to initialize PV-IOMMU
> > * introduce "dom0-iommu=no-dma" to make default context block all DMA
> >   (disables HAP and sync-pt), enforcing usage of PV-IOMMU for DMA
> >   Can be used to expose properly "Pre-boot DMA protection"
> > * redesigned locking logic for contexts
> >   + contexts are accessed using iommu_get_context and released with 
> > iommu_put_context
> > 
> > Changed in v5 :
> > * various PCI Passthrough related fixes
> >   + rewrote parts of PCI Passthrough logic
> >   + various other related bug fixes
> > * simplified VT-d DID (for hardware) management by only having one map 
> > instead of two
> >   (pseudo_domid map was previously used for old quarantine code then 
> > recycled for PV-IOMMU
> >    in addition to another map also tracing Domain<->VT-d DID, now there is 
> > only one
> >    map tracking both making things simpler)
> > * reworked parts of Xen quarantine logic (needed for PCI Passthrough)
> > * added cf_check annotations
> > * some changes to PV-IOMMU headers (Alejandro)
> > 
> > TODO:
> > * add stub implementations for bissecting needs and non-ported IOMMU 
> > implementations
> > * fix some issues with no-dma+PV and grants
> > * complete "no-dma" mode (expose to toolstack, add documentation, ...)
> > * properly define nested mode and PASID support
> > 
> > * make new quarantine code more unity region aware (isolate devices with
> >   different reserved regions regions using separate 'contexts')
> > * find a way to make PV-IOMMU work in DomUs (they don't see machine bdf)
> > * there are corner cases with PV-IOMMU and to-domain Xen PCI Passthrough
> >   (e.g pci-assignable-remove will reassign to context 0, while the driver
> >    expects the device to to be in context X)
> 
> Thanks for the updated patches. I have run them through gitlab-ci, and
> here are some observations:
> - I needed to disable CONFIG_AMD_IOMMU (it fails to build, as expected at 
> this point)
> - I needed to disable pvshim (it fails to build)
> - fails to build with clang: 
> https://gitlab.com/xen-project/people/marmarek/xen/-/jobs/8931373789/viewer#L3525
> - gcc-ibt build fails: 
> https://gitlab.com/xen-project/people/marmarek/xen/-/jobs/8931373785#L1314
> - fails to build for ARM (both 32 and 64) and PPC64
> - QEMU smoke test panic with PV dom0, looks like it runs on AMD, so it
>   may be related to the disabled CONFIG_AMD_IOMMU, but I wouldn't expect
>   it to panic on _PV_ dom0 boot...
> - PVH dom0 fails to boot (on real hw) with a lot of VT-d faults: 
> https://gitlab.com/xen-project/people/marmarek/xen/-/jobs/8931373875
> - PCI passthrough (with PV dom0) results in a lot of VT-d faults: 
> https://gitlab.com/xen-project/people/marmarek/xen/-/jobs/8931373881
> 
> Note this uses only this series, but plain Linux (appears to be 6.1.19).
> IIUC if one doesn't try to configure PV-IOMMU specifically (non-default
> contexts) it should still work.
> 
> BTW Linux says it detected "Xen version 4.19." - shouldn't it report
> 4.20 already at this point in release cycle?
> 
> All results:
> https://gitlab.com/xen-project/people/marmarek/xen/-/pipelines/1637849303

FWIW the test run rebased on staging looks similar:
https://gitlab.com/xen-project/people/marmarek/xen/-/pipelines/1638019332

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

Attachment: signature.asc
Description: PGP signature