Re: PCI devices passthrough on Arm design proposal

[Julien Grall <julien@xxxxxxx>]

On 18/07/2020 10:55, Bertrand Marquis wrote:
> > On 17 Jul 2020, at 18:05, Roger Pau Monné <roger.pau@xxxxxxxxxx> wrote:
> >
> > On Fri, Jul 17, 2020 at 03:47:25PM +0000, Bertrand Marquis wrote:
> > > > On 17 Jul 2020, at 17:26, Julien Grall <julien@xxxxxxx> wrote:
> > > > On 17/07/2020 15:47, Bertrand Marquis wrote:
> > > > > > > > * Dom0Less implementation will require to have the capacity inside Xen to 
> > > > > > > > discover the PCI devices (without depending on Dom0 to declare them to Xen).
> > > > > > > >
> > > > > > > > # Enable the existing x86 virtual PCI support for ARM:
> > > > > > > >
> > > > > > > > The existing VPCI support available for X86 is adapted for Arm. When the device is added to 
> > > > > > > > XEN via the hyper call “PHYSDEVOP_pci_device_add”, VPCI handler for the config 
> > > > > > > > space access is added to the PCI device to emulate the PCI devices.
> > > > > > > >
> > > > > > > > A MMIO trap handler for the PCI ECAM space is registered in XEN so that when 
> > > > > > > > guest is trying to access the PCI config space, XEN will trap the access and 
> > > > > > > > emulate read/write using the VPCI and not the real PCI hardware.
> > > > > > > >
> > > > > > > > Limitation:
> > > > > > > > * No handler is register for the MSI configuration.
> > > > > > > > * Only legacy interrupt is supported and tested as of now, MSI is not 
> > > > > > > > implemented and tested.
> > > > > > > IIRC, legacy interrupt may be shared between two PCI devices. How do you plan 
> > > > > > > to handle this on Arm?
> > > > > We plan to fix this by adding proper support for MSI in the long term.
> > > > > For the use case where MSI is not supported or not wanted we might have to find 
> > > > > a way to forward the hardware interrupt to several guests to emulate some kind 
> > > > > of shared interrupt.
> > > > Sharing interrupts are a bit pain because you couldn't take advantage of the 
> > > > direct EOI in HW and have to be careful if one guest doesn't EOI in timely 
> > > > maneer.
> > > >
> > > > This is something I would rather avoid unless there is a real use case for it.
> > > I would expect that most recent hardware will support MSI and this
> > > will not be needed.
> > Well, PCI Express mandates MSI support, so while this is just a spec,
> > I would expect most (if not all) devices to support MSI (or MSI-X), as
> > Arm platforms haven't implemented legacy PCI anyway.
> Yes that’s our assumption to. But we have to start somewhere so MSI is
> planned but in a future step. I would think that supporting non MSI if not
> impossible will be a lot more complex due to the interrupt sharing.
> I do think that not supporting non MSI should be ok on Arm.
>
> > > When MSI is not used, the only solution would be to enforce that
> > > devices assigned to different guest are using different interrupts
> > > which would limit the number of domains being able to use PCI
> > > devices on a bus to 4 (if the enumeration can be modified correctly
> > > to assign the interrupts properly).
> > >
> > > If we all agree that this is an acceptable limitation then we would
> > > not need the “interrupt sharing”.
> > I might be easier to start by just supporting devices that have MSI
> > (or MSI-X) and then move to legacy interrupts if required?
> MSI support requires also some support in the interrupt controller part
> on arm. So there is some work to achieve that.
>
> > You should have most of the pieces you require already implemented
> > since that's what x86 uses, and hence could reuse almost all of it?
> Inside PCI probably but the GIC part will require some work.
We already have an ITS implementation in Xen. This is required in order 
to use PCI devices in DOM0 on thunder-x (there is no legacy interrupts 
supported).
It wasn't yet exposed to the guest because we didn't fully investigate 
the security aspect of the implementation. However, for a tech preview 
this should be sufficient.

--
Julien Grall