Re: [Xen-devel] [RFC + Queries] Flow of PCI passthrough in ARM

[Stefano Stabellini <stefano.stabellini@xxxxxxxxxxxxx>]

As the discussion is becoming Xen specific, reduce the CC list.

On Mon, 6 Oct 2014, manish jaggi wrote:
> On 6 October 2014 19:41, Stefano Stabellini
> <stefano.stabellini@xxxxxxxxxxxxx> wrote:
> > On Mon, 6 Oct 2014, manish jaggi wrote:
> >> Below is the Design Flow:
> >>
> >> =====
> >> Xen PCI Passthrough support for ARM
> >>
> >> ----------------------------------------------
> >>
> >>
> >> Nomenclature:
> >>
> >> SBDF - segment:bus:device.function.
> >> Segment - Number of the PCIe RootComplex. (this is also referred to as
> >> domain in linux terminology Domain:BDF)
> >> domU sbdf - refers to the sbdf of the device assigned when enumerated
> >> in domU. This is different from the actual sbdf.
> >>
> >>
> >> What is the requirement
> >>
> >> 1. Any PCI device be assigned to any domU at runtime.
> >> 2. The assignment should not be static, system admin must be able to
> >> assign a PCI BDF at will.
> >> 3. SMMU should be programmed for the device/domain pair
> >> 4. MSI should be directly injected into domU.
> >> 5. Access to GIC should be trapped in Xen to program MSI(LPI) in ITS
> >> 6. FrontEnd - Backend communication between DomU-Dom0 must be limited
> >> to PCI config read writes.
> >>
> >> What is support today
> >>
> >> 1. Non PCI passthrough using a device tree. A device tree node can be
> >> passthrough to a domU.
> >> 2. SMMU is programmed (devices' stream ID is allowed to access domU
> >> guest memory)
> >> 3. By default a device is assigned to dom0 if not done a pci-hide.
> >> (SMMU for that device is programmed for dom0)
> >>
> >>
> >> Proposed Flow:
> >>
> >> 1. Xen parses its device tree to find the pci nodes. There nodes
> >> represent the PCIe Root Complexes
> >>
> >> 2. Xen parses its device tree to find smmu nodes which have mmu-master
> >> property which should point to the PCIe RCs (pci nodes)
> >> Note: The mapping between the pci node and the smmu node is based on:
> >> which smmu which translates r/w requests from that pcie
> >>
> >> 3. All the pci nodes are assigned to the dom0.
> >> 4. dom0 boots
> >>
> >> 5. dom0 enumerates all the PCI devices and calls xen hypercall
> >> PHYSDEVOP_pci_device_add. This in-turn programs the smmu
> >> Note: Also The MMIO regions for the device are assigned to the dom0.
> >>
> >> 6. dom0 boots domU.
> >> Note: in domU, the pcifront bus has a msi-controller attached. It is
> >> assumed that the domU device tree contains the 'gicv3-its' node.
> >>
> >> 7. dom0 calls xl pci-assignable-add <sbdf>
> >>
> >> 8. dom0 calls xl pci-attach <domU_id> '<sbdf>,permissive=1'
> >> Note: XEN_DOMCTL_assign_device is called. Implemented assign_device in 
> >> smmu.c
> >> The MMIO regions for the device are assigned to the respective domU.
> >>
> >> 9. Front end driver is notified by xenwatch daemon and it starts
> >> enumerating devices.
> >> Note: the check of initial_domain in register_xen_pci_notifier() is 
> >> removed.
> >>
> >>
> >> 10. The device driver requests the msi using pcie_enable_msi, which in
> >> turn calls the its driver which tries to issue commands like MAPD,
> >> MAPVI. which are trapped by the Xen ITS emulation driver.
> >> 10a. The MAPD command contains a device id, which is a stream ID
> >> constructed using the sbdf. The sbdf is specific to domU for that
> >> device.
> >> 10b. Xen ITS driver has to replace the domU sbdf to the actual sdbf
> >> and program the command in the command queue.
> >>
> >> Now there is a _problem_ that xen does not know the mapping between
> >> domU streamid (sbdf) and actual sbdf.
> >> For ex if two pci devices are assigned to domU, xen does not know that
> >> which domU sbdf maps to which pci device
> >> Thus the Xen ITS driver fails to program the MAPD command in command
> >> queue, which results LPIs not programmed for that device
> >>
> >> At the time xl pci-attach is called the domU sbdf of the device is not
> >> known, as the enumeration of the PCI device in domU has not started by
> >> that time.
> >> in xl pci-attach a virtual slot number can be provided but it is not
> >> used in ITS commands in domU.
> >>
> >> If it is known somehow (__we need help on this__) then dom0 could
> >> issue a hypercall to xen to map domU sbdf to actual in the following
> >> format
> >>
> >> PHYSDEVOP_map_domU_sbdf{
> >>
> >> actual sBDF,
> >> domU enumerated sBDF
> >> and domU_ID.
> >>
> >> }
> >
> > Now the problem is much much clearer, thank you!
> >
> > Actually the xen-pcifront driver in the guest knows the real PCI sbdf
> > for the assigned device, not just the virtual slot.
> Could you please help in the data structure where it is stored.

Actually I was wrong. It is true that the real sbdf is exposed to the
guest via xenstore (see the dev-0 backend node) but it is not currently
read by the guest and probably it shouldn't be readable in the first
place. It's best not to rely on it.

The virtual to physical sbdf mapping is done by pciback, see
drivers/xen/xen-pciback/xenbus.c and drivers/xen/xen-pciback/vpci.c.
Pciback should be one telling Xen what the mapping is.
Unfortunately I think that you'll probably have to introduce a new
hypercall to do it.



> PS: in this mail thread you were averse to the fact that why guest
> should know the real sbdf.

I am averse to having to rely on real StreamIDs being exposed and used
in the guest virtual ITS.


> >On x86 xen-pcifront
> > makes an hypercall to enable msi/msix on the device passing the real
> > sbdf as argument:
> On ARM since for non msi interrupts (SPI, SGI's) GIC access is
> directly trapped in Xen, it makes sense to use traps for LPIs
> The same logic is being proposed. Apart from PCI config space r/w the
> proposal is not to use front-end back-end communication for MSIs.
> In the long run, I plan to trap directly into Xen for PCI config space r/w.
> Also in our previous mails we agreed on utilising arm trap and emulate for 
> MSI.

OK

 
> >
> > drivers/pci/xen-pcifront.c:pci_frontend_enable_msix
> >
> > Could we use the same hypercall to enable msi/msix on ARM? That would be
> > ideal.
> >
> > Otherwise xen-pcifront could call a new hypercall to let Xen know the
> > virtual sbdf to sbdf mapping. But I would prefer not to introduce a new
> > hypercall and reuse the existing one.
> We are proposing removal of pcifront back communication for MSIs. I do
> not want to introduce a new hypercall
> If we know the guest sbdf it can be done in the pci-attach DOMCTL itself.

I agree that it would be nice to avoid a new hypercall but it might be
the only way to do it. Dig through the pciback code and see what you can
do, I am open to alternatives.
pciback code and see what you can do.

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel