Re: [Xen-devel] [RFC + Queries] Flow of PCI passthrough in ARM

[Konrad Rzeszutek Wilk <konrad.wilk@xxxxxxxxxx>]

On Wed, Oct 08, 2014 at 05:16:53PM +0530, manish jaggi wrote:
> On 7 October 2014 23:47, Stefano Stabellini
> <stefano.stabellini@xxxxxxxxxxxxx> wrote:
> > As the discussion is becoming Xen specific, reduce the CC list.
> >
> > On Mon, 6 Oct 2014, manish jaggi wrote:
> >> On 6 October 2014 19:41, Stefano Stabellini
> >> <stefano.stabellini@xxxxxxxxxxxxx> wrote:
> >> > On Mon, 6 Oct 2014, manish jaggi wrote:
> >> >> Below is the Design Flow:
> >> >>
> >> >> =====
> >> >> Xen PCI Passthrough support for ARM
> >> >>
> >> >> ----------------------------------------------
> >> >>
> >> >>
> >> >> Nomenclature:
> >> >>
> >> >> SBDF - segment:bus:device.function.
> >> >> Segment - Number of the PCIe RootComplex. (this is also referred to as
> >> >> domain in linux terminology Domain:BDF)
> >> >> domU sbdf - refers to the sbdf of the device assigned when enumerated
> >> >> in domU. This is different from the actual sbdf.
> >> >>
> >> >>
> >> >> What is the requirement
> >> >>
> >> >> 1. Any PCI device be assigned to any domU at runtime.
> >> >> 2. The assignment should not be static, system admin must be able to
> >> >> assign a PCI BDF at will.
> >> >> 3. SMMU should be programmed for the device/domain pair
> >> >> 4. MSI should be directly injected into domU.
> >> >> 5. Access to GIC should be trapped in Xen to program MSI(LPI) in ITS
> >> >> 6. FrontEnd - Backend communication between DomU-Dom0 must be limited
> >> >> to PCI config read writes.
> >> >>
> >> >> What is support today
> >> >>
> >> >> 1. Non PCI passthrough using a device tree. A device tree node can be
> >> >> passthrough to a domU.
> >> >> 2. SMMU is programmed (devices' stream ID is allowed to access domU
> >> >> guest memory)
> >> >> 3. By default a device is assigned to dom0 if not done a pci-hide.
> >> >> (SMMU for that device is programmed for dom0)
> >> >>
> >> >>
> >> >> Proposed Flow:
> >> >>
> >> >> 1. Xen parses its device tree to find the pci nodes. There nodes
> >> >> represent the PCIe Root Complexes
> >> >>
> >> >> 2. Xen parses its device tree to find smmu nodes which have mmu-master
> >> >> property which should point to the PCIe RCs (pci nodes)
> >> >> Note: The mapping between the pci node and the smmu node is based on:
> >> >> which smmu which translates r/w requests from that pcie
> >> >>
> >> >> 3. All the pci nodes are assigned to the dom0.
> >> >> 4. dom0 boots
> >> >>
> >> >> 5. dom0 enumerates all the PCI devices and calls xen hypercall
> >> >> PHYSDEVOP_pci_device_add. This in-turn programs the smmu
> >> >> Note: Also The MMIO regions for the device are assigned to the dom0.
> >> >>
> >> >> 6. dom0 boots domU.
> >> >> Note: in domU, the pcifront bus has a msi-controller attached. It is
> >> >> assumed that the domU device tree contains the 'gicv3-its' node.
> >> >>
> >> >> 7. dom0 calls xl pci-assignable-add <sbdf>
> >> >>
> >> >> 8. dom0 calls xl pci-attach <domU_id> '<sbdf>,permissive=1'
> >> >> Note: XEN_DOMCTL_assign_device is called. Implemented assign_device in 
> >> >> smmu.c
> >> >> The MMIO regions for the device are assigned to the respective domU.
> >> >>
> >> >> 9. Front end driver is notified by xenwatch daemon and it starts
> >> >> enumerating devices.
> >> >> Note: the check of initial_domain in register_xen_pci_notifier() is 
> >> >> removed.
> >> >>
> >> >>
> >> >> 10. The device driver requests the msi using pcie_enable_msi, which in
> >> >> turn calls the its driver which tries to issue commands like MAPD,
> >> >> MAPVI. which are trapped by the Xen ITS emulation driver.
> >> >> 10a. The MAPD command contains a device id, which is a stream ID
> >> >> constructed using the sbdf. The sbdf is specific to domU for that
> >> >> device.
> >> >> 10b. Xen ITS driver has to replace the domU sbdf to the actual sdbf
> >> >> and program the command in the command queue.
> >> >>
> >> >> Now there is a _problem_ that xen does not know the mapping between
> >> >> domU streamid (sbdf) and actual sbdf.
> >> >> For ex if two pci devices are assigned to domU, xen does not know that
> >> >> which domU sbdf maps to which pci device
> >> >> Thus the Xen ITS driver fails to program the MAPD command in command
> >> >> queue, which results LPIs not programmed for that device
> >> >>
> >> >> At the time xl pci-attach is called the domU sbdf of the device is not
> >> >> known, as the enumeration of the PCI device in domU has not started by
> >> >> that time.
> >> >> in xl pci-attach a virtual slot number can be provided but it is not
> >> >> used in ITS commands in domU.
> >> >>
> >> >> If it is known somehow (__we need help on this__) then dom0 could
> >> >> issue a hypercall to xen to map domU sbdf to actual in the following
> >> >> format
> >> >>
> >> >> PHYSDEVOP_map_domU_sbdf{
> >> >>
> >> >> actual sBDF,
> >> >> domU enumerated sBDF
> >> >> and domU_ID.
> >> >>
> >> >> }
> >> >
> >> > Now the problem is much much clearer, thank you!
> >> >
> >> > Actually the xen-pcifront driver in the guest knows the real PCI sbdf
> >> > for the assigned device, not just the virtual slot.
> >> Could you please help in the data structure where it is stored.
> >
> > Actually I was wrong. It is true that the real sbdf is exposed to the
> > guest via xenstore (see the dev-0 backend node) but it is not currently
> > read by the guest and probably it shouldn't be readable in the first
> > place. It's best not to rely on it.
> >
> > The virtual to physical sbdf mapping is done by pciback, see
> > drivers/xen/xen-pciback/xenbus.c and drivers/xen/xen-pciback/vpci.c.
> > Pciback should be one telling Xen what the mapping is.
> > Unfortunately I think that you'll probably have to introduce a new
> > hypercall to do it.
> >
> >
> >
> >> PS: in this mail thread you were averse to the fact that why guest
> >> should know the real sbdf.
> >
> > I am averse to having to rely on real StreamIDs being exposed and used
> > in the guest virtual ITS.
> >
> >
> >> >On x86 xen-pcifront
> >> > makes an hypercall to enable msi/msix on the device passing the real
> >> > sbdf as argument:
> >> On ARM since for non msi interrupts (SPI, SGI's) GIC access is
> >> directly trapped in Xen, it makes sense to use traps for LPIs
> >> The same logic is being proposed. Apart from PCI config space r/w the
> >> proposal is not to use front-end back-end communication for MSIs.
> >> In the long run, I plan to trap directly into Xen for PCI config space r/w.
> >> Also in our previous mails we agreed on utilising arm trap and emulate for 
> >> MSI.
> >
> > OK
> >
> >
> >> >
> >> > drivers/pci/xen-pcifront.c:pci_frontend_enable_msix
> >> >
> >> > Could we use the same hypercall to enable msi/msix on ARM? That would be
> >> > ideal.
> >> >
> >> > Otherwise xen-pcifront could call a new hypercall to let Xen know the
> >> > virtual sbdf to sbdf mapping. But I would prefer not to introduce a new
> >> > hypercall and reuse the existing one.
> >> We are proposing removal of pcifront back communication for MSIs. I do
> >> not want to introduce a new hypercall
> >> If we know the guest sbdf it can be done in the pci-attach DOMCTL itself.
> >
> > I agree that it would be nice to avoid a new hypercall but it might be
> > the only way to do it. Dig through the pciback code and see what you can
> > do, I am open to alternatives.
> > pciback code and see what you can do.
> Stefano, I need your help on this, can you point out someone who can help.
> I am adding Ryan Wilson the author as well. Is he still active on Xen ?

No, but what is the help you need?

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel