Re: [Xen-devel] [RFC + Queries] Flow of PCI passthrough in ARM

[manish jaggi <manishjaggi.oss@xxxxxxxxx>]

On 7 October 2014 23:47, Stefano Stabellini
<stefano.stabellini@xxxxxxxxxxxxx> wrote:
> As the discussion is becoming Xen specific, reduce the CC list.
>
> On Mon, 6 Oct 2014, manish jaggi wrote:
>> On 6 October 2014 19:41, Stefano Stabellini
>> <stefano.stabellini@xxxxxxxxxxxxx> wrote:
>> > On Mon, 6 Oct 2014, manish jaggi wrote:
>> >> Below is the Design Flow:
>> >>
>> >> =====
>> >> Xen PCI Passthrough support for ARM
>> >>
>> >> ----------------------------------------------
>> >>
>> >>
>> >> Nomenclature:
>> >>
>> >> SBDF - segment:bus:device.function.
>> >> Segment - Number of the PCIe RootComplex. (this is also referred to as
>> >> domain in linux terminology Domain:BDF)
>> >> domU sbdf - refers to the sbdf of the device assigned when enumerated
>> >> in domU. This is different from the actual sbdf.
>> >>
>> >>
>> >> What is the requirement
>> >>
>> >> 1. Any PCI device be assigned to any domU at runtime.
>> >> 2. The assignment should not be static, system admin must be able to
>> >> assign a PCI BDF at will.
>> >> 3. SMMU should be programmed for the device/domain pair
>> >> 4. MSI should be directly injected into domU.
>> >> 5. Access to GIC should be trapped in Xen to program MSI(LPI) in ITS
>> >> 6. FrontEnd - Backend communication between DomU-Dom0 must be limited
>> >> to PCI config read writes.
>> >>
>> >> What is support today
>> >>
>> >> 1. Non PCI passthrough using a device tree. A device tree node can be
>> >> passthrough to a domU.
>> >> 2. SMMU is programmed (devices' stream ID is allowed to access domU
>> >> guest memory)
>> >> 3. By default a device is assigned to dom0 if not done a pci-hide.
>> >> (SMMU for that device is programmed for dom0)
>> >>
>> >>
>> >> Proposed Flow:
>> >>
>> >> 1. Xen parses its device tree to find the pci nodes. There nodes
>> >> represent the PCIe Root Complexes
>> >>
>> >> 2. Xen parses its device tree to find smmu nodes which have mmu-master
>> >> property which should point to the PCIe RCs (pci nodes)
>> >> Note: The mapping between the pci node and the smmu node is based on:
>> >> which smmu which translates r/w requests from that pcie
>> >>
>> >> 3. All the pci nodes are assigned to the dom0.
>> >> 4. dom0 boots
>> >>
>> >> 5. dom0 enumerates all the PCI devices and calls xen hypercall
>> >> PHYSDEVOP_pci_device_add. This in-turn programs the smmu
>> >> Note: Also The MMIO regions for the device are assigned to the dom0.
>> >>
>> >> 6. dom0 boots domU.
>> >> Note: in domU, the pcifront bus has a msi-controller attached. It is
>> >> assumed that the domU device tree contains the 'gicv3-its' node.
>> >>
>> >> 7. dom0 calls xl pci-assignable-add <sbdf>
>> >>
>> >> 8. dom0 calls xl pci-attach <domU_id> '<sbdf>,permissive=1'
>> >> Note: XEN_DOMCTL_assign_device is called. Implemented assign_device in 
>> >> smmu.c
>> >> The MMIO regions for the device are assigned to the respective domU.
>> >>
>> >> 9. Front end driver is notified by xenwatch daemon and it starts
>> >> enumerating devices.
>> >> Note: the check of initial_domain in register_xen_pci_notifier() is 
>> >> removed.
>> >>
>> >>
>> >> 10. The device driver requests the msi using pcie_enable_msi, which in
>> >> turn calls the its driver which tries to issue commands like MAPD,
>> >> MAPVI. which are trapped by the Xen ITS emulation driver.
>> >> 10a. The MAPD command contains a device id, which is a stream ID
>> >> constructed using the sbdf. The sbdf is specific to domU for that
>> >> device.
>> >> 10b. Xen ITS driver has to replace the domU sbdf to the actual sdbf
>> >> and program the command in the command queue.
>> >>
>> >> Now there is a _problem_ that xen does not know the mapping between
>> >> domU streamid (sbdf) and actual sbdf.
>> >> For ex if two pci devices are assigned to domU, xen does not know that
>> >> which domU sbdf maps to which pci device
>> >> Thus the Xen ITS driver fails to program the MAPD command in command
>> >> queue, which results LPIs not programmed for that device
>> >>
>> >> At the time xl pci-attach is called the domU sbdf of the device is not
>> >> known, as the enumeration of the PCI device in domU has not started by
>> >> that time.
>> >> in xl pci-attach a virtual slot number can be provided but it is not
>> >> used in ITS commands in domU.
>> >>
>> >> If it is known somehow (__we need help on this__) then dom0 could
>> >> issue a hypercall to xen to map domU sbdf to actual in the following
>> >> format
>> >>
>> >> PHYSDEVOP_map_domU_sbdf{
>> >>
>> >> actual sBDF,
>> >> domU enumerated sBDF
>> >> and domU_ID.
>> >>
>> >> }
>> >
>> > Now the problem is much much clearer, thank you!
>> >
>> > Actually the xen-pcifront driver in the guest knows the real PCI sbdf
>> > for the assigned device, not just the virtual slot.
>> Could you please help in the data structure where it is stored.
>
> Actually I was wrong. It is true that the real sbdf is exposed to the
> guest via xenstore (see the dev-0 backend node) but it is not currently
> read by the guest and probably it shouldn't be readable in the first
> place. It's best not to rely on it.
>
> The virtual to physical sbdf mapping is done by pciback, see
> drivers/xen/xen-pciback/xenbus.c and drivers/xen/xen-pciback/vpci.c.
> Pciback should be one telling Xen what the mapping is.
> Unfortunately I think that you'll probably have to introduce a new
> hypercall to do it.
>
>
>
>> PS: in this mail thread you were averse to the fact that why guest
>> should know the real sbdf.
>
> I am averse to having to rely on real StreamIDs being exposed and used
> in the guest virtual ITS.
>
>
>> >On x86 xen-pcifront
>> > makes an hypercall to enable msi/msix on the device passing the real
>> > sbdf as argument:
>> On ARM since for non msi interrupts (SPI, SGI's) GIC access is
>> directly trapped in Xen, it makes sense to use traps for LPIs
>> The same logic is being proposed. Apart from PCI config space r/w the
>> proposal is not to use front-end back-end communication for MSIs.
>> In the long run, I plan to trap directly into Xen for PCI config space r/w.
>> Also in our previous mails we agreed on utilising arm trap and emulate for 
>> MSI.
>
> OK
>
>
>> >
>> > drivers/pci/xen-pcifront.c:pci_frontend_enable_msix
>> >
>> > Could we use the same hypercall to enable msi/msix on ARM? That would be
>> > ideal.
>> >
>> > Otherwise xen-pcifront could call a new hypercall to let Xen know the
>> > virtual sbdf to sbdf mapping. But I would prefer not to introduce a new
>> > hypercall and reuse the existing one.
>> We are proposing removal of pcifront back communication for MSIs. I do
>> not want to introduce a new hypercall
>> If we know the guest sbdf it can be done in the pci-attach DOMCTL itself.
>
> I agree that it would be nice to avoid a new hypercall but it might be
> the only way to do it. Dig through the pciback code and see what you can
> do, I am open to alternatives.
> pciback code and see what you can do.
Stefano, I need your help on this, can you point out someone who can help.
I am adding Ryan Wilson the author as well. Is he still active on Xen ?

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel