[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] [RFC] ARM PCI Passthrough design document

On 05/29/2017 03:30 AM, Manish Jaggi wrote:
Hi Julien,

Hello Manish,

On 5/26/2017 10:44 PM, Julien Grall wrote:
PCI pass-through allows the guest to receive full control of physical PCI
devices. This means the guest will have full and direct access to the PCI

ARM is supporting a kind of guest that exploits as much as possible
virtualization support in hardware. The guest will rely on PV driver only
for IO (e.g block, network) and interrupts will come through the
interrupt controller, therefore there are no big changes required
within the

As a consequence, it would be possible to replace PV drivers by
assigning real
devices to the guest for I/O access. Xen on ARM would therefore be
able to
run unmodified operating system.

To achieve this goal, it looks more sensible to go towards emulating the
host bridge (there will be more details later).
IIUC this means that domU would have an emulated host bridge and dom0
will see the actual host bridge?

You don't want the hardware domain and Xen access the configuration space at the same time. So if Xen is in charge of the host bridge, then an emulated host bridge should be exposed to the hardware.

Although, this is depending on who is in charge of the the host bridge. As you may have noticed, this design document is proposing two ways to handle configuration space access. At the moment any generic host bridge (see the definition in the design document) will be handled in Xen and the hardware domain will have an emulated host bridge.

If your host bridges is not a generic one, then the hardware domain will be in charge of the host bridges, any configuration access from Xen will be forward to the hardware domain.

At the moment, as part of the first implementation, we are only looking to implement a generic host bridge in Xen. We will decide on case by case basis for all the other host bridges whether we want to have the driver in Xen.



The IOMMU will be used to isolate the PCI device when accessing the
memory (e.g
DMA and MSI Doorbells). Often the IOMMU will be configured using a
(aka StreamID for ARM SMMU)  that can be deduced from the SBDF with
the help
of the firmware tables (see below).

Whilst in theory, all the memory transactions issued by a PCI device
go through the IOMMU, on certain platforms some of the memory
transaction may
not reach the IOMMU because they are interpreted by the host bridge. For
instance, this could happen if the MSI doorbell is built into the PCI
bridge or for P2P traffic. See [6] for more details.

XXX: I think this could be solved by using direct mapping (e.g GFN ==
this would mean the guest memory layout would be similar to the host
one when
PCI devices will be pass-throughed => Detail it.
In the example given in the IORT spec, for pci devices not behind an SMMU,
how would the writes from the device be protected.

I realize the XXX paragraph is quite confusing. I am not trying to solve the problem where PCI devices are not protected behind an SMMU but platform where some transactions (e.g P2P or MSI doorbell access) are by-passing the SMMU.

You may still want to allow PCI passthrough in that case, because you know that P2P cannot be done (or potentially disabled) and MSI doorbell access is protected (for instance a write in the ITS doorbell will be tagged with the device by the hardware). In order to support such platform you need to direct map the doorbel (e.g GFN == MFN) and carve out the P2P region from the guest memory map. Hence the suggestion to re-use the host memory layout for the guest.

Note that it does not mean the RAM region will be direct mapped. It is only there to ease carving out memory region by-passed by the SMMU.



### Host bridges

The static table MCFG (see 4.2 in [1]) will describe the host bridges
at boot and supporting ECAM. Unfortunately, there are platforms out there
(see [2]) that re-use MCFG to describe host bridge that are not fully

This means that Xen needs to account for possible quirks in the host
The Linux community are working on a patch series for this, see [2]
and [3],
where quirks will be detected with:
     * OEM ID
     * OEM Table ID
     * OEM Revision
     * PCI Segment
     * PCI bus number range (wildcard allowed)

Based on what Linux is currently doing, there are two kind of quirks:
     * Accesses to the configuration space of certain sizes are not
     * A specific driver is necessary for driving the host bridge

The former is straightforward to solve but the latter will require
more thought.
Instantiation of a specific driver for the host controller can be
easily done
if Xen has the information to detect it.
So Xen would parse the MCFG to find a hb, then map the config space in
dom0 stage2 ?
and then provide the same MCFG to dom0?

This is implementation details. I have been really careful so far to leave the implementation open as it does not matter at this stage how we are going to implement it in Xen.


## Discovering and registering host bridge

The approach taken in the document will require communication between
Xen and
the hardware domain. In this case, they would need to agree on the
number associated to an host bridge. However, this number is not
available in
the Device Tree case.

The hardware domain will register new host bridges using the existing


struct physdev_pci_mmcfg_reserved {
     /* IN */
     uint64_t    address;
     uint16_t    segment;
     /* Range of bus supported by the host bridge */
     uint8_t     start_bus;
     uint8_t     end_bus;

     uint32_t    flags;
So this hypercall is not required for ACPI?

This is not DT specific as even on ACPI there are platform not fully ECAM compliant. As I said above, we will need to decide whether we want to support non-ECAM compliant host bridges (e.g all host bridges have a specific drivers) in Xen. Likely this will be on case by case basis.


## Discovering and registering PCI devices

The hardware domain will scan the host bridge to find the list of PCI
available and then report it to Xen using the existing hypercall

#define XEN_PCI_DEV_EXTFN   0x1
#define XEN_PCI_DEV_VIRTFN  0x2
#define XEN_PCI_DEV_PXM     0x3

struct physdev_pci_device_add {
     /* IN */
     uint16_t    seg;
     uint8_t     bus;
     uint8_t     devfn;
     uint32_t    flags;
     struct {
         uint8_t bus;
         uint8_t devfn;
     } physfn;
      * Optional parameters array.
      * First element ([0]) is PXM domain associated with the device (if
      * XEN_PCI_DEV_PXM is set)
     uint32_t optarr[0];
For mapping the MMIO space of the device in Stage2, we need to add
support in Xen / via a map hypercall in linux/drivers/xen/pci.c

Mapping MMIO space in stage-2 is not PCI specific and already addressed in Xen 4.9 (see commit 80f9c31 "xen/arm: acpi: Map MMIO on fault in stage-2 page table for the hardware domain"). So I don't understand why we should care about that here...


Julien Grall

Xen-devel mailing list



Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.