Re: [Xen-devel] [Query] Flow of PCI device dedicated to a domU

[Stefano Stabellini <stefano.stabellini@xxxxxxxxxxxxx>]

Please don't use HTML emails and please don't top post.

On Tue, 22 Jul 2014, Xenbegn developer wrote:
> Ian Can you please help in this. I am trying to understand how things are 
> done in x86 xen. Based on that will
> implement in ARM PCI passthrough

The basic mechanism is that Xen assigns the PCI root complex/host
bridge to Dom0. Dom0 walks the bus and finds out the devices available.

When the user assigns a PCI device to a VM, the toolstack (xl/libxl)
issues hypercalls to remap interrupts and MMIO regions of that device
into the guest VM.
The device is added to the guest virtual PCI bus that is created by the
xen-pcifront driver at boot. PCI config space reads/writes go through
xen-pcifront that talks to xen-pciback in Dom0.

PCI bus enumeration is done by Dom0 at boot time. Dom0 issues hypercalls
to Xen to tell the hypervisor what devices are available. It would be
conceivable to do the enumeration directly in Xen if Xen had the
appropriate driver for the host bridge. It might be even better.

The main interaction between Xen and the config space of PCI devices is
about MSI programming. Xen programs MSIs directly because it is Xen that
is going to receive them and then inject them into guests.


If I were you I would base my work on Julien SMMU series and I would
ignore MSIs for now. I would try to get pcifront and pciback running and
then assign a PCI device to a guest, only using legacy interrupts.

After that you could try to add support for doing PCI config space reads
and writes in Xen. Then introduce MSI support but, as Ian wrote, you
would need to rebase your work on the GICv3 series. I don't think you
actually need full ITS support to be able to get MSI running though.
You might already have everything you need in terms of patches, but they
are still not applied to xen-unstable yet.



 
> On Tue, Jul 22, 2014 at 4:21 PM, Xenbegn developer <xen.begn.dev@xxxxxxxxx> 
> wrote:
> 
> 
> 
>       On Tue, Jul 22, 2014 at 4:03 PM, Andrew Cooper 
> <andrew.cooper3@xxxxxxxxxx> wrote:
>             On 22/07/14 11:13, Xenbegn developer wrote:
> 
> 
> 
>       On Tue, Jul 22, 2014 at 3:26 PM, Andrew Cooper 
> <andrew.cooper3@xxxxxxxxxx> wrote:
>             On 22/07/14 10:50, Xenbegn developer wrote:
> 
> 
> 
>       On Wed, Jul 16, 2014 at 3:20 PM, Andrew Cooper 
> <andrew.cooper3@xxxxxxxxxx> wrote:
>             On 16/07/14 07:12, Xenbegn developer wrote:
>       Hi,
> I am trying to understand the flow of how a PCI device driver in domU works
> after a PCI device is is assigned to a domU.
> 
> a) If a PCI device is assigned to a domU, this device has to be on a PCI
> bus. So as per my view xen would have to somehow provide a PCI Controller
> on which this device is attached.
> => Is my assumption correct ? If yes how it is done, No then also How
> enumeration of this device happens in domU kernel
> 
> 
> No.
> 
> PV guests have no PCI root ports/bridges; they use devices as single entities
> knowing that Xen/dom0 takes care of the other bits. HVM guests have their PCI
> devices attached to the virtual southbridge which is all emulated by Qemu.
> 
> 
> b) Is the Configuration space of the PCI device directly accessible to the
> domU (assuming the kernel accesses it using memory map) ? If not then it is
> trapped by xen
> 
> 
> All configuration space is trap+emulate in Xen, although almost all operations
> permitted.
> 
> 
> c) Who assigns the MSI (addr + value) in the PCI device. If Xen then how
> Xen does a translation from Physical MSI to guest MSI (where in code)
> 
> 
> Xen controls all interrupts on the system, which is why it needs to trap all
> config accesses to notice when a domain is attempting to change the interrupt
> information. In that case, Xen fixes up its delivery of interrupts to the 
> guest,
> but leaves the underlying interrupt information intact.
> 
> ~Andrew
> 
> 
> Linux kernel pci code has a function called __write_msi_msg. This method 
> writes to
> config space for MSI/MSIX
> pci_write_config_dword(dev, pos + PCI_MSI_ADDRESS_LO,
> ÂÂÂ ÂÂÂ ÂÂÂ ÂÂÂ ÂÂÂÂÂÂ msg->address_lo);
> 
> Now as per (b) and (c) it has to be trapped in xen. But in x86/traps.c 
> (guest_io_write)
> I dont see any MSI handling.
> Can you please explain (point to the code) where domain is attempting to 
> change the
> interrupt information.
> ÂÂÂ
> 
> 
> PV guests are expected to use PHYSOP hypercalls for interrupt management. 
> Xen does not
> provide transparent emulation support for them.
> 
> HVM guests do get transparent emulation support, as from the point of view of 
> an HVM guest,
> they are talking to a real PCI device.
> 
> Can you please point to a device driver code which is calling PHYSOP calls 
> for setting an MSI Addr
> + MSI number
> 
> 
> No. Partly because I don't know exactly, and mainly because this is 
> something you should trivially be
> able to search for.
> 
> It is not that trivial. Can anoyone from Citrix reply ?
>       ~Andrew
> 
> 
> 
> 
>

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel