Re: [Xen-devel] [early RFC] ARM PCI Passthrough design document

[Roger Pau Monn? <roger.pau@xxxxxxxxxx>]

On Wed, Mar 15, 2017 at 10:11:35AM -0500, Venu Busireddy wrote:
> On Wed, Mar 15, 2017 at 12:56:50PM +0000, Roger Pau Monn? wrote:
> > On Wed, Mar 15, 2017 at 08:42:04AM -0400, Konrad Rzeszutek Wilk wrote:
> > > On Wed, Mar 15, 2017 at 12:07:28PM +0000, Roger Pau Monn? wrote:
> > > > On Fri, Mar 10, 2017 at 10:28:43AM -0500, Konrad Rzeszutek Wilk wrote:
> > > > > On Fri, Mar 10, 2017 at 12:23:18PM +0900, Roger Pau Monn? wrote:
> > > > > > On Thu, Mar 09, 2017 at 07:29:34PM -0500, Konrad Rzeszutek Wilk 
> > > > > > wrote:
> > > > > > > On Thu, Mar 09, 2017 at 01:26:45PM +0000, Julien Grall wrote:
> > > > > > > > Hi Konrad,
> > > > > > > > 
> > > > > > > > On 09/03/17 11:17, Konrad Rzeszutek Wilk wrote:
> > > > > > > > > On Thu, Mar 09, 2017 at 11:59:51AM +0900, Roger Pau Monn? 
> > > > > > > > > wrote:
> > > > > > > > > > On Wed, Mar 08, 2017 at 02:12:09PM -0500, Konrad Rzeszutek 
> > > > > > > > > > Wilk wrote:
> > > > > > > > > > > On Wed, Mar 08, 2017 at 07:06:23PM +0000, Julien Grall 
> > > > > > > > > > > wrote:
> > > > > > > > > > > .. this as for SR-IOV devices you need the drivers to 
> > > > > > > > > > > kick the hardware
> > > > > > > > > > > to generate the new bus addresses. And those (along with 
> > > > > > > > > > > the BAR regions) are
> > > > > > > > > > > not visible in ACPI (they are constructued dynamically).
> > > > > > > > > > 
> > > > > > > > > > There's already code in Xen [0] to find out the size of the 
> > > > > > > > > > BARs of SR-IOV
> > > > > > > > > > devices, but I'm not sure what's the intended usage of 
> > > > > > > > > > that, does it need to
> > > > > > > > > > happen _after_ the driver in Dom0 has done whatever magic 
> > > > > > > > > > for this to work?
> > > > > > > > > 
> > > > > > > > > Yes. This is called via the PHYSDEVOP_pci_device_add 
> > > > > > > > > hypercall when
> > > > > > > > > the device driver in dom0 has finished "creating" the VF. See 
> > > > > > > > > drivers/xen/pci.c
> > > > > > > > 
> > > > > > > > We are thinking to not use PHYSDEVOP_pci_device_add hypercall 
> > > > > > > > for ARM and do
> > > > > > > > the PCI scanning in Xen.
> > > > > > > > 
> > > > > > > > If I understand correctly what you said, only the PCI driver 
> > > > > > > > will be able to
> > > > > > > > kick SR-IOV device and Xen would not be able to detect the 
> > > > > > > > device until it
> > > > > > > > has been fully configured. So it would mean that we have to keep
> > > > > > > > PHYSDEVOP_pci_device_add around to know when Xen can use the 
> > > > > > > > device.
> > > > > > > > 
> > > > > > > > Am I correct?
> > > > > > > 
> > > > > > > Yes. Unless the PCI drivers come up with some other way to tell 
> > > > > > > the
> > > > > > > OS that oh, hey, there is this new PCI device with this BDF.
> > > > > > > 
> > > > > > > Or the underlaying bus on ARM can send some 'new device' 
> > > > > > > information?
> > > > > > 
> > > > > > Hm, is this something standard between all the SR-IOV 
> > > > > > implementations, or each
> > > > > > vendors have their own sauce?
> > > > > 
> > > > > Gosh, all of them have their own sauce. The only thing that is the 
> > > > > same
> > > > > is that suddenly behind the PF device there are PCI devies that are 
> > > > > responding
> > > > > to 0xcfc requests. MAgic!
> > > > 
> > > > I'm reading the PCI SR-IOV 1.1 spec, and I think we don't need to wait 
> > > > for the
> > > > device driver in Dom0 in order to get the information of the VF 
> > > > devices, what
> > > > Xen cares about is the position of the BARs (so that they can be mapped 
> > > > into
> > > > Dom0 at boot), and the PCI SBDF of each PF/VF, so that Xen can trap 
> > > > accesses to
> > > > it.
> > > > 
> > > > AFAICT both of this can be obtained without any driver-specific code, 
> > > > since
> > > > it's all contained in the PCI SR-IOV spec (but maybe I'm missing 
> > > > something).
> > > 
> > > CC-ing Venu,
> > > 
> > > Roger, could you point out which of the chapters has this?
> > 
> > This would be chapter 2 ("Initialization and Resource Allocation"), and then
> > there's a "IMPLEMENTATION NOTE" that shows how the PF/VF are matched to
> > function numbers in page 45 (I have the following copy, which is the latest
> > revision: "Single Root I/O Virtualization and Sharing Specification Revision
> > 1.1" from January 20 2010 [0]).
> > 
> > The document is quite complex, but it is a standard that all SR-IOV devices
> > should follow so AFAICT Xen should be able to get all the information that 
> > it
> > needs from the PCI config space in order to detect the PF/VF BARs and the 
> > BDF
> > device addresses.
> > 
> > Roger.
> > 
> > [0] https://members.pcisig.com/wg/PCI-SIG/document/download/8238
> 
> I do not have access to this document, so I have to rely on Rev 1.0
> document, but I don't think this aspect of the spec changed much.
> 
> In any case, I am afraid I am not seeing the overall picture, but I
> would like to comment on the last part of this discussion. Indeed, the
> configuration space (including the SR-IOV extended capability) contains
> all the information, but only the information necessary for the OS to
> "enumerate" the device (PF as well as VFs). The bus and device number
> (SBDF) assignment, and programming of the BARs, are all done during that
> enumeration. In this discussion, which entity is doing the enumeration?
> Xen, or Dom0?

Xen needs to let Dom0 manage the device, but at the same time it needs to
correctly map the device BARs into Dom0 physmap. I think the easiest solution
is to let Dom0 manage the device, and Xen should setup a trap to detect Dom0
setting the VF Enable bit (bit 0 in SR-IOV Control (08h)), at which point Xen
will size the VF BARs (and map them into Dom0) and also enumerate the VF
devices.

Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel