Re: [RFC PATCH v1 1/4] arm/pci: PCI setup and PCI host bridge discovery within XEN on ARM.

[Stefano Stabellini <sstabellini@xxxxxxxxxx>]

On Tue, 28 Jul 2020, Roger Pau Monné wrote:
> On Mon, Jul 27, 2020 at 05:06:25PM -0700, Stefano Stabellini wrote:
> > On Mon, 27 Jul 2020, Roger Pau Monné wrote:
> > > On Sat, Jul 25, 2020 at 10:59:50AM +0100, Julien Grall wrote:
> > > > On Sat, 25 Jul 2020 at 00:46, Stefano Stabellini 
> > > > <sstabellini@xxxxxxxxxx> wrote:
> > > > >
> > > > > On Fri, 24 Jul 2020, Julien Grall wrote:
> > > > > > On Fri, 24 Jul 2020 at 19:32, Stefano Stabellini 
> > > > > > <sstabellini@xxxxxxxxxx> wrote:
> > > > > > > > If they are not equal, then I fail to see why it would be 
> > > > > > > > useful to have this
> > > > > > > > value in Xen.
> > > > > > >
> > > > > > > I think that's because the domain is actually more convenient to 
> > > > > > > use
> > > > > > > because a segment can span multiple PCI host bridges. So my
> > > > > > > understanding is that a segment alone is not sufficient to 
> > > > > > > identify a
> > > > > > > host bridge. From a software implementation point of view it 
> > > > > > > would be
> > > > > > > better to use domains.
> > > > > >
> > > > > > AFAICT, this would be a matter of one check vs two checks in Xen :).
> > > > > > But... looking at Linux, they will also use domain == segment for 
> > > > > > ACPI
> > > > > > (see [1]). So, I think, they still have to use (domain, bus) to do 
> > > > > > the lookup.
> > > 
> > > You have to use the (segment, bus) tuple when doing a lookup because
> > > MMCFG regions on ACPI are defined for a segment and a bus range, you
> > > can have a MMCFG region that covers segment 0 bus [0, 20) and another
> > > MMCFG region that covers segment 0 bus [20, 255], and those will use
> > > different addresses in the MMIO space.
> > 
> > Thanks for the clarification!
> > 
> > 
> > > > > > > > In which case, we need to use PHYSDEVOP_pci_mmcfg_reserved so
> > > > > > > > Dom0 and Xen can synchronize on the segment number.
> > > > > > >
> > > > > > > I was hoping we could write down the assumption somewhere that 
> > > > > > > for the
> > > > > > > cases we care about domain == segment, and error out if it is not 
> > > > > > > the
> > > > > > > case.
> > > > > >
> > > > > > Given that we have only the domain in hand, how would you enforce 
> > > > > > that?
> > > > > >
> > > > > > >From this discussion, it also looks like there is a mismatch 
> > > > > > >between the
> > > > > > implementation and the understanding on QEMU devel. So I am a bit
> > > > > > concerned that this is not stable and may change in future Linux 
> > > > > > version.
> > > > > >
> > > > > > IOW, we are know tying Xen to Linux. So could we implement
> > > > > > PHYSDEVOP_pci_mmcfg_reserved *or* introduce a new property that
> > > > > > really represent the segment?
> > > > >
> > > > > I don't think we are tying Xen to Linux. Rob has already said that
> > > > > linux,pci-domain is basically a generic device tree property.
> > > > 
> > > > My concern is not so much the name of the property, but the definition 
> > > > of it.
> > > > 
> > > > AFAICT, from this thread there can be two interpretation:
> > > >       - domain == segment
> > > >       - domain == (segment, bus)
> > > 
> > > I think domain is just an alias for segment, the difference seems to
> > > be that when using DT all bridges get a different segment (or domain)
> > > number, and thus you will always end up starting numbering at bus 0
> > > for each bridge?
> > >
> > > Ideally you would need a way to specify the segment and start/end bus
> > > numbers of each bridge, if not you cannot match what ACPI does. Albeit
> > > it might be fine as long as the OS and Xen agree on the segments and
> > > bus numbers that belong to each bridge (and thus each ECAM region).
> > 
> > That is what I thought and it is why I was asking to clarify the naming
> > and/or writing a document to explain the assumptions, if any.
> > 
> > Then after Julien's email I followed up in the Linux codebase and
> > clearly there is a different assumption baked in the Linux kernel for
> > architectures that have CONFIG_PCI_DOMAINS enabled (including ARM64).
> > 
> > The assumption is that segment == domain == unique host bridge. It
> > looks like it is coming from IEEE Std 1275-1994 but I am not certain.
> > In fact, it seems that ACPI MCFG and IEEE Std 1275-1994 don't exactly
> > match. So I am starting to think that domain == segment for IEEE Std
> > 1275-1994 compliant device tree based systems.
> 
> I don't think the ACPI MCFG spec contains the notion of bridges, it
> just describes ECAM (or MMCFG) regions, but those could be made up by
> concatenating different bridge ECAM regions by the firmware itself, so
> you could AFAICT end up with multiple bridges being aggregated into a
> single ECAM region, and thus using the same segment number, which
> seems not possible with the DT spec, where each bridge must get a
> different segment number?

Yes, that's my understanding too


> If you could assign both a segment number and a bus start and end
> values to a bridge then I think it would be kind of equivalent to ACPI
> MCFG.
> 
> I assume we would never support a system where Xen is getting the
> hardware description from a DT and the hardware domain is using ACPI
> (or the other way around)?

Yeah, I think it is a good assumption


> If so, I don't think we care that enumeration when using DT is
> different than when using ACPI, as we can only guarantee consistency
> when both Xen and the hardware domain use the same source for the
> hardware description.
> 
> If when using DT each bridge has a unique segment number that's fine
> as long as Xen and the OS agree to not change such values.

I agree