Re: [RFC PATCH V1 01/12] hvm/ioreq: Make x86's IOREQ feature common

[Stefano Stabellini <sstabellini@xxxxxxxxxx>]

On Thu, 6 Aug 2020, Julien Grall wrote:
> On 06/08/2020 01:37, Stefano Stabellini wrote:
> > On Wed, 5 Aug 2020, Julien Grall wrote:
> > > On 04/08/2020 20:11, Stefano Stabellini wrote:
> > > > On Tue, 4 Aug 2020, Julien Grall wrote:
> > > > > On 04/08/2020 12:10, Oleksandr wrote:
> > > > > > On 04.08.20 10:45, Paul Durrant wrote:
> > > > > > > > +static inline bool hvm_ioreq_needs_completion(const ioreq_t
> > > > > > > > *ioreq)
> > > > > > > > +{
> > > > > > > > +    return ioreq->state == STATE_IOREQ_READY &&
> > > > > > > > +           !ioreq->data_is_ptr &&
> > > > > > > > +           (ioreq->type != IOREQ_TYPE_PIO || ioreq->dir !=
> > > > > > > > IOREQ_WRITE);
> > > > > > > > +}
> > > > > > > I don't think having this in common code is correct. The short-cut
> > > > > > > of
> > > > > > > not
> > > > > > > completing PIO reads seems somewhat x86 specific.
> > > > > 
> > > > > Hmmm, looking at the code, I think it doesn't wait for PIO writes to
> > > > > complete
> > > > > (not read). Did I miss anything?
> > > > > 
> > > > > > Does ARM even
> > > > > > > have the concept of PIO?
> > > > > > 
> > > > > > I am not 100% sure here, but it seems that doesn't have.
> > > > > 
> > > > > Technically, the PIOs exist on Arm, however they are accessed the same
> > > > > way
> > > > > as
> > > > > MMIO and will have a dedicated area defined by the HW.
> > > > > 
> > > > > AFAICT, on Arm64, they are only used for PCI IO Bar.
> > > > > 
> > > > > Now the question is whether we want to expose them to the Device
> > > > > Emulator
> > > > > as
> > > > > PIO or MMIO access. From a generic PoV, a DM shouldn't have to care
> > > > > about
> > > > > the
> > > > > architecture used. It should just be able to request a given IOport
> > > > > region.
> > > > > 
> > > > > So it may make sense to differentiate them in the common ioreq code as
> > > > > well.
> > > > > 
> > > > > I had a quick look at QEMU and wasn't able to tell if PIOs and MMIOs
> > > > > address
> > > > > space are different on Arm as well. Paul, Stefano, do you know what
> > > > > they
> > > > > are
> > > > > doing?
> > > > 
> > > > On the QEMU side, it looks like PIO (address_space_io) is used in
> > > > connection with the emulation of the "in" or "out" instructions, see
> > > > ioport.c:cpu_inb for instance. Some parts of PCI on QEMU emulate PIO
> > > > space regardless of the architecture, such as
> > > > hw/pci/pci_bridge.c:pci_bridge_initfn.
> > > > 
> > > > However, because there is no "in" and "out" on ARM, I don't think
> > > > address_space_io can be accessed. Specifically, there is no equivalent
> > > > for target/i386/misc_helper.c:helper_inb on ARM.
> > > 
> > > So how PCI I/O BAR are accessed? Surely, they could be used on Arm, right?
> > 
> > PIO is also memory mapped on ARM and it seems to have its own MMIO
> > address window.
> This part is already well-understood :). However, this only tell us how an OS
> is accessing a PIO.
> 
> What I am trying to figure out is how the hardware (or QEMU) is meant to work.
> 
> From my understanding, the MMIO access will be received by the hostbridge and
> then forwarded to the appropriate PCI device. The two questions I am trying to
> answer is: How the I/O BARs are configured? Will it contain an MMIO address or
> an offset?
> 
> If the answer is the latter, then we will need PIO because a DM will never see
> the MMIO address (the hostbridge will be emulated in Xen).

Now I understand the question :-)

This is the way I understand it works. Let's say that the PIO aperture
is 0x1000-0x2000 which is aliased to 0x3eff0000-0x3eff1000.
0x1000-0x2000 are addresses that cannot be accessed directly.
0x3eff0000-0x3eff1000 is the range that works.

A PCI device PIO BAR will have an address in the 0x1000-0x2000 range,
for instance 0x1100.

However, when the operating system access 0x1100, it will issue a read
to 0x3eff0100.

Xen will trap the read to 0x3eff0100 and send it to QEMU.

QEMU has to know that 0x3eff0000-0x3eff1000 is the alias to the PIO
aperture and that 0x3eff0100 correspond to PCI device foobar. Similarly,
QEMU has also to know the address range of the MMIO aperture and its
remappings, if any (it is possible to have address remapping for MMIO
addresses too.)

I think today this information is "built-in" QEMU, not configurable. It
works fine because *I think* the PCI aperture is pretty much the same on
x86 boards, at least the one supported by QEMU for Xen.

On ARM, I think we should explicitly declare the PCI MMIO aperture and
its alias/address-remapping. When we do that, we can also declare the
PIO aperture and its alias/address-remapping.