Re: [PATCH RFC v2] vPCI: account for hidden devices

[Stefano Stabellini <sstabellini@xxxxxxxxxx>]

On Thu, 25 May 2023, Stefano Stabellini wrote:
> On Thu, 25 May 2023, Jan Beulich wrote:
> > On 25.05.2023 01:37, Stefano Stabellini wrote:
> > > On Wed, 24 May 2023, Jan Beulich wrote:
> > >>>> RFC: _setup_hwdom_pci_devices()' loop may want splitting: For
> > >>>>      modify_bars() to consistently respect BARs of hidden devices while
> > >>>>      setting up "normal" ones (i.e. to avoid as much as possible the
> > >>>>      "continue" path introduced here), setting up of the former may 
> > >>>> want
> > >>>>      doing first.
> > >>>
> > >>> But BARs of hidden devices should be mapped into dom0 physmap?
> > >>
> > >> Yes.
> > > 
> > > The BARs would be mapped read-only (not read-write), right? Otherwise we
> > > let dom0 access devices that belong to Xen, which doesn't seem like a
> > > good idea.
> > > 
> > > But even if we map the BARs read-only, what is the benefit of mapping
> > > them to Dom0? If Dom0 loads a driver for it and the driver wants to
> > > initialize the device, the driver will crash because the MMIO region is
> > > read-only instead of read-write, right?
> > > 
> > > How does this device hiding work for dom0? How does dom0 know not to
> > > access a device that is present on the PCI bus but is used by Xen?
> > 
> > None of these are new questions - this has all been this way for PV Dom0,
> > and so far we've limped along quite okay. That's not to say that we
> > shouldn't improve things if we can, but that first requires ideas as to
> > how.
> 
> For PV, that was OK because PV requires extensive guest modifications
> anyway. We only run Linux and few BSDs as Dom0. So, making the interface
> cleaner and reducing guest changes is nice-to-have but not critical.
> 
> For PVH, this is different. One of the top reasons for AMD to work on
> PVH is to enable arbitrary non-Linux OSes as Dom0 (when paired with
> dom0less/hyperlaunch). It could be anything from Zephyr to a
> proprietary RTOS like VxWorks. Minimal guest changes for advanced
> features (e.g. Dom0 S3) might be OK but in general I think we should aim
> at (almost) zero guest changes. On ARM, it is already the case (with some
> non-upstream patches for dom0less PCI.)
> 
> For this specific patch, which is necessary to enable PVH on AMD x86 in
> gitlab-ci, we can do anything we want to make it move faster. But
> medium/long term I think we should try to make non-Xen-aware PVH Dom0
> possible.

Like I wrote, personally I am happy with whatever gets us to have the PVH
test in gitlab-ci faster.

However, on the specific problem of PCI devices used by Xen and how to
deal with them for Dom0 PVH, I think they should be completely hidden.
Hidden in the sense that they don't appear on the Dom0 PCI bus. If the
hidden device is a function of a multi-function PCI device, then the
entire multi-function PCI device should be hidden.

I don't think this case is very important because devices used by Xen
are timers, IOMMUs, UARTs, all devices that typically are not
multi-function, so it is OK to be extra careful and remove the entire
device from Dom0 in the odd case that the device is both multi-function
and only partially used by Xen. This is what I would do for Xen on ARM
too.