Re: [Xen-devel] [Bug] Intel RMRR support with upstream Qemu

[Alexey G <x1917x@xxxxxxxxx>]

On Mon, 24 Jul 2017 21:39:08 +0100
Igor Druzhinin <igor.druzhinin@xxxxxxxxxx> wrote:
> > But, the problem is that overall MMIO hole(s) requirements are not known
> > exactly at the time the HVM domain being created. Some PCI devices will
> > be emulated, some will be merely passed through and yet there will be
> > some RMRR ranges. libxl can't know all this stuff - some comes from the
> > host, some comes from DM. So actual MMIO requirements are known to
> > hvmloader at the PCI bus enumeration time.
> >   
> 
> IMO hvmloader shouldn't really be allowed to relocate memory under any
> conditions. As Andrew said it's much easier to provision the hole
> statically in libxl during domain construction process and it doesn't
> really compromise any functionality. Having one more entity responsible
> for guest memory layout only makes things more convoluted.

If moving most tasks of hvmloader to libxl is a planned feature in Citrix,
please let it be discussed on xen-devel first as it may affect many
people... and not all of them might be happy. :)

(tons of IMO and TLDR ahead, be warned)

Moving PCI BAR allocation from guest side to libxl is a controversial step.
This may be the architecturally wrong way in fact. There are properties and
areas of responsibility. Among primary responsibilities of guest's firmware
is PCI BARs and MMIO hole size allocation. That's a guest's territory.
Guest relocates PCI BARs (and not just BIOS able to do this), guest
firmware relocates MMIO hole base for them. If it was a real system, all
tasks like PCI BAR allocation, remapping part of RAM above 4G etc were done
by system BIOS. In our case some of SeaBIOS/OVMF responsibilities were
offloaded to hvmloader, like PCI BARs allocation, sizing MMIO hole(s) for
them and generating ACPI tables. And that's ok as hvmloader can be
considered merely a 'supplemental' firmware to perform some tasks of
SeaBIOS/OVMF before passing control to them. This solution has some
architecture logic at least and doesn't look bad.

On other hand, moving PCI hole calculation to libxl just to let Xen/libxl
know what the MMIO size value is might be a bad idea.
Aside from some code duplication, straying too far from the real hw paths,
or breaking existing (or future) interfaces this might have some other
negative consequences. Ex. who will be initializing guest's ACPI tables if
only libxl will know the memory layout? Some new interfaces between libxl
and hvmloader just to let the latter know what values to write to ACPI
tables being created? Or libxl will be initializing guest's ACPI tables as
well (another guest's internal task)? Similar concerns are applicable to
guest's final E820 construction.

Another thing is that handling ioreq/PT MMIO ranges is somewhat a property
of the device model (at least for now). Right now it's QEMU who traps PCI
BAR accesses and tells Xen how to handle specific ranges of MMIO space. If
QEMU already talks to Xen which ranges should be passed through or trapped
-- it can tell him the current overall MMIO limits as well... or handle
these limits himself -- if the MMIO hole range check is all what required to
avoid MMIO space misusing, this check can be easily implemented in QEMU,
provided that QEMU knows what memory/MMIO layout is. There is a lot of
implementation freedom where to place restrictions and checks, Xen or QEMU.
Strictly speaking, the MMIO hole itself can be considered a property of the
emulated machine and may have implementation differences for different
emulated chipsets. For example, the real i440' NB do not have an idea of
high MMIO hole at all.

We have already a sort of an interface between hvmloader and QEMU --
hvmloader has to do basic initialization for some emulated chipset's
registers (and this depends on the machine). Providing additional handling
for few other registers (TOM/TOLUD/etc) will cost almost nothing and
purpose of this registers will actually match their usage in real HW. This
way we can use an existing available interface and don't stray too far from
the real HW ways. 

I want to try this approach for Q35 bringup patches for Xen I'm currently
working on. I'll send these patches as RFC and will be glad to receive some
constructive criticism.

> > libxl can be taught to retrieve all missing info from QEMU, but this way
> > will require to perform all grunt work of PCI BARs allocation in libxl
> > itself - in order to calculate the real MMIO hole(s) size, one needs to
> > take into account all PCI BARs sizes and their alignment requirements
> > diversity + existing gaps due to RMRR ranges... basically, libxl will
> > need to do most of hvmloader/pci.c's job.
> >   
> 
> The algorithm implemented in hvmloader for that is not complicated and
> can be moved to libxl easily. What we can do is to provision a hole big
> enough to include all the initially assigned PCI devices. We can also
> account for emulated MMIO regions if necessary. But, to be honest, it
> doesn't really matter since if there is no enough space in lower MMIO
> hole for some BARs - they can be easily relocated to upper MMIO
> hole by hvmloader or the guest itself (dynamically).

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel