Re: [Xen-devel] [PATCH 2/2] x86/pvshim: do not enable global pages in shim mode

[Roger Pau Monné <roger.pau@xxxxxxxxxx>]

On Fri, Nov 29, 2019 at 12:12:51PM +0000, Andrew Cooper wrote:
> On 29/11/2019 12:09, Jan Beulich wrote:
> > On 25.11.2019 18:22, Roger Pau Monne wrote:
> >> When using global pages a full tlb flush can only be performed by
> >> toggling the PGE bit in CR4, which is usually quite expensive in terms
> >> of performance when running virtualized. This is specially relevant on
> >> AMD hardware, which doesn't have the ability to do selective CR4
> >> trapping, but can also be relevant on Intel if the underlying
> >> hypervisor also traps on accesses to the PGE CR4 bit.
> >>
> >> In order to avoid this performance penalty, do not use global pages
> >> when running in shim mode. Note this is done when running on both
> >> Intel or AMD hardware, since older versions of Xen capable of running
> >> the shim don't make use of Intel selective CR4 trapping feature and
> >> will vmexit on every access to CR4.
> > So here you say you do this uniformly because of older Xen.
> > What about newer Xen? Is this still a win (or at least not a
> > loss) there? Independent of underlying hardware?

So on Intel hw that don't trap CR4 PGE accesses disabling PGE seems to
make performance slightly worse when doing a `make -j8 xen` on a 8 vCPU
pv-shim guest:

CR4 PGE enabled:
real    2m40.889s
real    2m41.700s
real    2m40.453s

CR4 PGE disabled:
real    2m43.197s
real    2m41.940s
real    2m42.727s

> > In case of
> > any kind of doubt I think this would want to be command line
> > controllable.
> 
> Older Xen has VMExits for all CR4.PGE flips.
> 
> Newer Xen (since 4.10? iirc) on Intel hardware (with HAP) arranged for
> CR4.PGE flips not to vmexit.
> 
> There is no ability to cause CR4.PGE flips to not vmexit on AMD, other
> than to give the guest full control of CR4 which is a BadThing(tm).
> 
> I agree that this wants a command line control, but it wants to be
> enabled by default any time we find ourselves nested on AMD hardware,
> not just in shim.

Only on AMD hardware? Newer versions of Xen don't trap CR4 PGE writes,
but what about other hypervisors?

I think it would be better to avoid using PGE when the hypervisor
CPUID bit is set, regardless of whether the hardware is AMD or not.
The performance penalty doesn't seem that bad, taking into account
that using PGE when CR4 is trapped is much worse. Alternatively we
could try to detect how slow a flush from CR4 is and act accordingly,
but that seems tricky.

I can add a command line option to force or prevent the usage of PGE.

Thanks, Roger.

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel