Re: [Xen-devel] [DESIGN] Feature Levelling improvements

[Konrad Rzeszutek Wilk <konrad.wilk@xxxxxxxxxx>]

Thank you for posting this!

Some comments below.

> Design
> ======
> 
> `struct sysctl_physinfo.levelling_caps`
> ---------------------------------------
> 
> Xen shall gain a new physinfo field which reports the degree to which it can
> influence `CPUID` executed by a PV guest.  This is a bitmap containing:
> 
> * `faulting`
>     * CPUID Faulting is available, and full control can be exercised.
> * `mask_ecx`
>     * Leaf 0x00000001.ECX
> * `mask_edx`
>     * Leaf 0x00000001.EDX
> * `mask_extd_ecx`
>     * Leaf 0x80000001.ECX
> * `mask_extd_edx`
>     * Leaf 0x80000001.EDX
> * `mask_xsave_eax`
>     * Leaf 0x0000000D[ECX=1].EAX
> * `mask_therm_ecx`
>     * Leaf 0x00000006.ECX
> * `mask_l7s0_eax`
>     * Leaf 0x00000007[ECX=0].EAX
> * `mask_l7s0_ebx`

Those 'l' look like '1' in the PDF.

Can it be called something else?

>     * Leaf 0x00000007[ECX=0].EBX
> 
> At the time of writing, these are all the masking MSRs known by Xen.  The
> bitmap shall be extended as new MSRs become available.
> 
> New 'featureset' API for use by the toolstack
> ---------------------------------------------
> 
> A featureset is a defined as a collection of words covering the cpuid leaves
> which report features to the caller.  It is variable length, and expected to
> grow over time as processors gain more features, or Xen starts supporting
> exposing more features to guests.
> 
> At the time of writing, the leaves containing feature bits are:
> 
> * 0x00000001.ECX
> * 0x00000001.EDX
> * 0x80000001.ECX
> * 0x80000001.EDX
> * 0x0000000D[ECX=1].EAX
> * 0x00000007[ECX=0].EBX
> * 0x00000006.EAX
> * 0x00000006.ECX
> * 0x0000000A.EAX
> * 0x0000000A.EBX
> * 0x0000000F[ECX=0].EDX
> * 0x0000000F[ECX=1].EDX
> 
> XEN_SYSCTL_get_featureset
> -------------------------
> 
> Xen shall on boot create a featureset for itself, and the maximum available
> features for each type of guest, based on hardware features, command line
> options etc.  A toolstack shall be able to query all of these.

maximum available features? As in two sets of features - one for
PV and another for HVM. The PV being a subset of HVM (since it is more
constrained)?

Command line options being the same old ones (the cpuid_mask..?) and then
more? Or just rewrite this to be:

cpuid=mask_therm_ecx=[blahbla],mask_xsave_eax=[blahbal] ?


> 
> Cpuid feature-verification library
> ----------------------------------
> 
> There shall be a new library (shared between Xen and libxc in the same
> way as
> libelf etc.) which can verify the a featureset.  In particular, it will

s/ a //
> confirm that no features are enabled without their dependent features.

And presumarily can compare these features and do a and-subset (or an
or-subset) ?

> 
> XEN_DOMCTL_set_cpuid
> --------------------
> 
> This is an existing hypercall.  Currently it just stashes the policy from
> userspace.  It shall be extended to provide verification of the policy, and
> reject attempts to advertise features which Xen is incapable of providing
> (via hardware or emulation support).

Where would be the code to trim the 'maximum available features' in the
subsets (like PV) with some cpuid=X flags from user-space?


> 
> VCPU context switch
> -------------------
> 
> Xen shall be updated to lazily context switch all available masking
> MSRs.  It
> is noted that this shall incur a performance overhead if restricted
> featuresets are assigned to PV guests, and _CPUID Faulting_ is not
> available.
> 
> It shall be the responsibility of the host administrator to avoid creating
> such a scenario, if the performance overhead is a concern.

.. and perhaps add warnings in the toolstack to tell the admin?

> 
> 
> Future work
> ===========
> 
> The above is a minimum quantity of work to support feature levelling, but
> further problems exist.  They are acknowledged as being issues, but are
> not in
> scope for fixing as part of feature levelling.
> 
> * Xen has no notion of per-cpu and per-package data in the cpuid policy.  In
>   particular, this causes issues for VMs attempting to detect topology,
> which
>   find inconsistent/incorrect cache information.
> 
> * In the case that `domain_cpuid()` can't locate a leaf in the topology, it
>   will fall back to issuing a plain `CPUID` instruction.  This breaks VM
>   encapsulation, as a VM which has migrated can observe differences which
>   should be hidden.
> 
> * There is currently a positioning issue with the domains cpuid policy.
>   Verifying the register state requires the policy, but the policy is behind
>   the register state in the migration stream.  The domains cpuid policy
> should
>   become an item in Xen's migration state for a VM.


And potentially code in libxl to allow subset manipulation to allow
leveling across different platforms. As in the common features would
be exposed while all the other ones are masked? And I suppose some
format to stash this so it can be ingested by the libxl tools?


> 

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel