Re: [PATCH 2/2] x86/cpuid: support LFENCE always serializing CPUID bit

[Roger Pau Monné <roger.pau@xxxxxxxxxx>]

On Wed, Apr 14, 2021 at 03:57:22PM +0200, Jan Beulich wrote:
> On 14.04.2021 15:25, Andrew Cooper wrote:
> > On 14/04/2021 14:05, Andrew Cooper wrote:
> >> On 14/04/2021 13:57, Jan Beulich wrote:
> >>> On 14.04.2021 13:04, Roger Pau Monne wrote:
> >>>> @@ -264,6 +265,38 @@ struct cpuid_policy
> >>>>              };
> >>>>              uint32_t nc:8, :4, apic_id_size:4, :16;
> >>>>              uint32_t /* d */:32;
> >>>> +
> >>>> +            uint64_t :64, :64; /* Leaf 0x80000009. */
> >>>> +            uint64_t :64, :64; /* Leaf 0x8000000a - SVM rev and 
> >>>> features. */
> >>>> +            uint64_t :64, :64; /* Leaf 0x8000000b. */
> >>>> +            uint64_t :64, :64; /* Leaf 0x8000000c. */
> >>>> +            uint64_t :64, :64; /* Leaf 0x8000000d. */
> >>>> +            uint64_t :64, :64; /* Leaf 0x8000000e. */
> >>>> +            uint64_t :64, :64; /* Leaf 0x8000000f. */
> >>>> +            uint64_t :64, :64; /* Leaf 0x80000010. */
> >>>> +            uint64_t :64, :64; /* Leaf 0x80000011. */
> >>>> +            uint64_t :64, :64; /* Leaf 0x80000012. */
> >>>> +            uint64_t :64, :64; /* Leaf 0x80000013. */
> >>>> +            uint64_t :64, :64; /* Leaf 0x80000014. */
> >>>> +            uint64_t :64, :64; /* Leaf 0x80000015. */
> >>>> +            uint64_t :64, :64; /* Leaf 0x80000016. */
> >>>> +            uint64_t :64, :64; /* Leaf 0x80000017. */
> >>>> +            uint64_t :64, :64; /* Leaf 0x80000018. */
> >>>> +            uint64_t :64, :64; /* Leaf 0x80000019 - TLB 1GB 
> >>>> Identifiers. */
> >>>> +            uint64_t :64, :64; /* Leaf 0x8000001a - Performance related 
> >>>> info. */
> >>>> +            uint64_t :64, :64; /* Leaf 0x8000001b - IBS feature 
> >>>> information. */
> >>>> +            uint64_t :64, :64; /* Leaf 0x8000001c. */
> >>>> +            uint64_t :64, :64; /* Leaf 0x8000001d - Cache properties. */
> >>>> +            uint64_t :64, :64; /* Leaf 0x8000001e - Extd APIC/Core/Node 
> >>>> IDs. */
> >>>> +            uint64_t :64, :64; /* Leaf 0x8000001f - AMD Secure 
> >>>> Encryption. */
> >>>> +            uint64_t :64, :64; /* Leaf 0x80000020 - Platform QoS. */
> >>>> +
> >>>> +            /* Leaf 0x80000021 - Extended Feature 2 */
> >>>> +            union {
> >>>> +                uint32_t e21a;
> >>>> +                struct { DECL_BITFIELD(e21a); };
> >>>> +            };
> >>>> +            uint32_t /* b */:32, /* c */:32, /* d */:32;
> >>>>          };
> >>>>      } extd;
> >>> Due to the effect of this on what guests get to see, I think this
> >>> wants to take my "x86/CPUID: shrink max_{,sub}leaf fields according
> >>> to actual leaf contents" as a prereq, which in turn may better
> >>> remain on top of "x86/CPUID: adjust extended leaves out of range
> >>> clearing" (both are neighbors in that over 4 months old series,
> >>> fair parts of which could imo go in irrespective of the unsettled
> >>> dispute on xvmalloc() - unfortunately I had made that patch 2 of
> >>> the series, not expecting it to be blocked for so long, and then
> >>> presumably signaling to others that the rest of the series is also
> >>> blocked).
> >> There is no shrinking to be done in this case.  The bit is set across
> >> the board on AMD/Hygon hardware, even on older parts.
> >>
> >> What does need changing however is the logic to trim max_extd_leaf down
> >> to what hardware supports, so the bit is visible on Rome/older
> >> hardware.  I.e. after this change, all VMs should get 0x80000021 by
> >> default on AMD hardware.
> >>
> >> (A curious observation of Milan hardware is that it actually advertises
> >> 0x80000023 as max_extd_leaf, and has two leaves of zeros at the end. 
> >> I've got an open query about this.)
> > 
> > Something like this:
> > 
> > diff --git a/xen/arch/x86/cpuid.c b/xen/arch/x86/cpuid.c
> > index 050cd5713e..d9eb2878c5 100644
> > --- a/xen/arch/x86/cpuid.c
> > +++ b/xen/arch/x86/cpuid.c
> > @@ -311,6 +311,7 @@ static void __init calculate_raw_policy(void)
> >  static void __init calculate_host_policy(void)
> >  {
> >      struct cpuid_policy *p = &host_cpuid_policy;
> > +    unsigned int max_extd_leaf;
> >  
> >      *p = raw_cpuid_policy;
> >  
> > @@ -318,7 +319,18 @@ static void __init calculate_host_policy(void)
> >          min_t(uint32_t, p->basic.max_leaf,   ARRAY_SIZE(p->basic.raw) - 1);
> >      p->feat.max_subleaf =
> >          min_t(uint32_t, p->feat.max_subleaf, ARRAY_SIZE(p->feat.raw) - 1);
> > -    p->extd.max_leaf = 0x80000000 | min_t(uint32_t, p->extd.max_leaf &
> > 0xffff,
> > +
> > +    max_extd_leaf = p->extd.max_leaf;
> > +
> > +    /*
> > +     * For AMD/Hygon hardware before Zen3, we modify LFENCE to be dispatch
> > +     * serialsing.  Extend max_extd_leaf beyond what hardware supports, to
> > +     * include the feature leaf containing this information.
> > +     */
> > +    if ( cpu_has_lfence_dispatch )
> > +        max_extd_leaf = max(max_extd_leaf, 0x80000021);
> > +
> > +    p->extd.max_leaf = 0x80000000 | min_t(uint32_t, max_extd_leaf & 0xffff,
> >                                            ARRAY_SIZE(p->extd.raw) - 1);
> >  
> >      cpuid_featureset_to_policy(boot_cpu_data.x86_capability, p);
> 
> Well, why not set it to ARRAY_SIZE() and then have
> x86_cpuid_policy_shrink_max_leaves() (from "x86/CPUID: shrink
> max_{,sub}leaf fields according to actual leaf contents") have
> a go? It'll recognize the non-zero leaf ... Otherwise, if we
> gain a few more such special cases, things are going to get
> ugly here.

I will wait for Jan to post the updated version of his shrink patch
and then rebase mine on top in order to set extd.max_leaf to
ARRAY_SIZE and let the shrink logic deal with it.

Thanks, Roger.