Re: [PATCH v4] x86/x2apic: introduce a mixed physical/cluster mode

[Jan Beulich <jbeulich@xxxxxxxx>]

On 06.11.2023 15:27, Roger Pau Monne wrote:
> The current implementation of x2APIC requires to either use Cluster Logical or
> Physical mode for all interrupts.  However the selection of Physical vs 
> Logical
> is not done at APIC setup, an APIC can be addressed both in Physical or 
> Logical
> destination modes concurrently.
> 
> Introduce a new x2APIC mode called Mixed, which uses Logical Cluster mode for
> IPIs, and Physical mode for external interrupts, thus attempting to use the
> best method for each interrupt type.
> 
> Using Physical mode for external interrupts allows more vectors to be used, 
> and
> interrupt balancing to be more accurate.
> 
> Using Logical Cluster mode for IPIs allows fewer accesses to the ICR register
> when sending those, as multiple CPUs can be targeted with a single ICR 
> register
> write.
> 
> A simple test calling flush_tlb_all() 10000 times on a tight loop on AMD EPYC
> 9754 with 512 CPUs gives the following figures in nano seconds:
> 
> x mixed
> + phys
> * cluster
>     N           Min           Max        Median           Avg        Stddev
> x  25 3.5131328e+08 3.5716441e+08 3.5410987e+08 3.5432659e+08     1566737.4
> +  12  1.231082e+09  1.238824e+09 1.2370528e+09 1.2357981e+09     2853892.9
> Difference at 95.0% confidence
>       8.81472e+08 +/- 1.46849e+06
>       248.774% +/- 0.96566%
>       (Student's t, pooled s = 2.05985e+06)
> *  11 3.5099276e+08 3.5561459e+08 3.5461234e+08 3.5415668e+08     1415071.9
> No difference proven at 95.0% confidence
> 
> So Mixed has no difference when compared to Cluster mode, and Physical mode is
> 248% slower when compared to either Mixed or Cluster modes with a 95%
> confidence.
> 
> Note that Xen uses Cluster mode by default, and hence is already using the
> fastest way for IPI delivery at the cost of reducing the amount of vectors
> available system-wide.
> 
> Make the newly introduced mode the default one.
> 
> Note the printing of the APIC addressing mode done in connect_bsp_APIC() has
> been removed, as with the newly introduced mixed mode this would require more
> fine grained printing, or else would be incorrect.  The addressing mode can
> already be derived from the APIC driver in use, which is printed by different
> helpers.
> 
> Suggested-by: Andrew Cooper <andrew.cooper3@xxxxxxxxxx>
> Signed-off-by: Roger Pau Monné <roger.pau@xxxxxxxxxx>

Reviewed-by: Jan Beulich <jbeulich@xxxxxxxx>