Re: [PATCH] x86/x2apic: introduce a mixed physical/cluster mode

[Roger Pau Monné <roger.pau@xxxxxxxxxx>]

On Mon, Oct 30, 2023 at 05:34:48PM +0100, Jan Beulich wrote:
> On 30.10.2023 17:10, Roger Pau Monné wrote:
> > On Mon, Oct 30, 2023 at 03:32:56PM +0100, Jan Beulich wrote:
> >> On 24.10.2023 15:51, Roger Pau Monne wrote:
> >>> The current implementation of x2APIC requires to either use Cluster 
> >>> Logical or
> >>> Physical mode for all interrupts.  However the selection of Physical vs 
> >>> Logical
> >>> is not done at APIC setup, an APIC can be addressed both in Physical or 
> >>> Logical
> >>> destination modes concurrently.
> >>>
> >>> Introduce a new x2APIC mode called mixed, which uses Logical Cluster mode 
> >>> for
> >>> IPIs, and Physical mode for external interrupts, thus attempting to use 
> >>> the
> >>> best method for each interrupt type.
> >>>
> >>> Using Physical mode for external interrupts allows more vectors to be 
> >>> used, and
> >>> interrupt balancing to be more accurate.
> >>>
> >>> Using Logical Cluster mode for IPIs allows less accesses to the ICR 
> >>> register
> >>> when sending those, as multiple CPUs can be targeted with a single ICR 
> >>> register
> >>> write.
> >>>
> >>> A simple test calling flush_tlb_all() 10000 times in a tight loop on a 96 
> >>> CPU
> >>> box gives the following average figures:
> >>>
> >>> Physical mode: 26617931ns
> >>> Mixed mode:    23865337ns
> >>>
> >>> So ~10% improvement versus plain Physical mode.
> >>
> >> Nice.
> >>
> >>>  Note that Xen uses Cluster
> >>> mode by default, and hence is already using the fastest way for IPI 
> >>> delivery at
> >>> the cost of reducing the amount of vectors available system-wide.
> >>>
> >>> Make the newly introduced mode the default one.
> >>>
> >>> Suggested-by: Andrew Cooper <andrew.cooper3@xxxxxxxxxx>
> >>> Signed-off-by: Roger Pau Monné <roger.pau@xxxxxxxxxx>
> >>> ---
> >>> Do we want to keep a full Logical Cluster mode available?  I don't see a 
> >>> reason
> >>> to target external interrupts in Logical Cluster mode, but maybe there's
> >>> something I'm missing.
> >>>
> >>> It's not clear to me whether the ACPI FADT flags are meant to apply only 
> >>> to
> >>> external interrupt delivery mode, or also to IPI delivery.  If
> >>> ACPI_FADT_APIC_PHYSICAL is only meant to apply to external interrupts and 
> >>> not
> >>> IPIs then we could possibly get rid of physical mode IPI delivery.
> >>>
> >>> Still need to put this under XenServer extensive testing, but wanted to 
> >>> get
> >>> some feedback before in case approach greatly changes.
> >>
> >> Looks quite okay, just a couple of minor remarks:
> > 
> > Thanks.
> > 
> > Do we still want to keep the pure cluster mode?
> 
> I think we should keep it for a full release cycle or two.

OK, will make the requested changes and send v2 then.

Thanks, Roger.