RE: [XEN RFC PATCH 37/40] xen: introduce an arch helper to do NUMA init failed fallback

[Wei Chen <Wei.Chen@xxxxxxx>]

Hi Jan,

> -----Original Message-----
> From: Jan Beulich <jbeulich@xxxxxxxx>
> Sent: 2021年8月30日 17:52
> To: Wei Chen <Wei.Chen@xxxxxxx>
> Cc: Bertrand Marquis <Bertrand.Marquis@xxxxxxx>; Julien Grall
> <julien@xxxxxxx>; xen-devel@xxxxxxxxxxxxxxxxxxxx; sstabellini@xxxxxxxxxx
> Subject: Re: [XEN RFC PATCH 37/40] xen: introduce an arch helper to do
> NUMA init failed fallback
> 
> On 28.08.2021 05:45, Wei Chen wrote:
> >> From: Xen-devel <xen-devel-bounces@xxxxxxxxxxxxxxxxxxxx> On Behalf Of
> Wei
> >> Chen
> >> Sent: 2021年8月28日 11:09
> >>
> >>> From: Julien Grall <julien@xxxxxxx>
> >>> Sent: 2021年8月27日 22:30
> >>>
> >>>> --- a/xen/arch/arm/numa.c
> >>>> +++ b/xen/arch/arm/numa.c
> >>>> @@ -140,3 +140,16 @@ int __init arch_meminfo_get_ram_bank_range(int
> >> bank,
> >>>>
> >>>>          return 0;
> >>>>   }
> >>>> +
> >>>> +void __init arch_numa_init_failed_fallback(void)
> >>>> +{
> >>>> +    int i, j;
> >>>> +
> >>>> +    /* Reset all node distance to remote_distance */
> >>>> +    for ( i = 0; i < MAX_NUMNODES; i++ ) {
> >>>> +        for ( j = 0; j < MAX_NUMNODES; j++ ) {
> >>>> +            numa_set_distance(i, j,
> >>>> +                (i == j) ? NUMA_LOCAL_DISTANCE :
> >> NUMA_REMOTE_DISTANCE);
> >>>> +        }
> >>>> +    }
> >>>> +}
> >>>
> >>> ... this implementation looks fairly generic. So can you explain why
> we
> >>> need it on Arm but not x86?
> >>>
> >>
> >> This implementation is DT only, for x86, it's using acpi_slit.
> >> For now, I am not quit sure ACPI need to do fallback or not.
> >> Or say in another way, I don't know how to implement the fallback
> >> for ACPI. I planned to solve it in Arm ACPI version NUMA, so I left
> >> an empty helper for x86.
> >>
> >> @Jan Beulich Could you give me some suggestion about x86 fallback?
> >>
> >>
> >
> > I have a quick look into Linux. When Arch do numa init failed,
> > the numa_free_distance will be invoked to revert numa_distance.
> 
> Does this matter in the first place? Don't we fall back to single
> node mode, in which case the sole entry of the distance table
> will say "local" anyway?
> 

Thank you for providing another way of thinking. Yes, once NUMA init
is failed, the system will fall back to single node mode. If we call
__node_distance normally, we will not pass two different nodes to
this function. Even if we don't revert the values in distance table,
we will not trigger the condition of "node_a != node_b". We will
always get "local" from __node_distance.

But for closed-loop of code, I tend to revert data when initialization
is failed.

Cheers,
Wei Chen

> Jan