Re: [Xen-devel] arm64: Approach for DT based NUMA and issues

[Dario Faggioli <dario.faggioli@xxxxxxxxxx>]

On Sun, 2016-11-27 at 12:23 +0000, Julien Grall wrote:
> Hi Dario,
> 
Hi,

> On 27/11/2016 01:01, Dario Faggioli wrote:
> > On Sat, 2016-11-26 at 12:29 +0530, Vijay Kilari wrote:
> > I agree that we need to support vNUMA for Dom0 sooner rather than
> > later, and I agree that Dom0 is a bit special, so some tricks may
> > be
> > necessary. But until we don't implement vNUMA for Dom0, Dom0 is
> > just a
> > non-NUMA virtual machine, and the kernel running inside that should
> > just behave like it behaves on a non-NUMA box.
> > 
> > Again, I don't know much about ARM, but I think that, until we
> > don't
> > have vNUMA for Dom0, that devm_zalloc() thing will just see 1 and
> > only
> > 1 NUMA node from which to allocate memory.
> 
> I would rather divide the NUMA work for ARM in 2 distinct tasks:
>       - Make Xen NUMA-aware
>       - Make DOM0 NUMA-aware
> 
That makes perfect sense to me, and FWIW, is also what I'd do. In fact,
the whole point of what I was saying was not to confuse Xen NUMA
support and Dom0 NUMA support; if we want to do both of them, the
latter right after the former, fine, but they're separate things
indeed.

> Vijay, if I understood correctly what Dario said, on x86 DOM0 is not
> yet 
> NUMA-aware. 
>
You did. It is not.

> > > Ex: SMMU driver of device on node 1 tries to allocate memory
> > > on node 1.
> > > 
> > > ISSUE:
> > >  - Dom0's memory should be split across all the available memory
> > > nodes
> > >    of the system and memory nodes should be generated
> > > accordingly.
> > > 
> > ...This is the default behavior, at least on x86.
> 
> Are you speaking about the command line parameter dom0_nodes?
> 
Not exactly. As said, Dom0 is not NUMA aware and does not have any
virtual NUMA layout.

This means that, by default, Dom0 memory is indeed spread among various
existing nodes. Eg., on my NUMA test box here at home, here's how
things are for Dom0:

(XEN) [  970.100116] NODE0 start->1720320 size->1572864 free->0
(XEN) [  970.100122] NODE1 start->0 size->1720320 free->460155
(XEN) [  970.100130] CPU0...7 -> NODE0
(XEN) [  970.100136] CPU8...15 -> NODE1
(XEN) [  970.100140] Memory location of each domain:
(XEN) [  970.100149] Domain 0 (total: 258512):
(XEN) [  970.102268]     Node 0: 159254
(XEN) [  970.102273]     Node 1: 99258

dom0_nodes=x is a way to tell Xen to (try as hard as it can) to only
allocate the memory for dom0 only from NUMA node x but, even if more
than one node is specified, that does not include giving to him a
virtual NUMA topology, nor making it aware of the underline NUMA
topology of the host in any way.

> > Generating DT nodes for Dom0 is exactly what I mean when I say
> > "implementing / enabling vNUMA for Dom0" (in this case on ARM).
> > 
> > So, yes, let's do it, but let's discuss how to do it properly
> > (e.g., if
> > there's anything that can be common between archs, such as some
> > bits of
> > the interface).
> 
> I would expect vNUMA for Dom0 to be common between x86 and ARM.
> 
As much as possible, indeed.

Regards,
Dario
-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

Attachment: signature.asc
Description: This is a digitally signed message part

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel