Re: [Xen-users] query memory allocation per NUMA node

[Dario Faggioli <dario.faggioli@xxxxxxxxxx>]

On Mon, 2017-01-16 at 13:18 +0100, Eike Waldt wrote:
> On 01/12/2017 01:45 AM, Dario Faggioli wrote:
> > On Mon, 2017-01-09 at 15:47 +0100, Eike Waldt wrote:
> > > Only doing soft-pinning is way worse for the overall performance,
> > > as
> > > hard-pinning (according to my first tests).
> > > 
> > Can you elaborate on this? I'm curious (what tests, what does the
> > numbers look like in the two cases, etc).
> > 
> setup:
> - 144 vCPUs on a server with 4 NUMA Nodes
> - pinning Dom0 CPUs (0-15)
> - running 60 DomUs (40 Linux (para), 20 Windows (HVM))
> - doing 2/3 CPU load with stressaptest(CPU,RAM) and one fio(write
> I/O)
> thread in all linux VMs
> 
Ok. You didn't say how many vCPUs each VM has. I'm assuming 1?

Also, how are you "pinning Dom0 CPUs", and why?

> soft-pinning whole NUMA nodes per DomU (depending on NUMA Node memory
> placement):
> The load on Dom0 is about 200,
> the i/o wait is about 30 and
> the cpu steal time for each vCPU in Dom0 is about 50!
> Dom0 and DomUs respond very slow.
> 
> hard-pinning whole NUMA nodes per DomU (depending on NUMA Node memory
> placement):
> The load on Dom0 is about 90,
> the i/o wait is about 30 and
> the cpu steal time is about 2!
> Dom0 and DomUs respond ok.
> 
Mmm..  If possible, I'd like to see the output of the following
commands, with all the domains created (it's not important that they
run a benchmark, they just need to be live.

# xl info -n
# xl list -n
# xl vcpu-list
# xl debug-key u ; xl dmesg

And this is for both the configuration you say you've tried above.

> This simple test tells me, that soft-pinning is way worse than hard-
> pinning.
>
That may well be. But it sounds strange. I'd be inclined to think that
there is something else going on.. Or maybe I'm just not understanding
what you mean with "pinning while NUMA nodes per DomU" (and that's why
I'm asking for the commands output :-)).

> It may be a corner case though and nobody might ever tested it in
> this
> "dimension" ;)
> 
Actually, we tested it even for higher "dimensions"! But true, corner
cases will always exist. :-)

Regards,
Dario
-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

Attachment: signature.asc
Description: This is a digitally signed message part

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
https://lists.xen.org/xen-users