[Xen-devel] [PATCH] xen: turn numa=on by default

[Andre Przywara <andre.przywara@xxxxxxx>]

Hi,

attached a patch to turn numa=on by default.
I did some benchmark runs (lmbench & kernel compile) with a number of 
guests running in parallel to compare the performance of numa=on vs. 
numa=off.
As soon as one starts to load the machine, the performance goes down in
the numa=off case.
The tests were done on an 8-node machine (4 cores each).
lmbench (actually copying large amounts of memory) shows a dramatic
dropdown, but I even noticed significant performance decrease for a
tmpfs based Linux kernel compile. Here a summary of the data:

lmbench's rd benchmark (normalized to native Linux (=100)):
guests    numa=off       numa=on       avg increase
       min  avg  max  min  avg  max
    1       78.0           102.3
    7  37.4 45.6 62.0 90.6 102.3 110.9 124.4%
   15  21.0 25.8 31.7 41.7 48.7 54.1    88.2%
   23  13.4 17.5 23.2 25.0 28.0 30.1    60.2%

kernel compile in tmpfs, 1 VCPU, 2GB RAM, average of elapsed time:
guests    numa=off   numa=on   increase
     1    480.610    464.320    3.4%
     7    482.109    461.721    4.2%
    15    515.297    477.669    7.3%
    23    548.427    495.180    9.7%
again with 2 VCPUs and make -j2:
     1    264.580    261.690    1.1%
     7    279.763    258.907    7.7%
    15    330.385    272.762   17.4%
    23    463.510    390.547   15.7% (46 VCPUs on 32pCPUs)

Selected tests on a 4-node machine showed similar behavior (7.9 % 
increase with 6 parallel guests on the 2 VCPU kernel compile benchmark).
So I ask for turning numa=on by default.

Note that this does not affect non-NUMA machines at all, since NUMA will 
be turned off again by the code if no NUMA topology is detected.
Signed-off-by: Andre Przywara <andre.przywara@xxxxxxx>

Please apply!

Regards,
Andre.

--
Andre Przywara
AMD-Operating System Research Center (OSRC), Dresden, Germany
Tel: +49 351 488-3567-12
----to satisfy European Law for business letters:
Advanced Micro Devices GmbH
Karl-Hammerschmidt-Str. 34, 85609 Dornach b. Muenchen
Geschaeftsfuehrer: Jochen Polster; Thomas M. McCoy; Giuliano Meroni
Sitz: Dornach, Gemeinde Aschheim, Landkreis Muenchen
Registergericht Muenchen, HRB Nr. 43632

diff -r bdf4109edffb xen/arch/x86/numa.c
--- a/xen/arch/x86/numa.c       Mon Nov 30 16:13:26 2009 +0100
+++ b/xen/arch/x86/numa.c       Tue Dec 01 09:59:33 2009 +0100
@@ -41,8 +41,7 @@
 
 nodemask_t __read_mostly node_online_map = { { [0] = 1UL } };
 
-/* Default NUMA to off for now. acpi=on required to enable it. */
-int numa_off __devinitdata = 1;
+int numa_off __devinitdata = 0;
 
 int acpi_numa __devinitdata;
 

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel