[Xen-devel] Memory reservation for 32bit guests on high RAM systems

[Peter Sandin <psandin@xxxxxxxxxx>]

We've been looking in to support of 32bit domains on machines with large 
amounts of RAM. Working from the ballooning and reservation support added a few 
years ago:

http://lists.xen.org/archives/html/xen-devel/2009-11/msg01277.html

32 bit domains eventually fail to boot with the following Xen complaint:

xm create: Error: (4, 'Out of memory', "panic: xc_dom_boot.c:159: 
xc_dom_boot_mem_init: can't allocate low memory for domain")

Booting with dom0_mem=128G, with enable-dom0-ballooning=yes and 
total_available_memory=128 set in xend's configuration works as expected.  
128GB of RAM is avalable 32bit guests and 128GB for 64 bit guests.  I'm able to 
account for the RAM in the box by totaling 32bit domains + 64bit domains + 
free_memory as reported by xm info + memory remaining in the dom0 balloon.

However this config doesn't allow use of all of the RAM available in the 
physical machine, and increasing dom0_mem beyond 128G decreases the amount of 
32bit domains I am able to get to boot!  It appears that for every 1G over 128G 
for dom0_mem reduces the ability to boot 32bit domains by 512M.

The balloon appears to be starting too low in RAM. The output of "xm debug-key 
h" indicates that a 'D' will dump the p2m table which sounds like it would be 
really useful in determining where the various domains are actually mapped in 
physical RAM, however "xm debug-key D" doesn't produce any output on the 
console or xm dmesg.

Settings and RAM-worth of different bitness guests that were bootable on a box 
with 384G of RAM:

dom0_mem         total_available_memory    32bit     64bit
min:3G,max:128G           128               125G      127G
-128G                     128                65G      128G
min:3G,max:-128G          240                65G      239G
min:3G,max:160G           150               115G      149G

We're either missing something fundamental about how to configure this, or 
there's a bug in ballooning or reservation that is preventing us from using the 
full 128G towards 32bit domains while being able to use all of the remaining 
RAM for 64 bit guests.

Anyone have any insight on using ballooning and memory reservations in machines 
with more than 256GB of RAM? Alternaively any information on getting a dump of 
p2m map or otherwise getting a clear picture of where in physical RAM each 
domain resides would be helpful in troubleshooting this further.

Regards,
Peter Sandin
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel