Re: [Xen-devel] RE: Ballooning up

[Jeremy Fitzhardinge <jeremy@xxxxxxxx>]

 On 09/15/2010 11:06 AM, Dan Magenheimer wrote:
>> From: Jeremy Fitzhardinge [mailto:jeremy@xxxxxxxx]
>> Sent: Wednesday, September 15, 2010 11:29 AM
>> To: Ian Campbell
>> Cc: Dan Magenheimer; Stefano Stabellini; Xen-devel@xxxxxxxxxxxxxxxxxxx;
>> Daniel Kiper; Konrad Wilk
>> Subject: Re: [Xen-devel] RE: Ballooning up
>>
>>  On 09/15/2010 12:10 AM, Ian Campbell wrote:
>>>> Indeed.  I think adding general 32x limit between base and max size
>> will
>>>> prevent a completely unusable system, and then just suggest using
>> mem=
>>>> to control that more precisely (esp for dom0).
>>> Sounds reasonable.
>> I found 32x doesn't work; there seems to be a lot more per-page
>> overhead
>> than I expected.  I made the limit 10x, which I determined empirically
>> and somewhat arbitrarily, but it does seem reasonable.
> Any idea what amount/percent of memory is "wasted" with this limit?
> (e.g. assuming a system with 10GB physical memory and dom0_mem=1G
> and no up-ballooning)

Not sure precisely.  It probably depends quite a bit on your kernel
config (32 vs 64 bit, the various sparsemem options, among other things).

> So if one knows a priori that dom0 will not be ballooned up
> above dom0_mem, one specifies dom0_mem= on the xen boot line
> and mem= on the dom0 "module" line?

Yes, but if dom0_mem is more than about 3G its probably worth setting
mem to dom0_mem+1G.

> IIRC the Linux mem=1G option doesn't really limit physical
> memory to 1G, just specifies the highest legal address, ignoring
> holes.  Dunno if dom0_mem has this problem (on xenolinux)
> but I think it does not.
>

It does effectively, because the kernel will free any Xen-provided
memory which lies in the PCI hole (and any other holes in the E820
map).  It will then add that freed memory to the extra memory space so
you can balloon it back in again - but if you use "mem=" with the same
limit as the dom0_mem then it will truncate that region so you can't use
it (and those pages are lost to dom0, but usable by other domains).

The xenolinux kernel treated the machine physical memory address space
and the pseudophysical address space as being two completely distinct
ones; as a result it was quite happy to have RAM at the same address as
PCI devices.  This gets very messy in modern (-ish, ie, post about
2.6.20 or so) kernels because they maintain a unified resource tree
which tracks device mappings and RAM in the same structure.

To avoid this mess, the pvops dom0 kernels treat them as being in the
same address space, so it punches holes out of the RAM where devices
want to live.  However, the Xen domain builder still maps memory from
0-dom0_mem linearly, and the kernel needs to free up any memory which
overlaps with devices (=E820 holes).

This means that its generally impossible to give dom0 between 3G-4G of
memory, as the memory in the holes is always freed.  If you want to give
the domain 4G of accessible RAM then you need to set dom0_mem to 4G,
mem=5G and balloon in all freed pages.

    J

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel