Re: [Xen-devel] [Xen-users] xend (12, 'Cannot allocate memory')

[Keir Fraser <keir.fraser@xxxxxxxxxxxxx>]

Yes. So, if you are happy for dom0 to always run with a static amount of
memory then:

Modify xend-config.sxp to have:
(dom0-min-mem 0)

Modify /boot/grub/menu.lst to include dom0_mem=256M (for example) on the Xen
command line.

You'll now have all the memory reported as free in 'xm info' to play with,
and xend will not screw you around.

 -- Keir

On 26/3/08 16:16, "Kai Meyer" <kai@xxxxxxxxxxxxx> wrote:

> Just to reiterate so I understand:
> You think that xend is not 'balooning' Dom-0's memory correctly, and
> your suggestion is to make modifications to both xend-config.sxp and the
> grub.conf files to statically set Dom-0's memory allocation?
> 
> I've heard people talk about grub and options you can set for the
> hypervisor in the grub.conf file. Do you have some documentation for me
> to read?
> 
> For the xend-config.sxp file, I have already set:
> (dom0-min-mem 256)
> Dom-0 could survive on 256 or 512 MB of memory. I would probably set it
> to 512 if I'm going to statically set it for now.
> 
> Thanks for the info, it's been days since I posted to the Xen-users
> list, and never got a response.
> 
> -Kai Meyer
> 
> Keir Fraser wrote:
>> Depending on your usage scenario for dom0 itself, you could could turn off
>> auto-ballooning entirely (it can be done by modifying xend-config.sxp) and
>> limit dom0's initial memory allocation on Xen's command line (e.g.,
>> modifying Xen boot parameters in your GRUB menu file).
>> 
>>  -- Keir
>> 
>> On 26/3/08 15:25, "Kai Meyer" <kai@xxxxxxxxxxxxx> wrote:
>> 
>>   
>>> Well, that's no so great. It thinks I have barely a fraction of what I'm
>>> supposed to have available. Free shows 10GB available, xm info shows 1
>>> MB available. If I stop a virtual machine, the memory in xminfo
>>> increases by the amount of memory used by that xen machine, which is to
>>> be expected.
>>> 
>>> Is there a way to force Xen to refresh that number, or let me force an
>>> override?
>>> 
>>> -Kai Meyer
>>> 
>>> Keir Fraser wrote:
>>>     
>>>> Given you are running dom0 with plenty of swap, it's most likely that Xen
>>>> has run out of memory to allocate to your new domU, rather than the
>>>> allocation failure coming from within dom0. Probably auto-ballooning is
>>>> screwing up somehow (that's the bit of xend that gives away memory back to
>>>> Xen to make sure it has enough memory to create your new domU).
>>>> 
>>>> Sorry I don't have a precise diagnosis for you, but I think looking at
>>>> Xen's
>>>> allocator is at least the right direction for you to take. 'xm info' tells
>>>> you how much memory Xen thinks it has.
>>>> 
>>>>  -- Keir
>>>> 
>>>> On 26/3/08 14:54, "Kai Meyer" <kai@xxxxxxxxxxxxx> wrote:
>>>> 
>>>>   
>>>>       
>>>>> I'm getting the error:
>>>>> xend (12, 'Cannot allocate memory')
>>>>>  From the command line after the Dom-0 goes through a period of heavy
>>>>> memory usage. I don't have any hard numbers on how to duplicate it, but
>>>>> I'll give you the steps I've gone through to duplicate the problem.
>>>>> 
>>>>> First, some information about my setup. It's a new setup, and has been
>>>>> in heavy development for the past few months. I'm new to Xen as of this
>>>>> project we started in January. As far as I can tell, this problem has
>>>>> existed since we started, and may have even happened before but we were
>>>>> too new to think it wasn't our fault (which it may still be....)  I've
>>>>> found the same error addressed here:
>>>>> http://archive.netbsd.se/?ml=xen-devel&a=2007-04&t=3887921
>>>>> The 'hack' to fix this problem is already built into my
>>>>> XendDomainInfo.py script already, so I'm a little more than lost.
>>>>> 
>>>>> [root@xen1 ~]# xm info
>>>>> host                   : xen1.fiber.net
>>>>> release                : 2.6.18-53.1.13.el5xen
>>>>> version                : #1 SMP Tue Feb 12 13:33:07 EST 2008
>>>>> machine                : x86_64
>>>>> nr_cpus                : 8
>>>>> nr_nodes               : 1
>>>>> sockets_per_node       : 2
>>>>> cores_per_socket       : 4
>>>>> threads_per_core       : 1
>>>>> cpu_mhz                : 1866
>>>>> hw_caps                :
>>>>> bfebfbff:20100800:00000000:00000140:0004e3bd:00000000:00000001
>>>>> total_memory           : 16382
>>>>> free_memory            : 2
>>>>> xen_major              : 3
>>>>> xen_minor              : 1
>>>>> xen_extra              : .0-53.1.13.el5
>>>>> xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32
>>>>> hvm-3.0-x86_32p hvm-3.0-x86_64
>>>>> xen_pagesize           : 4096
>>>>> platform_params        : virt_start=0xffff800000000000
>>>>> xen_changeset          : unavailable
>>>>> cc_compiler            : gcc version 4.1.2 20070626 (Red Hat 4.1.2-14)
>>>>> cc_compile_by          : mockbuild
>>>>> cc_compile_domain      :
>>>>> cc_compile_date        : Tue Feb 12 12:55:35 EST 2008
>>>>> xend_config_format     : 2
>>>>> 
>>>>> 
>>>>> Here's the error in action:
>>>>> [root@xen1 centos-5.1_core_image]# xm create centos-5.1_core_image
>>>>> Using config file "./centos-5.1_core_image".
>>>>> Error: (12, 'Cannot allocate memory')
>>>>> [root@xen1 centos-5.1_core_image]#
>>>>> 
>>>>> 
>>>>> Here is the output from xend.log that occurs when I get this error:
>>>>> [2008-03-19 14:07:43 xend.XendDomainInfo 3069] DEBUG
>>>>> (XendDomainInfo:200) XendDomainInfo.create(['vm', ['name',
>>>>> 'centos-5.1_core_image'], ['memory', 4024], ['maxmem', 4024],
>>>>> ['on_poweroff', 'destroy'], ['on_reboot', 'restart'], ['on_crash',
>>>>> 'restart'], ['vcpus', 6], ['image', ['hvm', ['kernel',
>>>>> '/usr/lib/xen/boot/hvmloader'], ['device_model',
>>>>> '/usr/lib64/xen/bin/qemu-dm'], ['pae', 1], ['vcpus', 6], ['boot',
>>>>> 'cda'], ['serial', 'pty'], ['vnc', 1], ['vncunused', 1], ['xauthority',
>>>>> '/root/.Xauthority'], ['keymap', 'en-us']]], ['device', ['vbd',
>>>>> ['uname', 'file:/xen/domains2/centos-5.1_core_image/new_disk.img'],
>>>>> ['dev', 'hda'], ['mode', 'w']]], ['device', ['vbd', ['uname',
>>>>> 'file:/xen/domains2/centos-5.1_core_image/swap.img'], ['dev', 'hdb'],
>>>>> ['mode', 'w']]], ['device', ['vif', ['bridge', 'xenbr1'], ['mac',
>>>>> '00:16:3e:4c:c0:01'], ['type', 'ioemu']]]])
>>>>> [2008-03-19 14:07:43 xend.XendDomainInfo 3069] DEBUG
>>>>> (XendDomainInfo:306) parseConfig: config is ['vm', ['name',
>>>>> 'centos-5.1_core_image'], ['memory', 4024], ['maxmem', 4024],
>>>>> ['on_poweroff', 'destroy'], ['on_reboot', 'restart'], ['on_crash',
>>>>> 'restart'], ['vcpus', 6], ['image', ['hvm', ['kernel',
>>>>> '/usr/lib/xen/boot/hvmloader'], ['device_model',
>>>>> '/usr/lib64/xen/bin/qemu-dm'], ['pae', 1], ['vcpus', 6], ['boot',
>>>>> 'cda'], ['serial', 'pty'], ['vnc', 1], ['vncunused', 1], ['xauthority',
>>>>> '/root/.Xauthority'], ['keymap', 'en-us']]], ['device', ['vbd',
>>>>> ['uname', 'file:/xen/domains2/centos-5.1_core_image/new_disk.img'],
>>>>> ['dev', 'hda'], ['mode', 'w']]], ['device', ['vbd', ['uname',
>>>>> 'file:/xen/domains2/centos-5.1_core_image/swap.img'], ['dev', 'hdb'],
>>>>> ['mode', 'w']]], ['device', ['vif', ['bridge', 'xenbr1'], ['mac',
>>>>> '00:16:3e:4c:c0:01'], ['type', 'ioemu']]]]
>>>>> [2008-03-19 14:07:43 xend.XendDomainInfo 3069] DEBUG
>>>>> (XendDomainInfo:411) parseConfig: result is {'shadow_memory': None,
>>>>> 'start_time': None, 'uuid': None, 'on_crash': 'restart', 'on_reboot':
>>>>> 'restart', 'localtime': None, 'image': ['hvm', ['kernel',
>>>>> '/usr/lib/xen/boot/hvmloader'], ['device_model',
>>>>> '/usr/lib64/xen/bin/qemu-dm'], ['pae', 1], ['vcpus', 6], ['boot',
>>>>> 'cda'], ['serial', 'pty'], ['vnc', 1], ['vncunused', 1], ['xauthority',
>>>>> '/root/.Xauthority'], ['keymap', 'en-us']], 'on_poweroff': 'destroy',
>>>>> 'bootloader_args': None, 'cpus': None, 'name': 'centos-5.1_core_image',
>>>>> 'backend': [], 'vcpus': 6, 'cpu_weight': None, 'features': None,
>>>>> 'vcpu_avail': None, 'memory': 4024, 'device': [('vbd', ['vbd', ['uname',
>>>>> 'file:/xen/domains2/centos-5.1_core_image/new_disk.img'], ['dev',
>>>>> 'hda'], ['mode', 'w']]), ('vbd', ['vbd', ['uname',
>>>>> 'file:/xen/domains2/centos-5.1_core_image/swap.img'], ['dev', 'hdb'],
>>>>> ['mode', 'w']]), ('vif', ['vif', ['bridge', 'xenbr1'], ['mac',
>>>>> '00:16:3e:4c:c0:01'], ['type', 'ioemu']])], 'bootloader': None, 'cpu':
>>>>> None, 'maxmem': 4024}
>>>>> [2008-03-19 14:07:43 xend.XendDomainInfo 3069] DEBUG
>>>>> (XendDomainInfo:1349) XendDomainInfo.construct: None
>>>>> [2008-03-19 14:07:43 xend 3069] DEBUG (balloon:127) Balloon: 2276 KiB
>>>>> free; need 2048; done.
>>>>> [2008-03-19 14:07:43 xend.XendDomainInfo 3069] ERROR
>>>>> (XendDomainInfo:212) Domain construction failed
>>>>> Traceback (most recent call last):
>>>>>  File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py",
>>>>> line 204, in create
>>>>>    vm.construct()
>>>>>  File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py",
>>>>> line 1369, in construct
>>>>>    hvm = hvm)
>>>>> Error: (12, 'Cannot allocate memory')
>>>>> 
>>>>> 
>>>>> Here's the configuration file being used:
>>>>> name = "centos-5.1_core_image"
>>>>> maxmem = 4024
>>>>> memory = 4024
>>>>> vcpus = 6
>>>>> builder = "hvm"
>>>>> kernel = "/usr/lib/xen/boot/hvmloader"
>>>>> boot = "cda"
>>>>> pae = 1
>>>>> acpi = 0
>>>>> apic = 0
>>>>> on_poweroff = "destroy"
>>>>> on_reboot = "restart"
>>>>> on_crash = "restart"
>>>>> device_model = "/usr/lib64/xen/bin/qemu-dm"
>>>>> sdl = 0
>>>>> vnc = 1
>>>>> vncunused = 1
>>>>> keymap = "en-us"
>>>>> disk = [ 'file:/xen/domains2/centos-5.1_core_image/disk.img,hda,w' ,
>>>>> 'file:/xen/domains2/centos-5.1_core_image/swap.img,hdb,w' ]
>>>>> vif = [ "mac=00:16:3e:4c:c0:01,bridge=xenbr1,type=ioemu" ]
>>>>> serial = "pty"
>>>>> 
>>>>> At the time the error occured, my memory looked like this:
>>>>> [root@xen1 centos-5.1_core_image]# free
>>>>>             total       used       free     shared    buffers     cached
>>>>> Mem:      15845376   15821080      24296          0     113596   15081468
>>>>> -/+ buffers/cache:     626016   15219360
>>>>> Swap:      4192956          4    4192952
>>>>> 
>>>>> The only way I've been able to duplicate this error is by making
>>>>> multiple copies of the 6GB disk images (created with dd, so ls shows 6GB
>>>>> but du shows 2GB) for different virtual machines and booting them up. I
>>>>> can get up to about 8 new virtual machines roughly, before I start
>>>>> seeing my memory error. If you're familiar with IBM's 'xen_deploy.pl'
>>>>> script, I've modified it slightly to meet our needs, and launch our
>>>>> systems with that tool. It's basic functionality is to read in a xen
>>>>> configuration file you want to use as a template, and output a modified
>>>>> one based on arguments you pass to the scripts, and then create a floppy
>>>>> disk image and copy the image from the template config file, and then
>>>>> boot the new system.
>>>>> 
>>>>> I've read in previous emails that this error occurs because of a bug in
>>>>> the hvmloader, and look for an update. That email response was for a 3.0
>>>>> version of xen, and I'm on 3.1.
>>>>> 
>>>>> What other information can I provide to help diagnose this problem?
>>>>> 
>>>>> -Kai Meyer
>>>>> 
>>>>> _______________________________________________
>>>>> Xen-devel mailing list
>>>>> Xen-devel@xxxxxxxxxxxxxxxxxxx
>>>>> http://lists.xensource.com/xen-devel
>>>>>     
>>>>>         
>>>> 
>>>> _______________________________________________
>>>> Xen-devel mailing list
>>>> Xen-devel@xxxxxxxxxxxxxxxxxxx
>>>> http://lists.xensource.com/xen-devel
>>>>   
>>>>       
>>> _______________________________________________
>>> Xen-devel mailing list
>>> Xen-devel@xxxxxxxxxxxxxxxxxxx
>>> http://lists.xensource.com/xen-devel
>>>     
>> 
>>   
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@xxxxxxxxxxxxxxxxxxx
> http://lists.xensource.com/xen-devel



_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel