Re: [Xen-devel] [Xen-users] xend (12, 'Cannot allocate memory')

[Keir Fraser <keir.fraser@xxxxxxxxxxxxx>]

Depending on your usage scenario for dom0 itself, you could could turn off
auto-ballooning entirely (it can be done by modifying xend-config.sxp) and
limit dom0's initial memory allocation on Xen's command line (e.g.,
modifying Xen boot parameters in your GRUB menu file).

 -- Keir

On 26/3/08 15:25, "Kai Meyer" <kai@xxxxxxxxxxxxx> wrote:

> 
> Well, that's no so great. It thinks I have barely a fraction of what I'm
> supposed to have available. Free shows 10GB available, xm info shows 1
> MB available. If I stop a virtual machine, the memory in xminfo
> increases by the amount of memory used by that xen machine, which is to
> be expected.
> 
> Is there a way to force Xen to refresh that number, or let me force an
> override?
> 
> -Kai Meyer
> 
> Keir Fraser wrote:
>> Given you are running dom0 with plenty of swap, it's most likely that Xen
>> has run out of memory to allocate to your new domU, rather than the
>> allocation failure coming from within dom0. Probably auto-ballooning is
>> screwing up somehow (that's the bit of xend that gives away memory back to
>> Xen to make sure it has enough memory to create your new domU).
>> 
>> Sorry I don't have a precise diagnosis for you, but I think looking at Xen's
>> allocator is at least the right direction for you to take. 'xm info' tells
>> you how much memory Xen thinks it has.
>> 
>>  -- Keir
>> 
>> On 26/3/08 14:54, "Kai Meyer" <kai@xxxxxxxxxxxxx> wrote:
>> 
>>   
>>> I'm getting the error:
>>> xend (12, 'Cannot allocate memory')
>>>  From the command line after the Dom-0 goes through a period of heavy
>>> memory usage. I don't have any hard numbers on how to duplicate it, but
>>> I'll give you the steps I've gone through to duplicate the problem.
>>> 
>>> First, some information about my setup. It's a new setup, and has been
>>> in heavy development for the past few months. I'm new to Xen as of this
>>> project we started in January. As far as I can tell, this problem has
>>> existed since we started, and may have even happened before but we were
>>> too new to think it wasn't our fault (which it may still be....)  I've
>>> found the same error addressed here:
>>> http://archive.netbsd.se/?ml=xen-devel&a=2007-04&t=3887921
>>> The 'hack' to fix this problem is already built into my
>>> XendDomainInfo.py script already, so I'm a little more than lost.
>>> 
>>> [root@xen1 ~]# xm info
>>> host                   : xen1.fiber.net
>>> release                : 2.6.18-53.1.13.el5xen
>>> version                : #1 SMP Tue Feb 12 13:33:07 EST 2008
>>> machine                : x86_64
>>> nr_cpus                : 8
>>> nr_nodes               : 1
>>> sockets_per_node       : 2
>>> cores_per_socket       : 4
>>> threads_per_core       : 1
>>> cpu_mhz                : 1866
>>> hw_caps                :
>>> bfebfbff:20100800:00000000:00000140:0004e3bd:00000000:00000001
>>> total_memory           : 16382
>>> free_memory            : 2
>>> xen_major              : 3
>>> xen_minor              : 1
>>> xen_extra              : .0-53.1.13.el5
>>> xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32
>>> hvm-3.0-x86_32p hvm-3.0-x86_64
>>> xen_pagesize           : 4096
>>> platform_params        : virt_start=0xffff800000000000
>>> xen_changeset          : unavailable
>>> cc_compiler            : gcc version 4.1.2 20070626 (Red Hat 4.1.2-14)
>>> cc_compile_by          : mockbuild
>>> cc_compile_domain      :
>>> cc_compile_date        : Tue Feb 12 12:55:35 EST 2008
>>> xend_config_format     : 2
>>> 
>>> 
>>> Here's the error in action:
>>> [root@xen1 centos-5.1_core_image]# xm create centos-5.1_core_image
>>> Using config file "./centos-5.1_core_image".
>>> Error: (12, 'Cannot allocate memory')
>>> [root@xen1 centos-5.1_core_image]#
>>> 
>>> 
>>> Here is the output from xend.log that occurs when I get this error:
>>> [2008-03-19 14:07:43 xend.XendDomainInfo 3069] DEBUG
>>> (XendDomainInfo:200) XendDomainInfo.create(['vm', ['name',
>>> 'centos-5.1_core_image'], ['memory', 4024], ['maxmem', 4024],
>>> ['on_poweroff', 'destroy'], ['on_reboot', 'restart'], ['on_crash',
>>> 'restart'], ['vcpus', 6], ['image', ['hvm', ['kernel',
>>> '/usr/lib/xen/boot/hvmloader'], ['device_model',
>>> '/usr/lib64/xen/bin/qemu-dm'], ['pae', 1], ['vcpus', 6], ['boot',
>>> 'cda'], ['serial', 'pty'], ['vnc', 1], ['vncunused', 1], ['xauthority',
>>> '/root/.Xauthority'], ['keymap', 'en-us']]], ['device', ['vbd',
>>> ['uname', 'file:/xen/domains2/centos-5.1_core_image/new_disk.img'],
>>> ['dev', 'hda'], ['mode', 'w']]], ['device', ['vbd', ['uname',
>>> 'file:/xen/domains2/centos-5.1_core_image/swap.img'], ['dev', 'hdb'],
>>> ['mode', 'w']]], ['device', ['vif', ['bridge', 'xenbr1'], ['mac',
>>> '00:16:3e:4c:c0:01'], ['type', 'ioemu']]]])
>>> [2008-03-19 14:07:43 xend.XendDomainInfo 3069] DEBUG
>>> (XendDomainInfo:306) parseConfig: config is ['vm', ['name',
>>> 'centos-5.1_core_image'], ['memory', 4024], ['maxmem', 4024],
>>> ['on_poweroff', 'destroy'], ['on_reboot', 'restart'], ['on_crash',
>>> 'restart'], ['vcpus', 6], ['image', ['hvm', ['kernel',
>>> '/usr/lib/xen/boot/hvmloader'], ['device_model',
>>> '/usr/lib64/xen/bin/qemu-dm'], ['pae', 1], ['vcpus', 6], ['boot',
>>> 'cda'], ['serial', 'pty'], ['vnc', 1], ['vncunused', 1], ['xauthority',
>>> '/root/.Xauthority'], ['keymap', 'en-us']]], ['device', ['vbd',
>>> ['uname', 'file:/xen/domains2/centos-5.1_core_image/new_disk.img'],
>>> ['dev', 'hda'], ['mode', 'w']]], ['device', ['vbd', ['uname',
>>> 'file:/xen/domains2/centos-5.1_core_image/swap.img'], ['dev', 'hdb'],
>>> ['mode', 'w']]], ['device', ['vif', ['bridge', 'xenbr1'], ['mac',
>>> '00:16:3e:4c:c0:01'], ['type', 'ioemu']]]]
>>> [2008-03-19 14:07:43 xend.XendDomainInfo 3069] DEBUG
>>> (XendDomainInfo:411) parseConfig: result is {'shadow_memory': None,
>>> 'start_time': None, 'uuid': None, 'on_crash': 'restart', 'on_reboot':
>>> 'restart', 'localtime': None, 'image': ['hvm', ['kernel',
>>> '/usr/lib/xen/boot/hvmloader'], ['device_model',
>>> '/usr/lib64/xen/bin/qemu-dm'], ['pae', 1], ['vcpus', 6], ['boot',
>>> 'cda'], ['serial', 'pty'], ['vnc', 1], ['vncunused', 1], ['xauthority',
>>> '/root/.Xauthority'], ['keymap', 'en-us']], 'on_poweroff': 'destroy',
>>> 'bootloader_args': None, 'cpus': None, 'name': 'centos-5.1_core_image',
>>> 'backend': [], 'vcpus': 6, 'cpu_weight': None, 'features': None,
>>> 'vcpu_avail': None, 'memory': 4024, 'device': [('vbd', ['vbd', ['uname',
>>> 'file:/xen/domains2/centos-5.1_core_image/new_disk.img'], ['dev',
>>> 'hda'], ['mode', 'w']]), ('vbd', ['vbd', ['uname',
>>> 'file:/xen/domains2/centos-5.1_core_image/swap.img'], ['dev', 'hdb'],
>>> ['mode', 'w']]), ('vif', ['vif', ['bridge', 'xenbr1'], ['mac',
>>> '00:16:3e:4c:c0:01'], ['type', 'ioemu']])], 'bootloader': None, 'cpu':
>>> None, 'maxmem': 4024}
>>> [2008-03-19 14:07:43 xend.XendDomainInfo 3069] DEBUG
>>> (XendDomainInfo:1349) XendDomainInfo.construct: None
>>> [2008-03-19 14:07:43 xend 3069] DEBUG (balloon:127) Balloon: 2276 KiB
>>> free; need 2048; done.
>>> [2008-03-19 14:07:43 xend.XendDomainInfo 3069] ERROR
>>> (XendDomainInfo:212) Domain construction failed
>>> Traceback (most recent call last):
>>>  File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py",
>>> line 204, in create
>>>    vm.construct()
>>>  File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py",
>>> line 1369, in construct
>>>    hvm = hvm)
>>> Error: (12, 'Cannot allocate memory')
>>> 
>>> 
>>> Here's the configuration file being used:
>>> name = "centos-5.1_core_image"
>>> maxmem = 4024
>>> memory = 4024
>>> vcpus = 6
>>> builder = "hvm"
>>> kernel = "/usr/lib/xen/boot/hvmloader"
>>> boot = "cda"
>>> pae = 1
>>> acpi = 0
>>> apic = 0
>>> on_poweroff = "destroy"
>>> on_reboot = "restart"
>>> on_crash = "restart"
>>> device_model = "/usr/lib64/xen/bin/qemu-dm"
>>> sdl = 0
>>> vnc = 1
>>> vncunused = 1
>>> keymap = "en-us"
>>> disk = [ 'file:/xen/domains2/centos-5.1_core_image/disk.img,hda,w' ,
>>> 'file:/xen/domains2/centos-5.1_core_image/swap.img,hdb,w' ]
>>> vif = [ "mac=00:16:3e:4c:c0:01,bridge=xenbr1,type=ioemu" ]
>>> serial = "pty"
>>> 
>>> At the time the error occured, my memory looked like this:
>>> [root@xen1 centos-5.1_core_image]# free
>>>             total       used       free     shared    buffers     cached
>>> Mem:      15845376   15821080      24296          0     113596   15081468
>>> -/+ buffers/cache:     626016   15219360
>>> Swap:      4192956          4    4192952
>>> 
>>> The only way I've been able to duplicate this error is by making
>>> multiple copies of the 6GB disk images (created with dd, so ls shows 6GB
>>> but du shows 2GB) for different virtual machines and booting them up. I
>>> can get up to about 8 new virtual machines roughly, before I start
>>> seeing my memory error. If you're familiar with IBM's 'xen_deploy.pl'
>>> script, I've modified it slightly to meet our needs, and launch our
>>> systems with that tool. It's basic functionality is to read in a xen
>>> configuration file you want to use as a template, and output a modified
>>> one based on arguments you pass to the scripts, and then create a floppy
>>> disk image and copy the image from the template config file, and then
>>> boot the new system.
>>> 
>>> I've read in previous emails that this error occurs because of a bug in
>>> the hvmloader, and look for an update. That email response was for a 3.0
>>> version of xen, and I'm on 3.1.
>>> 
>>> What other information can I provide to help diagnose this problem?
>>> 
>>> -Kai Meyer
>>> 
>>> _______________________________________________
>>> Xen-devel mailing list
>>> Xen-devel@xxxxxxxxxxxxxxxxxxx
>>> http://lists.xensource.com/xen-devel
>>>     
>> 
>> 
>> 
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@xxxxxxxxxxxxxxxxxxx
>> http://lists.xensource.com/xen-devel
>>   
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@xxxxxxxxxxxxxxxxxxx
> http://lists.xensource.com/xen-devel



_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel