Re: [Xen-devel] Memory allocation with rebooted HVM

[Stefano Stabellini <stefano.stabellini@xxxxxxxxxxxxx>]

On Sun, 11 Oct 2009, John wrote:
> Is there a known bug in xen-unstable with rebooting of HVMs w/stubdom, and 
> having the new domU come up before the old
> domU has been shut down and its memory fully reclaimed? I have had a couple 
> of problems recently when customers rebooted
> their HVM domU through the OS; the last time, Xend froze, complaining in the 
> log that it couldn't free up enough memory
> to start the rebooted domain. Of course, that memory should have been free, 
> if the old domain had been fully shut down.
>  
> The xend.log looks like what I'd expect with an out of memory situation:
>  
> [2009-10-11 14:23:19 6419] DEBUG (XendDomainInfo:121) 
> XendDomainInfo.create_from_dict(...
> [2009-10-11 14:23:19 6419] DEBUG (XendDomainInfo:2369) 
> XendDomainInfo.constructDomain
> [2009-10-11 14:23:19 6419] DEBUG (balloon:181) Balloon: 602420 KiB free; need 
> 4096; done.
> [2009-10-11 14:23:19 6419] DEBUG (XendDomain:454) Adding Domain: 21
> [2009-10-11 14:23:19 6419] DEBUG (XendDomainInfo:2614) 
> XendDomainInfo.initDomain: 21 256
> [2009-10-11 14:23:19 6419] DEBUG (image:336) Stored a VNC password for vfb 
> access
> [2009-10-11 14:23:19 6419] DEBUG (image:843) args: boot, val: cd
> [2009-10-11 14:23:19 6419] DEBUG (image:843) args: fda, val: None
> [2009-10-11 14:23:19 6419] DEBUG (image:843) args: fdb, val: None
> [2009-10-11 14:23:19 6419] DEBUG (image:843) args: soundhw, val: None
> [2009-10-11 14:23:19 6419] DEBUG (image:843) args: localtime, val: 0
> [2009-10-11 14:23:19 6419] DEBUG (image:843) args: serial, val: None
> [2009-10-11 14:23:19 6419] DEBUG (image:843) args: std-vga, val: 0
> [2009-10-11 14:23:19 6419] DEBUG (image:843) args: isa, val: 0
> [2009-10-11 14:23:19 6419] DEBUG (image:843) args: acpi, val: 1
> [2009-10-11 14:23:19 6419] DEBUG (image:843) args: usb, val: 0
> [2009-10-11 14:23:19 6419] DEBUG (image:843) args: usbdevice, val: tablet
> [2009-10-11 14:23:19 6419] DEBUG (image:843) args: gfx_passthru, val: None
> [2009-10-11 14:23:19 6419] INFO (image:779) Need to create platform 
> device.[domid:21]
> [2009-10-11 14:23:19 6419] DEBUG (XendDomainInfo:2641) 
> _initDomain:shadow_memory=0x9, memory_static_max=0x40000000,
> memory_static_min=0x0.
>  [2009-10-11 14:23:19 6419] DEBUG (balloon:135) Balloon: tmem relinquished -1 
> KiB of 464976 KiB requested.
> [2009-10-11 14:23:19 6419] DEBUG (balloon:187) Balloon: 601008 KiB free; 0 to 
> scrub; need 1065984; retries: 20.
> [2009-10-11 14:23:19 6419] DEBUG (balloon:187) Balloon: 601008 KiB free; 0 to 
> scrub; need 1065984; retries: 20.
> [2009-10-11 14:23:42 6419] ERROR (XendDomainInfo:2694) 
> XendDomainInfo.initDomain: exception occurred
> Traceback (most recent call last):
>   File "/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py", line 
> 2655, in _initDomain
>     balloon.free(memory + shadow + vtd_mem, self)
>   File "/usr/lib64/python2.6/site-packages/xen/xend/balloon.py", line 225, in 
> free
>     free_mem + scrub_mem + dom0_alloc - dom0_min_mem))
> VmError: I need 1065984 KiB, but dom0_min_mem is 196608 and shrinking to 
> 196608 KiB would leave only 924848 KiB free.
> [2009-10-11 14:23:42 6419] ERROR (XendDomainInfo:467) VM start failed
> Traceback (most recent call last):
>   File "/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py", line 
> 453, in start
>     XendTask.log_progress(31, 60, self._initDomain)
>   File "/usr/lib64/python2.6/site-packages/xen/xend/XendTask.py", line 209, 
> in log_progress
>     retval = func(*args, **kwds)
>   File "/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py", line 
> 2697, in _initDomain
>     raise exn
> VmError: I need 1065984 KiB, but dom0_min_mem is 196608 and shrinking to 
> 196608 KiB would leave only 924848 KiB free.
> [2009-10-11 14:23:42 6419] DEBUG (XendDomainInfo:2844) 
> XendDomainInfo.destroy: domid=21
> [2009-10-11 14:23:42 6419] DEBUG (XendDomainInfo:2279) Destroying device model
> [2009-10-11 14:23:42 6419] ERROR (XendDomainInfo:2282) Device model destroy 
> failed X86_HVM_ImageHandler instance has no
> attribute 'sentinel_lock'
> Traceback (most recent call last):
>   File "/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py", line 
> 2280, in _releaseDevices
>     self.image.destroyDeviceModel()
> [2009-10-11 14:23:42 6419] DEBUG (XendDomainInfo:2286) Releasing devices
> [2009-10-11 14:23:42 6419] ERROR (XendDomainInfo:126) Domain construction 
> failed
> Traceback (most recent call last):
>   File "/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py", line 
> 124, in create_from_dict
>     vm.start()
>   File "/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py", line 
> 453, in start
>     XendTask.log_progress(31, 60, self._initDomain)
>   File "/usr/lib64/python2.6/site-packages/xen/xend/XendTask.py", line 209, 
> in log_progress
>     retval = func(*args, **kwds)
>   File "/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py", line 
> 2697, in _initDomain
>     raise exn
> VmError: I need 1065984 KiB, but dom0_min_mem is 196608 and shrinking to 
> 196608 KiB would leave only 924848 KiB free.
> [2009-10-11 14:23:42 6419] DEBUG (XendDomainInfo:2844) 
> XendDomainInfo.destroy: domid=20
> [2009-10-11 14:23:42 6419] ERROR (XendDomainInfo:2117) Failed to restart 
> domain 19.
> Traceback (most recent call last):
>   File "/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py", line 
> 2100, in _restart
>     new_dom_info)
>   File "/usr/lib64/python2.6/site-packages/xen/xend/XendDomain.py", line 989, 
> in domain_create_from_dict
>     dominfo = XendDomainInfo.create_from_dict(config_dict)
>   File "/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py", line 
> 124, in create_from_dict
>     vm.start()
>   File "/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py", line 
> 453, in start
>     XendTask.log_progress(31, 60, self._initDomain)
>   File "/usr/lib64/python2.6/site-packages/xen/xend/XendTask.py", line 209, 
> in log_progress
>     retval = func(*args, **kwds)
>   File "/usr/lib64/python2.6/site-packages/xen/xend/XendDomainInfo.py", line 
> 2697, in _initDomain
>     raise exn
> VmError: I need 1065984 KiB, but dom0_min_mem is 196608 and shrinking to 
> 196608 KiB would leave only 924848 KiB free.
> After that, Xend was unresponsive. Killing it and restarting it killed all 
> the other domU's, but new ones still couldn't
> be created, forcing me to reboot the physical machine.
>  
> Unfortunately, this doesn't seem to happen every time. I'm still trying to 
> determine how I can reproduce it reliably.

Could you please post the log of what happened right before?
.From that it should be clear if the destruction of the old guest or
stubdom didn't go smoothly.

Are you sure that this is a stubdom specific issue?
Because from the logs it seems that your host ran out of memory.
For example, were you trying to start another guest while rebooting the
first one?

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel