------------------------------------------------------------------
åääïAndrew Cooper <andrew.cooper3@xxxxxxxxxx>
åéæéï2014å9æ15æ(ææä) 22:01
æääïåæé(äæ) <huaixin.chx@xxxxxxxxxxxxxxx>ïkeir <keir@xxxxxxx>ïIan.Campbell <Ian.Campbell@xxxxxxxxxx>ïstefano.stabellini <stefano.stabellini@xxxxxxxxxxxxx>ïxen-devel <xen-devel@xxxxxxxxxxxxxxxxxxx>ïIan.Jackson <Ian.Jackson@xxxxxxxxxxxxx>ïgeorge.dunlap <george.dunlap@xxxxxxxxxxxxx>
æãéïååæ(åè) <jinsong.liu@xxxxxxxxxxxxxxx>
äãéïRe: live-migration restore failed error
On 15/09/2014 10:41, åæé(äæ) wrote:
We are working on
live-migration based on Xen-4.0.1(For history reason, and
meantime we are upgrading our Xen to very latest version).
Restore failed when live migrating ubuntu12.04 on xen-4.0.1.
To be more specific, error occurred when populating memory.
Error messages are as follow:
[2014-09-12 22:40:40 7331 1189091648] DEBUG
(XendCheckpoint:307) [xc_restore]:
/usr/lib64/xen/bin/xc_restore 4 2763 3 4 1 1 1 0
[2014-09-12 22:40:40 7331 1189091648] DEBUG
(XendCheckpoint:428) Thread-40188
[2014-09-12 22:40:40 7331 1172306240] INFO
(XendCheckpoint:476) Thread-40188:xc_domain_restore start:
p2m_size = fefff
[2014-09-12 22:40:40 7331 1172306240] INFO
(XendCheckpoint:476) Thread-40188:Reloading memory pages: 0%
[2014-09-12 22:40:50 7331 1172306240] INFO
(XendCheckpoint:476) Thread-40188:Failed allocation for dom
2763: 128 extents of order 0
[2014-09-12 22:40:50 7331 1172306240] INFO
(XendCheckpoint:476) Thread-40188:ERROR Internal error: Failed
to allocate memory for batch.!
[2014-09-12 22:40:50 7331 1172306240] INFO
(XendCheckpoint:476) Thread-40188:
[2014-09-12 22:40:50 7331 1172306240] INFO
(XendCheckpoint:476) Thread-40188:Restore exit with rc=1
[2014-09-12 22:40:50 7331 1189091648] DEBUG
(XendCheckpoint:462) /usr/lib64/xen/bin/xc_restore 4 2763 3 4
1 1 1 0 failed status 256
[2014-09-12 22:40:50 7331 1189091648] DEBUG
(XendDomainInfo:3845) XendDomainInfo.destroy: domid=2763
In
this case, populate_physmap terminated with nr_done 127. So
xc_memory_op return 127 while nr_extents equals 128.
This problem happends once every 1770th live migration or so.
As I am debugging this issue, I'm sending this email to ask
for suggestions on this issue.
Thanks,
Huaixin
Chang
Xen is unable to fulfil the allocation request. You have run out of
host memory.
~Andrew
Here are some more clues.
I'm migrating ubuntu12.04(with 1G or 512M memory) on two machines with around 96G of memory, back and forth. The issue occurs around 1770 times of migration erery time, whether guest memory is 512M or 1G.
In the pasted xend log, a request of 128 pages of non-contiguous memory failed. Currently, I am conducting another round of migration test, which has completed 230 times of migration, and hopefully will terminate after about one day. So far, I do not see a major decrese of hypervisor memory. I will check whether there is memory issues when the problem shows up.
total_memory : 98276
free_memory : 84454
Sorry for not being able to provide a hypervisor log at the moment. Previously I printed too many of messages, most of them were suppressed and no helpful message could be found. I will also check whether this round will help.
Thanks,
Huaixin Chang
