[Xen-devel] 101th domU fails to start with "SETVCPUCONTEXT failed"

[Lennert Van Alboom <lennert.vanalboom@xxxxxxxxxx>]

Hello there,


After asking in ##xen on freenode and on the xen-users mailinglist,
people advised me to ask here seeing the nature of our issue.


We (a small hosting community) are running a steadily growing number 
of Xen domUs on a quad dualcore Xeon server with 64GB ram. We've got 
100 running domUs at the moment. Trying to create a new one results in 
this error:
Error: (1, 'Internal error', 'launch_vm: SETVCPUCONTEXT failed (rc=-1)\n')

If I shut down another domain, I can create one again, but the limit
seems to be 100 domUs. Each domU has one network interface and one or 
two disk devices (raid-1 md device consisting of two iscsi luns).
Version of Xen is rather old: 3.2.1-amd64 (Debian). 

I've been tracking this problem through the xen source, and found this:
The launch_vm code is located in
http://lxr.mstier.de/Xen/source/xen_3.2.1/tools/libxc/xc_dom_boot.c?v=3.2.1#049
; the function that is causing the return code of -1 is do_domctl.

I'm unsure as to where do_domctl is really located - I've found two
locations:
http://lxr.mstier.de/Xen/source/xen_3.2.1/tools/libxc/xc_private.h?v=3.2.1#099
and
http://lxr.mstier.de/Xen/source/xen_3.2.1/xen/common/domctl.c?v=3.2.1#179
. Neither give me much of a clue. 

In case of the first definition, the returncode is given by 
ret = do_xen_hypercall(xc_handle, &hypercall)
which leads me further to 
ioctl(xc_handle, cmd, data);
and after that I'm stumped. 

In the second case, the -1 comes *either* from IS_PRIV(current->domain)
failing, causing an EPERM error. This sounds unlikely. The other place
where a -1 could be given is ret = xsm_setvcpucontext(d);
which translates back to
xsm_call(setvcpucontext(d));
and there I'm stuck again. I'm totally not capable of deciphering
xsm_ops and its meanings.


Does anyone with some insight in the innards of Xen have a clue on what
might be causing our issues?


Kind regards,


Lennert

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel