Re: [Xen-users] 1000 Domains: Not able to access Domu via xm console from Dom0

[Paul Harvey <jhebus@xxxxxxxxxxxxxx>]

So, i attached strace to xenconsoled to see i could find what was going on and i got this

ioctl(1023, SNDCTL_TMR_TIMEBASE or TCGETS, {B38400 opost isig icanon echo ...}) = 0
ioctl(1023, TIOCGPTN, [345])            = 0
stat("/dev/pts/345", {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 345), ...}) = 0
open("/dev/pts/345", O_RDWR|O_NOCTTY)   = -1 EMFILE (Too many open files)
close(1023)                             = 0
write(2, "Failed to create tty for domain-"..., 70) = 70
open("/etc/localtime", O_RDONLY|O_CLOEXEC) = 1023
fstat(1023, {st_mode=S_IFREG|0644, st_size=3661, ...}) = 0
fstat(1023, {st_mode=S_IFREG|0644, st_size=3661, ...}) = 0


So this is definitely a problem with file limits, but i don't understand as the current limit on files per process is 65000

On 12 December 2012 12:02, Paul Harvey <jhebus@xxxxxxxxxxxxxx> wrote:

Further to that last email looking in the xen store is confirming that the tty (pty) is not being assigned to the domains above 338

root@desktop:~# xenstore-ls /local/domain/339/console
ring-ref = "750902"
port = "2"
limit = "1048576"
type = "xenconsoled"

Whereas for 338 we get:

root@desktop:~# xenstore-ls /local/domain/338/console
ring-ref = "737537"
port = "2"
limit = "1048576"
type = "xenconsoled"
tty = "/dev/pts/342"

On 11 December 2012 22:07, Paul Harvey <jhebus@xxxxxxxxxxxxxx> wrote:

On 7 December 2012 10:03, Ian Campbell <Ian.Campbell@xxxxxxxxxx> wrote:

On Thu, 2012-12-06 at 23:27 +0000, Paul Harvey wrote:

> Any help, or is this a limitation of Xen?

One limit you might be hitting is the number of event channels which
dom0 can handle. The maximum is currently 1024 for a 32-bit domains and
4096 for 64-bit (that's per domains, not total in the system). Depending
on the configuration of the mini-os domains (e.g. number of devices etc)
you might be hitting this -- "lsevtchn 0" might give a clue if this is
happening (that tool is in /usr/lib/xen somewhere).

Work has just started on expanding these limits to ~32k and ~512k for
32- and 64-bit domains respectively, the hope is that this will be done
in time for 4.3. Look for posts from Wei Liu on xen-devel this week.

If you aren't hitting the evtchn limits then maybe you are hitting some
dom0 OS level limitation, i.e. a ulimit on the number of open file
descriptors which xenconsoled can have or some limit on the number of
pty's.

Ian.

Hi Ian, 

Thanks for the quick reply!

Have looked into your suggestions and:

* It is NOT the number of evntchns, this is much less that the limits you mention

* It is NOT the number of allowable PTY's, the number used is much less than the limit

* The number of per process file descriptors was set to 1024, but i have increased this to thousands : 

ulimint -n

10240

To hammer this point home, i built a wee C file to allocate pty's. Before i changed the limit i got problems past 1024, now it work fine as root, or any user.

But, when i create ~350 domains: 

cat /proc/<xenconsoled>/fd | wc -l  

1024

only ever goes as high as 1024, and does not increase for subsequently added domains.

Any other ideas?

Also, as a side note, any idea why the domain creation time grows quadratically?

Thanks 

Paul

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users