[Top] [All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] blktap2 device creation failing after 162 devices w/Xen4.0 + linux-2.6.31.13

On 04/14/2010 01:24 AM, Daniel Stodden wrote:

On Wed, 2010-04-14 at 01:40 -0400, John McCullough wrote:

Daniel,

That did the trick and got us up to 256, Thanks!

Out of curiosity, what's standing in the way of more devices?

I must admit I never tried. Lack of maybe a couple sparse tables here
and there?


We tried raising the MAX_*_DEVICES constants in these files to 512, but
didn't have any luck:
linux-2.6-pvops.git/drivers/xen/blktap/blktap.h
tools/blktap2/include/blktaplib.h
tools/blktap/lib/blktaplib.h

(The error is now "vbd open failed: -6")

That would be an ENXIO, probably while trying to open the ring (can you
verify that with an strace -f?)


Looks like it is ENXIO:

...
[pid  4154] open("/dev/xen/blktap-2/blktap256", O_RDWR <unfinished ...>

[pid  4153] close(4)                    = 0
[pid  4153] fcntl(3, F_GETFL)           = 0 (flags O_RDONLY)
[pid  4153] brk(0)                      = 0xa1e000
[pid  4153] brk(0xa3f000)               = 0xa3f000
[pid  4153] fstat(3, {st_mode=S_IFIFO|0600, st_size=0, ...}) = 0
[pid 4153] mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f6844b40000 
[pid  4153] lseek(3, 0, SEEK_CUR)       = -1 ESPIPE (Illegal seek)
[pid  4153] read(3, <unfinished ...>
[pid 4154] <... open resumed> ) = -1 ENXIO (No such device or address) 
[pid  4154] gettimeofday({1271262219, 398530}, NULL) = 0
[pid 4154] sendto(0, "<27>Apr 14 09:23:39 tapdisk2[415"..., 115, MSG_NOSIGNAL, NULL, 0) = 115 
[pid  4154] close(3)                    = 0
[pid  4154] gettimeofday({1271262219, 398836}, NULL) = 0
[pid 4154] sendto(0, "<30>Apr 14 09:23:39 tapdisk2[415"..., 109, MSG_NOSIGNAL, NULL, 0) = 109 
[pid  4154] gettimeofday({1271262219, 399161}, NULL) = 0
[pid 4154] sendto(0, "<27>Apr 14 09:23:39 tapdisk2[415"..., 86, MSG_NOSIGNAL, NULL, 0) = 86 
[pid  4154] write(1, "-6: vbd open failed: -6\n", 24) = 24
... (err, ioctl to control, close and exit)


Thanks,
John


Should indeed work for up to 2^20, or MAX_BLKTAP_DEVICEs.

I don't think this fails in the ring code, we return ENODEV there (which
is a bug) and the kernel won't correct that code.

Daniel


I noticed an artificial limit of 26*26 in the tapdev naming scheme, but
I didn't look very thoroughly.

Thanks again,
John


Daniel Stodden wrote:

Hi.

Please echo $((N * (32 * 11 + 50) + SOME_HEADROOM))
to /proc/sys/fs/aio-max-nr. Or set it up in sysctl.conf.

Where N is the number of devices you desire.

As for the apparently missing big fat complaint you should have seen pop
in syslog, I'll keep it in mind  for the next update. :}

Cheers,
Daniel

On Tue, 2010-04-13 at 20:20 -0400, John McCullough wrote:


I have been working with a colleague to get a large number of small VMs
running on a single system.  We were hoping for at least 100, but we
seem to be topping out around 81.  Each VM has a disk image and a swap
image.  It seemed like we were hitting a blktap limit, so we tried
bumping up the MAX macros in tool/blktap2 and the linux driver, with no
change.  (Though we haven't hit the theoretical 256 blktap devices yet).

(Initially we were only able to get 64 VMs until we bumped
CONFIG_NR_CPUS from 8 to 64 to increase the number of dynirqs).

To isolate the problem, I tried creating a large number of blktap
devices in the dom0 with no guests running and I ran into the same
ceiling (162 total devices).   Commands to reproduce the problem follow:

echo 9>  /sys/class/blktap2/verbosity

for x in `seq 0 163`; do
          if ( ! dd if=/dev/zero of=/scratch/test-$x.img bs=1 count=1
seek=1M 2>  /dev/null); then
                  echo "Qemu fail on $x"; exit 1
          fi
          if ( ! tapdisk2 -n aio:/scratch/test-$x.img) ; then
                  echo "blktap fail on $x"; exit 1
          fi
done

The result:
...
/dev/xen/blktap-2/tapdev159
/dev/xen/blktap-2/tapdev160
/dev/xen/blktap-2/tapdev161
/dev/xen/blktap-2/tapdev162
unrecognized child response
blktap fail on 163

Dmesg output associated with 163:
[ 1288.839978] blktap_sysfs_create: adding attributes for dev
ffff88019e4d1e00
[ 1288.840947] blktap_sysfs_destroy

(Output for the prior devices includes processing a request, and a
blktap_device_finish_request)

No related xm dmesg output.

$ hg tip
changeset:   21091:f28f1ee587c8
tag:         tip
user:        Keir Fraser<keir.fraser@xxxxxxxxxx>
date:        Wed Apr 07 12:38:28 2010 +0100
summary:     Added signature for changeset 484179b2be5d

$ uname -a
Linux sysnet121 2.6.32-3-amd64 #1 SMP Wed Feb 24 18:07:42 UTC 2010
x86_64 GNU/Linux

Has anyone had contrary experience? Does anyone know where the 162 max
is coming from?

Thanks,
John

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel




_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel



_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel



_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel



_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel

 

©2013 Xen Project, A Linux Foundation Collaborative Project. All Rights Reserved.
Linux Foundation is a registered trademark of The Linux Foundation.
Xen Project is a trademark of The Linux Foundation.

Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.