[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Xen-users] i can't run a xen - virtual machine on a glusterfs mount point


  • To: Xen-users@xxxxxxxxxxxxxxxxxxx
  • From: Jordi Moles Blanco <jordi@xxxxxxxxx>
  • Date: Mon, 07 Jul 2008 18:54:21 +0200
  • Cc:
  • Delivery-date: Mon, 07 Jul 2008 09:55:05 -0700
  • List-id: Xen user discussion <xen-users.lists.xensource.com>

hi,

i've been using glusterfs for a while now in different scenarios and now i'm facing a really strange problem, or at least it seems so to me....

the thing is that i've got a glusterfs, with six nodes and several clients (xen-servers) mounting the shared system. The idea is to have a big shared storage for all the vm's i've got.

when i try to run a vm created from xen 3.2 which is stored on gluster, it won't run.

it says:

Error: Device 2049 (vbd) could not be connected. Backend device not found.

and if you look at the xen log's file:

*****************

[2008-07-07 18:51:31 4020] DEBUG (__init__:1072) XendDomainInfo.create(['vm', ['name', 'xen_testdedicat1'], ['memory', '256'], ['on_poweroff', 'destroy'], ['on_reboot', 'restart'], ['on_crash', 'restart'], ['vcpus', 1], ['on_xend_start', 'ignore'], ['on_xend_stop', 'ignore'], ['image', ['linux', ['kernel', '/boot/vmlinuz-2.6.18-6-xen-amd64'], ['ramdisk', '/boot/initrd.img-2.6.18-6-xen-amd64'], ['root', '/dev/sda1 ro']]], ['device', ['vbd', ['uname', 'file:/mnt/gluster/domains/xen_testdedicat1/disk.img'], ['dev', 'sda1'], ['mode', 'w']]], ['device', ['vif', ['ip', '192.168.1.164']]]]) [2008-07-07 18:51:31 4020] DEBUG (__init__:1072) XendDomainInfo.constructDomain [2008-07-07 18:51:31 4020] DEBUG (__init__:1072) Balloon: 262956 KiB free; need 2048; done.
[2008-07-07 18:51:31 4020] DEBUG (__init__:1072) Adding Domain: 2
[2008-07-07 18:51:31 4020] DEBUG (__init__:1072) XendDomainInfo.initDomain: 2 256 [2008-07-07 18:51:31 4020] DEBUG (__init__:1072) _initDomain:shadow_memory=0x0, memory_static_max=0x10000000, memory_static_min=0x0. [2008-07-07 18:51:31 4020] DEBUG (__init__:1072) Balloon: 262948 KiB free; need 262144; done. [2008-07-07 18:51:31 4020] INFO (__init__:1072) buildDomain os=linux dom=2 vcpus=1
[2008-07-07 18:51:31 4020] DEBUG (__init__:1072) domid          = 2
[2008-07-07 18:51:31 4020] DEBUG (__init__:1072) memsize        = 256
[2008-07-07 18:51:31 4020] DEBUG (__init__:1072) image = /boot/vmlinuz-2.6.18-6-xen-amd64
[2008-07-07 18:51:31 4020] DEBUG (__init__:1072) store_evtchn   = 1
[2008-07-07 18:51:31 4020] DEBUG (__init__:1072) console_evtchn = 2
[2008-07-07 18:51:31 4020] DEBUG (__init__:1072) cmdline = root=/dev/sda1 ro [2008-07-07 18:51:31 4020] DEBUG (__init__:1072) ramdisk = /boot/initrd.img-2.6.18-6-xen-amd64
[2008-07-07 18:51:31 4020] DEBUG (__init__:1072) vcpus          = 1
[2008-07-07 18:51:31 4020] DEBUG (__init__:1072) features       =
[2008-07-07 18:51:31 4020] INFO (__init__:1072) createDevice: vbd : {'uuid': 'a4d3fb99-4d54-687e-1e5e-b3c82264547d', 'bootable': 1, 'driver': 'paravirtualised', 'dev': 'sda1', 'uname': 'file:/mnt/gluster/domains/xen_testdedicat1/disk.img', 'mode': 'w'} [2008-07-07 18:51:31 4020] DEBUG (__init__:1072) DevController: writing {'virtual-device': '2049', 'device-type': 'disk', 'protocol': 'x86_64-abi', 'backend-id': '0', 'state': '1', 'backend': '/local/domain/0/backend/vbd/2/2049'} to /local/domain/2/device/vbd/2049. [2008-07-07 18:51:31 4020] DEBUG (__init__:1072) DevController: writing {'domain': 'xen_testdedicat1', 'frontend': '/local/domain/2/device/vbd/2049', 'uuid': 'a4d3fb99-4d54-687e-1e5e-b3c82264547d', 'dev': 'sda1', 'state': '1', 'params': '/mnt/gluster/domains/xen_testdedicat1/disk.img', 'mode': 'w', 'online': '1', 'frontend-id': '2', 'type': 'file'} to /local/domain/0/backend/vbd/2/2049. [2008-07-07 18:51:31 4020] INFO (__init__:1072) createDevice: vif : {'ip': '192.168.1.164', 'mac': '00:16:3e:29:f1:ad', 'uuid': '2746c385-a8e6-e5e7-00bc-32fc7f3d6272'} [2008-07-07 18:51:31 4020] DEBUG (__init__:1072) DevController: writing {'mac': '00:16:3e:29:f1:ad', 'handle': '0', 'protocol': 'x86_64-abi', 'backend-id': '0', 'state': '1', 'backend': '/local/domain/0/backend/vif/2/0'} to /local/domain/2/device/vif/0. [2008-07-07 18:51:31 4020] DEBUG (__init__:1072) DevController: writing {'domain': 'xen_testdedicat1', 'handle': '0', 'uuid': '2746c385-a8e6-e5e7-00bc-32fc7f3d6272', 'script': '/etc/xen/scripts/vif-bridge', 'ip': '192.168.1.164', 'state': '1', 'frontend': '/local/domain/2/device/vif/0', 'mac': '00:16:3e:29:f1:ad', 'online': '1', 'frontend-id': '2'} to /local/domain/0/backend/vif/2/0. [2008-07-07 18:51:32 4020] DEBUG (__init__:1072) Storing VM details: {'on_xend_stop': 'ignore', 'shadow_memory': '0', 'uuid': '34b70208-d172-6d75-344e-888579b73c2c', 'on_reboot': 'restart', 'start_time': '1215449492.96', 'on_poweroff': 'destroy', 'on_xend_start': 'ignore', 'on_crash': 'restart', 'xend/restart_count': '0', 'vcpus': '1', 'vcpu_avail': '1', 'image': "(linux (kernel /boot/vmlinuz-2.6.18-6-xen-amd64) (ramdisk /boot/initrd.img-2.6.18-6-xen-amd64) (args 'root=/dev/sda1 ro ') (notes (FEATURES 'writable_page_tables|writable_descriptor_tables|auto_translated_physmap|pae_pgdir_above_4gb|supervisor_mode_kernel') (VIRT_BASE 18446744071562067968) (GUEST_VERSION 2.6) (PADDR_OFFSET 18446744071562067968) (GUEST_OS linux) (HYPERCALL_PAGE 18446744071564189696) (LOADER generic) (ENTRY 18446744071564165120) (XEN_VERSION xen-3.0)))", 'name': 'xen_testdedicat1'} [2008-07-07 18:51:32 4020] DEBUG (__init__:1072) Storing domain details: {'console/ring-ref': '2214313', 'image/entry': '18446744071564165120', 'console/port': '2', 'store/ring-ref': '2214314', 'image/loader': 'generic', 'vm': '/vm/34b70208-d172-6d75-344e-888579b73c2c', 'control/platform-feature-multiprocessor-suspend': '1', 'image/guest-os': 'linux', 'image/features/writable-descriptor-tables': '1', 'image/virt-base': '18446744071562067968', 'memory/target': '262144', 'image/guest-version': '2.6', 'image/features/supervisor-mode-kernel': '1', 'console/limit': '1048576', 'image/paddr-offset': '18446744071562067968', 'image/hypercall-page': '18446744071564189696', 'cpu/0/availability': 'online', 'image/features/pae-pgdir-above-4gb': '1', 'image/features/writable-page-tables': '1', 'console/type': 'xenconsoled', 'image/features/auto-translated-physmap': '1', 'name': 'xen_testdedicat1', 'domid': '2', 'image/xen-version': 'xen-3.0', 'store/port': '1'} [2008-07-07 18:51:32 4020] DEBUG (__init__:1072) DevController: writing {'protocol': 'x86_64-abi', 'state': '1', 'backend-id': '0', 'backend': '/local/domain/0/backend/console/2/0'} to /local/domain/2/device/console/0. [2008-07-07 18:51:32 4020] DEBUG (__init__:1072) DevController: writing {'domain': 'xen_testdedicat1', 'protocol': 'vt100', 'uuid': '254941ec-2571-00da-95a9-a8d6fcce4414', 'frontend': '/local/domain/2/device/console/0', 'state': '1', 'location': '2', 'online': '1', 'frontend-id': '2'} to /local/domain/0/backend/console/2/0. [2008-07-07 18:51:32 4020] DEBUG (__init__:1072) XendDomainInfo.handleShutdownWatch
[2008-07-07 18:51:32 4020] DEBUG (__init__:1072) Waiting for devices vif.
[2008-07-07 18:51:32 4020] DEBUG (__init__:1072) Waiting for 0.
[2008-07-07 18:51:32 4020] DEBUG (__init__:1072) hotplugStatusCallback /local/domain/0/backend/vif/2/0/hotplug-status. [2008-07-07 18:51:32 4020] DEBUG (__init__:1072) hotplugStatusCallback /local/domain/0/backend/vif/2/0/hotplug-status.
[2008-07-07 18:51:32 4020] DEBUG (__init__:1072) hotplugStatusCallback 1.
[2008-07-07 18:51:32 4020] DEBUG (__init__:1072) Waiting for devices vbd.
[2008-07-07 18:51:32 4020] DEBUG (__init__:1072) Waiting for 2049.
[2008-07-07 18:51:32 4020] DEBUG (__init__:1072) hotplugStatusCallback /local/domain/0/backend/vbd/2/2049/hotplug-status. [2008-07-07 18:51:32 4020] DEBUG (__init__:1072) hotplugStatusCallback /local/domain/0/backend/vbd/2/2049/hotplug-status.
[2008-07-07 18:51:32 4020] DEBUG (__init__:1072) hotplugStatusCallback 2.
[2008-07-07 18:51:32 4020] DEBUG (__init__:1072) XendDomainInfo.destroy: domid=2 [2008-07-07 18:51:32 4020] DEBUG (__init__:1072) XendDomainInfo.destroyDomain(2)
[2008-07-07 18:51:32 4020] DEBUG (__init__:1072) Destroying device model
[2008-07-07 18:51:32 4020] DEBUG (__init__:1072) Releasing devices
[2008-07-07 18:51:32 4020] DEBUG (__init__:1072) Removing vif/0
[2008-07-07 18:51:32 4020] DEBUG (__init__:1072) XendDomainInfo.destroyDevice: deviceClass = vif, device = vif/0
[2008-07-07 18:51:32 4020] DEBUG (__init__:1072) Removing vbd/2049
[2008-07-07 18:51:32 4020] DEBUG (__init__:1072) XendDomainInfo.destroyDevice: deviceClass = vbd, device = vbd/2049
[2008-07-07 18:51:32 4020] DEBUG (__init__:1072) Removing console/0
[2008-07-07 18:51:32 4020] DEBUG (__init__:1072) XendDomainInfo.destroyDevice: deviceClass = console, device = console/0


*****************


nodes don't log any error or warning at all, and the xen-server which is trying to run the vm says this:

************
2008-07-07 17:43:07 E [afr.c:2391:afr_writev_cbk] grup1: (path=/domains/xen_testdedicat1/disk.img child=espai2) op_ret=-1 op_errno=22 2008-07-07 17:43:07 E [fuse-bridge.c:1645:fuse_writev_cbk] glusterfs-fuse: 656: WRITE => -1 (22) 2008-07-07 17:43:07 E [afr.c:2699:afr_flush] grup1: afrfdp->fdstate[] is 0, returning ENOTCONN 2008-07-07 17:43:07 E [fuse-bridge.c:945:fuse_err_cbk] glusterfs-fuse: 657: (16) ERR => -1 (107)
**************

i'm using this software versions

linux debian etch, kernel 2.6.18-6-xen-amd64
xen-hypervisor-3.2-1-amd64
fuse-2.7.3glfs10
glusterfs--mainline--2.5--patch-788

the thing is, however, that if i move the virtual machine create from xen 3.2 from the shared gluster mount point to a local path, it will run without any problem at all... is it that gluster is not fast enough to provide xen with the data needed to launch the vm? could i do anything to fix that?

these are the specs files:

node side:

***********

volume espa
       type storage/posix
       option directory /mnt/compartit
end-volume

volume spai
       type performance/io-threads
       option thread-count 4
       option cache-size 32MB
       subvolumes espa
end-volume

volume espai
       type performance/write-behind
       option aggregate-size 1MB
       option flush-behind on
       subvolumes spai
end-volume

volume ultim
       type protocol/server
       subvolumes espai
       option transport-type tcp/server
       option auth.ip.espai.allow *
end-volume

***********

client/xen side:

***********

volume espai1
   type protocol/client
   option transport-type tcp/client
   option remote-host 192.168.1.204
   option remote-subvolume espai
end-volume

volume namespace1
   type protocol/client
   option transport-type tcp/client
   option remote-host 192.168.1.204
   option remote-subvolume nm
end-volume

volume espai2
   type protocol/client
   option transport-type tcp/client
   option remote-host 192.168.1.206
   option remote-subvolume espai
end-volume

volume espai3
   type protocol/client
   option transport-type tcp/client
   option remote-host 192.168.1.213
   option remote-subvolume espai
end-volume

volume espai4
   type protocol/client
   option transport-type tcp/client
   option remote-host 192.168.1.161
   option remote-subvolume espai
end-volume

volume namespace2
   type protocol/client
   option transport-type tcp/client
   option remote-host 192.168.1.161
   option remote-subvolume nm
end-volume

volume espai5
   type protocol/client
   option transport-type tcp/client
   option remote-host 192.168.1.162
   option remote-subvolume espai
end-volume

volume espai6
   type protocol/client
   option transport-type tcp/client
   option remote-host 192.168.1.163
   option remote-subvolume espai
end-volume

volume grup1
   type cluster/afr
   subvolumes espai1 espai2
end-volume

volume grup2
   type cluster/afr
   subvolumes espai3 espai4
end-volume

volume grup3
   type cluster/afr
   subvolumes espai5 espai6
end-volume

volume nm1
   type cluster/afr
   subvolumes namespace1 namespace2
end-volume

volume ultim
   type cluster/unify
   subvolumes grup1 grup2 grup3
   option scheduler rr
   option namespace nm1
end-volume

volume iot
 type performance/io-threads
 option thread-count 4
 option cache-size 32MB
 subvolumes ultim
end-volume

volume ioc
type performance/io-cache
option cache-size 64MB
option page-size 1MB
option force-revalidate-timeout 2
subvolumes iot
end-volume

***********

i've already tried for a while to play with the spec file... adding and removing write and read buffers... but the final result is the same... the vm won't run.


any idea?

thank you.

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users


 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.