[Xen-users] xm create fails when run in parallel

[Nick Craig-Wood <nick@xxxxxxxxxxxxxx>]

We've been starting xen domains in parallel in our init scripts in
order to speed up starting lots of domains at once.  However this
seems to cause a problem with xen-3.0.2.

  # xm create sarge & xm create sarge2 & xm create sarge3 & xm create sarge4 & 
xm create sarge5 & xm create sarge6 &
  [2] 9555
  Using config file "/etc/xen/sarge2".
  Using config file "/etc/xen/sarge".
  [3] 9556
  Using config file "/etc/xen/sarge3".
  [4] 9559
  [5] 9560
  [6] 9561
  [7] 9562
  Using config file "/etc/xen/sarge4".
  Using config file "/etc/xen/sarge6".
  Using config file "/etc/xen/sarge5".
  Started domain sarge2
  Error: Device 2049 (vbd) could not be connected. Hotplug scripts not working.
  Error: Device 2049 (vbd) could not be connected. Hotplug scripts not working.
  Error: Device 2049 (vbd) could not be connected. Hotplug scripts not working.
  Error: Device 2050 (vbd) could not be connected. Hotplug scripts not working.
  Started domain sarge6

As compared to this which works fine

  # xm create sarge ; xm create sarge2 ; xm create sarge3 ; xm create sarge4 ; 
xm create sarge5 ; xm create sarge6
  Using config file "/etc/xen/sarge".
  Started domain sarge
  Using config file "/etc/xen/sarge2".
  Started domain sarge2
  Using config file "/etc/xen/sarge3".
  Started domain sarge3
  Using config file "/etc/xen/sarge4".
  Started domain sarge4
  Using config file "/etc/xen/sarge5".
  Started domain sarge5
  Using config file "/etc/xen/sarge6".
  Started domain sarge6

The error message seems to suggest it is a hotplug problem.  This used
to work with xen2.

The domains are routed not switched.

xen-hotplug.log:

  ifdown: interface vif107.0 not configured
  Cannot find device "vif107.0"
  ifdown: interface vif109.0 not configured
  Cannot find device "vif109.0"
  ifdown: interface vif111.0 not configured
  Cannot find device "vif111.0"
  ioctl: LOOP_CLR_FD: Device or resource busy
  ifdown: interface vif108.0 not configured
  Cannot find device "vif108.0"

xend.log for one domain:

  [2006-07-05 16:02:50 xend.XendDomainInfo] DEBUG (XendDomainInfo:1196) 
XendDomainInfo.initDomain: 109 1.0
  [2006-07-05 16:02:50 xend] INFO (image:135) buildDomain os=linux dom=109 
vcpus=4
  [2006-07-05 16:02:50 xend] DEBUG (image:173) dom            = 109
  [2006-07-05 16:02:50 xend] DEBUG (DevController:103) DevController: writing 
{'virtual-device': '2049', 'backend-id': '0', 'state': '1', 'backend': 
'/local/domain/0/backend/vbd/109/2049'} to /local/domain/109/device/vbd/2049.
  [2006-07-05 16:02:50 xend] DEBUG (DevController:105) DevController: writing 
{'domain': 'sarge4', 'frontend': '/local/domain/109/device/vbd/2049', 'dev': 
'sda1', 'state': '1', 'params': '/home/sarge4/root_fs', 'mode': 'w', 
'frontend-id': '109', 'type': 'file'} to /local/domain/0/backend/vbd/109/2049.
  [2006-07-05 16:02:50 xend] DEBUG (DevController:103) DevController: writing 
{'virtual-device': '2050', 'backend-id': '0', 'state': '1', 'backend': 
'/local/domain/0/backend/vbd/109/2050'} to /local/domain/109/device/vbd/2050.
  [2006-07-05 16:02:50 xend] DEBUG (DevController:105) DevController: writing 
{'domain': 'sarge4', 'frontend': '/local/domain/109/device/vbd/2050', 'dev': 
'sda2', 'state': '1', 'params': '/home/sarge4/swap', 'mode': 'w', 
'frontend-id': '109', 'type': 'file'} to /local/domain/0/backend/vbd/109/2050.
  [2006-07-05 16:02:50 xend] DEBUG (DevController:103) DevController: writing 
{'backend-id': '0', 'mac': '00:16:3e:7e:ba:84', 'handle': '0', 'state': '1', 
'backend': '/local/domain/0/backend/vif/109/0'} to 
/local/domain/109/device/vif/0.
  [2006-07-05 16:02:50 xend] DEBUG (DevController:105) DevController: writing 
{'domain': 'sarge4', 'handle': '0', 'script': '/etc/xen/scripts/vif-route', 
'ip': '10.21.1.14', 'state': '1', 'frontend': '/local/domain/109/device/vif/0', 
'mac': '00:16:3e:7e:ba:84', 'frontend-id': '109'} to 
/local/domain/0/backend/vif/109/0.
  [2006-07-05 16:02:50 xend.XendDomainInfo] DEBUG (XendDomainInfo:696) Storing 
domain details: {'console/ring-ref': '186518', 'console/port': '2', 
'cpu/3/availability': 'online', 'name': 'sarge4', 'console/limit': '1048576', 
'cpu/2/availability': 'online', 'vm': 
'/vm/d0498745-14be-28dc-0a35-bb6cd6f2b47f', 'domid': '109', 
'cpu/0/availability': 'online', 'memory/target': '32768', 'store/ring-ref': 
'186519', 'cpu/1/availability': 'online', 'store/port': '1'}
  [2006-07-05 16:02:51 xend] DEBUG (DevController:403) hotplugStatusCallback 
/local/domain/0/backend/vif/109/0/hotplug-status.
  [2006-07-05 16:02:51 xend] DEBUG (DevController:403) hotplugStatusCallback 
/local/domain/0/backend/vbd/109/2049/hotplug-status.
  [2006-07-05 16:03:01 xend.XendDomainInfo] DEBUG (XendDomainInfo:1327) 
XendDomainInfo.destroy: domid=109
  [2006-07-05 16:03:01 xend.XendDomainInfo] DEBUG (XendDomainInfo:1335) 
XendDomainInfo.destroyDomain(109)
  [2006-07-05 16:03:12 xend] DEBUG (DevController:403) hotplugStatusCallback 
/local/domain/0/backend/vbd/109/2049/hotplug-status.

The host is running debian/testing with udev 0.091-2.

We can stop doing this easily enough of course, but it would be a
shame to lose the speed bump, especially when destroying the domains
(which seems to show a similar problem - the network teardown code
moans about non-existent devices)

Any ideas?

Thanks

-- 
Nick Craig-Wood <nick@xxxxxxxxxxxxxx> -- http://www.craig-wood.com/nick

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users