[Xen-users] DomU fails to reboot with storage driver domain

[Alex Velazquez <alex.j.velazquez@xxxxxxxxx>]

Hello,

I am running Xen 4.6.0, with Ubuntu 14.04 as my Domain-0.

I have a storage driver domain (PV guest running Ubuntu 14.04) that
serves a disk backend to a PV DomU (also running Ubuntu 14.04).

Here is the XL config file of StorageDom:

> name = "StorageDom"
> memory = 1024
> maxmem = 1024
> vcpus = 2
> maxvcpus = 2
> driver_domain = 1
> pci = [ "84:00.0" ]
> builder = "generic"
> kernel = "/var/lib/xen/images/vmlinuz-3.19.0-56-generic"
> ramdisk = "/var/lib/xen/images/initrd.img-3.19.0-56-generic"
> cmdline = "root=/dev/sda1 ro"


Here is the XL config file of ClientDom:

> name = "ClientDom"
> memory = 1024
> maxmem = 1024
> vcpus = 2
> maxvcpus = 2
> builder = "generic"
> kernel = "/usr/local/lib/xen/boot/pv-grub-x86_64.gz"
> cmdline = "(hd0,0)/boot/grub/menu.lst"
> disk = [ "format=raw,vdev=xvda,access=rw,backend=StorageDom,target=/dev/sdb" ]


When I start ClientDom, everything looks good. Here is the backend
entry in xenstore:

> user@ubuntu ~> $ sudo xenstore-ls /local/domain/1/backend/vbd
> 2 = ""
>  51712 = ""
>   frontend = "/local/domain/2/device/vbd/51712"
>   params = "/dev/sdb"
>   script = "/etc/xen/scripts/block"
>   frontend-id = "2"
>   online = "1"
>   removable = "0"
>   bootable = "1"
>   state = "4"
>   dev = "xvda"
>   type = "phy"
>   mode = "w"
>   device-type = "disk"
>   discard-enable = "1"
>   physical-device = "8:10"
>   feature-flush-cache = "1"
>   feature-discard = "0"
>   feature-barrier = "1"
>   feature-persistent = "1"
>   feature-max-indirect-segments = "256"
>   sectors = "1562824368"
>   info = "2"
>   sector-size = "512"
>   physical-sector-size = "512"
>   hotplug-status = "connected"


And here is the corresponding frontend entry:

> user@ubuntu ~> $ sudo xenstore-ls /local/domain/2/device/vbd
> 51712 = ""
>  backend = "/local/domain/1/backend/vbd/2/51712"
>  backend-id = "1"
>  state = "4"
>  virtual-device = "51712"
>  device-type = "disk"
>  protocol = "x86_64-abi"
>  ring-ref = "8"
>  event-channel = "17"
>  feature-persistent = "1"


I run into problems if I try to reboot ClientDom (either from within
the VM, or by calling "xl reboot ClientDom" from Domain-0). As
ClientDom goes down, the backend entry is cleared out:

> user@ubuntu ~> $ sudo xenstore-ls /local/domain/1/backend/vbd
> 2 = ""


Then ClientDom comes back up with ID 3, but the new backend/frontend
are not created:

> user@ubuntu ~> $ sudo xenstore-ls /local/domain/1/backend/vbd
> 2 = ""


> user@ubuntu ~> $ sudo xenstore-ls /local/domain/3/device/vbd
> xenstore-ls: xs_directory (/local/domain/3/device/vbd): No such file or 
> directory


> user@ubuntu ~> $ sudo xenstore-ls /local/domain/3/device
> suspend = ""
>  event-channel = ""


Connecting to ClientDom's console shows the PvGrub prompt, because it
can't find its boot disk:

>
>     GNU GRUB  version 0.97  (1048576K lower / 0K upper memory)
>
>        [ Minimal BASH-like line editing is supported.   For
>          the   first   word,  TAB  lists  possible  command
>          completions.  Anywhere else TAB lists the possible
>          completions of a device/filename. ]
>
> grubdom> root (hd0,0)
>
> Error 21: Selected disk does not exist
>
> grubdom>


If I shutdown ClientDom and start it again ("xl destroy", followed by
"xl create"), everything works again:

> user@ubuntu ~> $ sudo xenstore-ls /local/domain/1/backend/vbd
> 2 = ""
> 4 = ""
>  51712 = ""
>   frontend = "/local/domain/4/device/vbd/51712"
>   params = "/dev/sdb"
>   script = "/etc/xen/scripts/block"
>   frontend-id = "4"
>   online = "1"
>   removable = "0"
>   bootable = "1"
>   state = "4"
>   dev = "xvda"
>   type = "phy"
>   mode = "w"
>   device-type = "disk"
>   discard-enable = "1"
>   physical-device = "8:10"
>   feature-flush-cache = "1"
>   feature-discard = "0"
>   feature-barrier = "1"
>   feature-persistent = "1"
>   feature-max-indirect-segments = "256"
>   sectors = "1562824368"
>   info = "2"
>   sector-size = "512"
>   physical-sector-size = "512"
>   hotplug-status = "connected"


> user@ubuntu ~> $ sudo xenstore-ls /local/domain/4/device/vbd
> 51712 = ""
>  backend = "/local/domain/1/backend/vbd/4/51712"
>  backend-id = "1"
>  state = "4"
>  virtual-device = "51712"
>  device-type = "disk"
>  protocol = "x86_64-abi"
>  ring-ref = "8"
>  event-channel = "17"
>  feature-persistent = "1"


However, I can't prevent a user from attempting to reboot ClientDom
from within the VM, and when it drops to the PvGrub prompt,
intervention on Domain-0 is required to restart it.

I don't run into this problem if Domain-0 is the disk backend. When
the domain reboots, the old backend entry is removed xenstore (not
just cleared out, as in the case of StorageDom), the new
backend/frontend entries are created, and ClientDom starts up
correctly. One difference I can see is that StorageDom is running an
"xl devd" process (which is started by the xendriverdomain init
script), whereas Domain-0 is not. Is the use-case of rebooting a DomU
supported by storage driver domains?

Has anyone else encountered problems rebooting a DomU that uses a
storage driver domain?

Thanks,
Alex

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users