Re: [Xen-users] Error with DRBD9 and Xen 4.6.5 with HVM Guest

[Justin Pretorius <justin.pretorius@xxxxxxxxx>]

On 09/07/18 09:08, Roger Pau Monné wrote:
> On Thu, Sep 06, 2018 at 04:41:11PM +0200, Justin Pretorius wrote:
> > Hello
> >
> > This is my first ever email list help request, apologies if I have not 100%
> > got the protocol correct.
> >
> > I am currently using Xen on Ubuntu 16.04, and am using a number of PVM
> > guests successfully using DRBD9 as primary root disk, but when testing an
> > HVM PfSense (BSD) guest, I am having some issues.
> >
> > DRBD9 is successfully set up and when creating the guest using the config as
> > follows
> >
> > disk=['drbd:disk1-firewall2,hda,w']
> >
> > The guest will not create with the following info
> >
> > xl -v create /etc/xen/firewall2.cfg
> >
> > xc: detail: elf_parse_binary: phdr: paddr=0x100000 memsz=0x5b844
> > xc: detail: elf_parse_binary: memory: 0x100000 -> 0x15b844
> > xc: detail: VIRTUAL MEMORY ARRANGEMENT:
> > xc: detail:   Loader:   0000000000100000->000000000015b844
> > xc: detail:   Modules:  0000000000000000->0000000000000000
> > xc: detail:   TOTAL:    0000000000000000->000000001f800000
> > xc: detail:   ENTRY:    0000000000100630
> > xc: detail: PHYSICAL MEMORY ALLOCATION:
> > xc: detail:   4KB PAGES: 0x0000000000000200
> > xc: detail:   2MB PAGES: 0x00000000000000fb
> > xc: detail:   1GB PAGES: 0x0000000000000000
> > xc: detail: elf_load_binary: phdr 0 at 0x7f979ffaa000 -> 0x7f979fffbdd1
> > domainbuilder: detail: xc_dom_gnttab_hvm_seed: called, pfn=0xff000
> > libxl: error: libxl_dm.c:1872:device_model_spawn_outcome: domain 74 device
> > model: spawn failed (rc=-3)
> > libxl: error: libxl_create.c:1344:domcreate_devmodel_started: device model
> > did not start: -3
> > libxl: error: libxl_dm.c:1987:kill_device_model: Device Model already exited
> > libxl: error: libxl.c:1610:libxl__destroy_domid: non-existant domain 74
> > libxl: error: libxl.c:1568:domain_destroy_callback: unable to destroy guest
> > with domid 74
> > libxl: error: libxl.c:1495:domain_destroy_cb: destruction of domain 74
> > failed
> >
> > Contents of guest log
> >
> > qemu-system-i386: -drive
> > file=disk1-firewall2,if=ide,index=0,media=disk,format=raw,cache=writeback:
> > Could not open 'disk1-firewall2': No such file or directory
> This is wrong, file= should point to some block device created by the
> drbd hotplug script.
Does this mean there is an issue with the handoff from Xen to qemu while 
building whichever command is required to start the HVM guest. I have 
used the hotplug script in the documented method which works for PV 
guests. Are you saying that using the following format
disk=['drbd:disk1-firewall2,hda,w']

Is not the correct way for HVM guests? I have confirmed, that using the 
actual dev mount works correctly 
(disk=['phy:/dev/vg/disk1-firewall2_00,hda,w']), but this will not live 
migrate, or even start on any node besides the last node that was primary.
> > Contents of hotplug log
> >
> > disk1-firewall2: State change failed: (-12) Device is held open by someone
> > Command 'drbdsetup secondary disk1-firewall2' terminated with exit code 11
> I'm afraid this is due to block hotplug scripts being run twice for
> HVM guests, once for the PV disk and another time for the emulated
> disk.
This could be a red herring, the row show here may be from previous 
attempts to start the guest. Subsequent creates have not seemed to 
create this line. I have deleted the hotplug.log and rerun, and the file 
is still empty, I have restarted the xen service, in case a new file 
handle was required, but it still is empty after a few failed create 
attempts.

> Could you please paste the full output of `xl -vvv create ...`, that
> should contain all the calls to the hotplug scripts.
Parsing config from firewall2.cfg
libxl: debug: libxl_create.c:1563:do_domain_create: ao 0x1462fb0: 
create: how=(nil) callback=(nil) poller=0x1462960
libxl: debug: libxl_device.c:337:libxl__device_disk_set_backend: Disk 
vdev=hda spec.backend=unknown
libxl: debug: libxl_device.c:260:disk_try_backend: Disk vdev=hda, uses 
script=... assuming phy backend
libxl: debug: libxl_device.c:366:libxl__device_disk_set_backend: Disk 
vdev=hda, using backend phy
libxl: debug: libxl_create.c:947:initiate_domain_create: running bootloader
libxl: debug: libxl_bootloader.c:324:libxl__bootloader_run: not a PV 
domain, skipping bootloader
libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch 
w=0x1460808: deregister unregistered
xc: detail: elf_parse_binary: phdr: paddr=0x100000 memsz=0x5b844
xc: detail: elf_parse_binary: memory: 0x100000 -> 0x15b844
xc: detail: VIRTUAL MEMORY ARRANGEMENT:
xc: detail:   Loader:   0000000000100000->000000000015b844
xc: detail:   Modules:  0000000000000000->0000000000000000
xc: detail:   TOTAL:    0000000000000000->000000001f800000
xc: detail:   ENTRY:    0000000000100630
xc: detail: PHYSICAL MEMORY ALLOCATION:
xc: detail:   4KB PAGES: 0x0000000000000200
xc: detail:   2MB PAGES: 0x00000000000000fb
xc: detail:   1GB PAGES: 0x0000000000000000
xc: detail: elf_load_binary: phdr 0 at 0x7f53733d5000 -> 0x7f5373426dd1
domainbuilder: detail: xc_dom_gnttab_hvm_seed: called, pfn=0xff000
libxl: debug: libxl_device.c:337:libxl__device_disk_set_backend: Disk 
vdev=hda spec.backend=phy
libxl: debug: libxl_device.c:260:disk_try_backend: Disk vdev=hda, uses 
script=... assuming phy backend
libxl: debug: libxl_event.c:639:libxl__ev_xswatch_register: watch 
w=0x1463b60 wpath=/local/domain/0/backend/vbd/85/768/state token=3/0: 
register slotnum=3
libxl: debug: libxl_create.c:1586:do_domain_create: ao 0x1462fb0: 
inprogress: poller=0x1462960, flags=i
libxl: debug: libxl_event.c:576:watchfd_callback: watch w=0x1463b60 
wpath=/local/domain/0/backend/vbd/85/768/state token=3/0: event 
epath=/local/domain/0/backend/vbd/85/768/state
libxl: debug: libxl_event.c:884:devstate_callback: backend 
/local/domain/0/backend/vbd/85/768/state wanted state 2 still waiting 
state 1
libxl: debug: libxl_event.c:576:watchfd_callback: watch w=0x1463b60 
wpath=/local/domain/0/backend/vbd/85/768/state token=3/0: event 
epath=/local/domain/0/backend/vbd/85/768/state
libxl: debug: libxl_event.c:880:devstate_callback: backend 
/local/domain/0/backend/vbd/85/768/state wanted state 2 ok
libxl: debug: libxl_event.c:677:libxl__ev_xswatch_deregister: watch 
w=0x1463b60 wpath=/local/domain/0/backend/vbd/85/768/state token=3/0: 
deregister slotnum=3
libxl: debug: libxl_device.c:991:device_backend_callback: calling 
device_backend_cleanup
libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch 
w=0x1463b60: deregister unregistered
libxl: debug: libxl_linux.c:229:libxl__hotplug_disk: Args and 
environment ready
libxl: debug: libxl_device.c:1088:device_hotplug: calling hotplug 
script: /etc/xen/scripts/block-drbd add
libxl: debug: libxl_aoutils.c:593:libxl__async_exec_start: forking to 
execute: /etc/xen/scripts/block-drbd add
libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch 
w=0x1463c60: deregister unregistered
libxl: debug: libxl_linux.c:246:libxl__get_hotplug_script_info: num_exec 
1, not running hotplug scripts
libxl: debug: libxl_device.c:1075:device_hotplug: No hotplug script to 
execute
libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch 
w=0x1463c60: deregister unregistered
libxl: debug: libxl_dm.c:1796:libxl__spawn_local_dm: Spawning 
device-model /usr/bin/qemu-system-i386 with arguments:
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm: 
/usr/bin/qemu-system-i386
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -xen-domid
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   85
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -chardev
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm: 
socket,id=libxl-cmd,path=/var/run/xen/qmp-libxl-85,server,nowait
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -no-shutdown
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -mon
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm: 
chardev=libxl-cmd,mode=control
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -chardev
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm: 
socket,id=libxenstat-cmd,path=/var/run/xen/qmp-libxenstat-85,server,nowait
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -mon
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm: 
chardev=libxenstat-cmd,mode=control
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -nodefaults
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -name
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   firewall2
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -vnc
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   :0,to=99
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -display
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   none
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -serial
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   pty
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -device
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm: 
cirrus-vga,vgamem_mb=8
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -boot
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   order=c
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -net
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   none
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -machine
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   xenfv
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -m
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   504
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm:   -drive
libxl: debug: libxl_dm.c:1798:libxl__spawn_local_dm: 
file=disk1-firewall2,if=ide,index=0,media=disk,format=raw,cache=writeback
libxl: debug: libxl_dm.c:1800:libxl__spawn_local_dm: Spawning 
device-model /usr/bin/qemu-system-i386 with additional environment:
libxl: debug: libxl_dm.c:1802:libxl__spawn_local_dm: 
XEN_QEMU_CONSOLE_LIMIT=1048576
libxl: debug: libxl_event.c:639:libxl__ev_xswatch_register: watch 
w=0x1460b00 wpath=/local/domain/0/device-model/85/state token=3/1: 
register slotnum=3
libxl: debug: libxl_event.c:576:watchfd_callback: watch w=0x1460b00 
wpath=/local/domain/0/device-model/85/state token=3/1: event 
epath=/local/domain/0/device-model/85/state
libxl: debug: libxl_exec.c:399:spawn_watch_event: domain 85 device 
model: spawn watch p=(null)
libxl: debug: libxl_event.c:677:libxl__ev_xswatch_deregister: watch 
w=0x1460b00 wpath=/local/domain/0/device-model/85/state token=3/1: 
deregister slotnum=3
libxl: error: libxl_dm.c:1872:device_model_spawn_outcome: domain 85 
device model: spawn failed (rc=-3)
libxl: error: libxl_create.c:1344:domcreate_devmodel_started: device 
model did not start: -3
libxl: error: libxl_dm.c:1987:kill_device_model: Device Model already exited
libxl: debug: libxl_event.c:639:libxl__ev_xswatch_register: watch 
w=0x1467550 wpath=/local/domain/0/backend/vbd/85/768/state token=3/2: 
register slotnum=3
libxl: debug: libxl_event.c:576:watchfd_callback: watch w=0x1467550 
wpath=/local/domain/0/backend/vbd/85/768/state token=3/2: event 
epath=/local/domain/0/backend/vbd/85/768/state
libxl: debug: libxl_event.c:880:devstate_callback: backend 
/local/domain/0/backend/vbd/85/768/state wanted state 6 ok
libxl: debug: libxl_event.c:677:libxl__ev_xswatch_deregister: watch 
w=0x1467550 wpath=/local/domain/0/backend/vbd/85/768/state token=3/2: 
deregister slotnum=3
libxl: debug: libxl_device.c:991:device_backend_callback: calling 
device_backend_cleanup
libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch 
w=0x1467550: deregister unregistered
libxl: debug: libxl_linux.c:229:libxl__hotplug_disk: Args and 
environment ready
libxl: debug: libxl_device.c:1088:device_hotplug: calling hotplug 
script: /etc/xen/scripts/block-drbd remove
libxl: debug: libxl_aoutils.c:593:libxl__async_exec_start: forking to 
execute: /etc/xen/scripts/block-drbd remove
libxl: debug: libxl_event.c:545:watchfd_callback: watch 
epath=/local/domain/0/backend/vbd/85/768/state token=3/2: empty slot
libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch 
w=0x1467650: deregister unregistered
libxl: debug: libxl_linux.c:246:libxl__get_hotplug_script_info: num_exec 
1, not running hotplug scripts
libxl: debug: libxl_device.c:1075:device_hotplug: No hotplug script to 
execute
libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch 
w=0x1467650: deregister unregistered
libxl: debug: libxl_linux.c:267:libxl__get_hotplug_script_info: 
backend_kind 6, no need to execute scripts
libxl: debug: libxl_device.c:1075:device_hotplug: No hotplug script to 
execute
libxl: debug: libxl_event.c:691:libxl__ev_xswatch_deregister: watch 
w=0x1467ca0: deregister unregistered
libxl: debug: libxl.c:1738:devices_destroy_cb: forked pid 26093 for 
destroy of domain 85
libxl: debug: libxl_event.c:1874:libxl__ao_complete: ao 0x1462fb0: 
complete, rc=-3
libxl: debug: libxl_event.c:1843:libxl__ao__destroy: ao 0x1462fb0: destroy
libxl: debug: libxl.c:1477:libxl_domain_destroy: ao 0x1462250: create: 
how=(nil) callback=(nil) poller=0x1462960
libxl: error: libxl.c:1610:libxl__destroy_domid: non-existant domain 85
libxl: error: libxl.c:1568:domain_destroy_callback: unable to destroy 
guest with domid 85
libxl: error: libxl.c:1495:domain_destroy_cb: destruction of domain 85 
failed
libxl: debug: libxl_event.c:1874:libxl__ao_complete: ao 0x1462250: 
complete, rc=-21
libxl: debug: libxl.c:1486:libxl_domain_destroy: ao 0x1462250: 
inprogress: poller=0x1462960, flags=ic
libxl: debug: libxl_event.c:1843:libxl__ao__destroy: ao 0x1462250: destroy
xc: debug: hypercall buffer: total allocations:648 total releases:648
xc: debug: hypercall buffer: current allocations:0 maximum allocations:3
xc: debug: hypercall buffer: cache current size:3
xc: debug: hypercall buffer: cache hits:632 misses:3 toobig:13



> > And the guest is not started
> >
> > If i try to attach to dom 0 :
> >
> > xl block-list 0
> > Vdev  BE  handle state evt-ch ring-ref BE-path
> >
> > xl block-attach 0 'drbd:disk1-firewall2,hda,w'      (no errors or output)
> >
> > xl block-list 0
> >
> > Vdev  BE  handle state evt-ch ring-ref BE-path
> > 768   0   0      4     102    8        /local/domain/0/backend/vbd/0/768
> >
> > xl block-detach 0 768   (no errors or output)
> >
> > xl block-list 0
> >
> > Vdev  BE  handle state evt-ch ring-ref BE-path
> >
> > So it mounts and unmounts fine manually, but not via the hotplug script?
> When you attach to Dom0 you are also using the hotplug script.
OK, which is possibly a useless test then, as the hotplug script then 
works as expected for PV, and when run manually via block-attach, so I 
think this confirms the script in fact works, just not in the HVM case.
_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-users