[Xen-devel] [RFC PATCH v2 00/17] Add support for qemu-xen runnning in a Linux-based stubdomain.

[Marek Marczykowski-Górecki <marmarek@xxxxxxxxxxxxxxxxxxxxxx>]

General idea is to allow freely set device_model_version and
device_model_stubdomain_override and choose the right options based on this 
choice.
Also, allow to specific path to stubdomain kernel/ramdisk, for greater 
flexibility.
Right now when qemu-xen in stubdomain is selected, it is assumed it's
Linux-based and few decisions are based on it, specifically:
 - qemu args encoding (\x1b as separator, to allow spaces in arguments)
 - save/restore stream access (specific FDs passed to qemu by a wrapper script)
 - QMP access, if any

It may be a good idea to document "stubdomain API" somewhere. Is there such
document for MiniOS one? Here is an initial version for Linux one:

    Assumptions about Linux-based stubdomain for qemu-xen:
    * qemu command line is stored by toolstack in
      /vm/<target-uuid>/image/dmargs xenstore entry, arguments are separated
      with \x1b character
    * qemu can access saved state (if any) at its FD 3
    * qemu can write its state (for save/migration) to its FD 4
    * qemu expects backend for serial console at /dev/hvc3
    * disks configured for the target are available as /dev/xvd*, in
      configuration order
    * qemu can call /etc/qemu-ifup and /etc/qemu-ifdown to connect/disconnect
      network interface to appropriate network

Initial version has no QMP support - in initial patches it is completely
disabled, which means no suspend/restore. QMP normally would be used for PCI
passthrough setup, but it is worked around with MiniOS-like control protocol
over xenstore, which then is translated to QMP on stubdomain side.
Some option is to use separate console for that, but that require
xenstoled supporting multiple consoles per domain (the goal is to not have qemu
in dom0 at all). Also, it would be preferable to evaluate how libxl
handle potentially malicious QMP responses.
Another option is to use xenstore - either translate everything needed to
MiniOS-like thing, or write already json-formatted requests to xenstore and
watch there for responses. When using separate xenstore dir for that, with
per-command entries (command id as a key name?) that would solve concurrency
problem.

QMP support over separate console: patch "libxl: access QMP socket via console
for qemu-in-stubdomain" implements some early PoC of this.
Major limitation: only one connection at a time and no means of out of
band reset (and re-negotiate). I've tried adding very simple `qmp_reset`
command for in-band connection reset, but it is unreliable because of the
first limitation - xl process running in background hold this connection open
and every other xl instance interfere with it. On the other hand, for libvirt
use case one connection could be enough (leaving alone libvirtd restart).

Xenconsoled patches add support for multiple consoles in xenconsoled, to avoid 
the need
for qemu in dom0 for this to work. Multiple consoles for a stubdomain are used 
for:
 - logs (console 0)
 - save + restore (console 1, 2)
 - qmp (console 3)
 - serial terminal to target domU (console 4)
Xenconsoled patches are in fact a separate series, but I'm sending them here to
ease dependencies handling (latter libxl patches use that).

What qmp-libxenstat socket is for?

PCI passthrough support require some more love. Right now, libxl tries to setup
pcifront for both target HVM and stubdomain and the former fails (race
condition):
    xen-pciback pci-259-0: 22 Couldn't locate PCI device (0000:00:1b.0)! 
perhaps already in-use?
Fortunately it isn't needed. There is also a PCI related problem on
domain shutdown - it looks like first stubdomain is paused and then libxl waits
for pcifront there.
Also, MSI doesn't work (qemu output):

    [00:05.0] xen_pt_msgctrl_reg_write: setup MSI (register: 81).
    [00:05.0] msi_msix_setup: requested pirq 55 for MSI (vec: 0, entry: 0)
    [00:05.0] msi_msix_setup: Error: Mapping of MSI (err: 1, vec: 0, entry 0)
    [00:05.0] xen_pt_msgctrl_reg_write: Warning: Can not map MSI (register: 80)!
    [00:05.0] Write-back to unknown field 0x44 (partially) inhibited (0x00)
    [00:05.0] If the device doesn't work, try enabling permissive mode
    [00:05.0] (unsafe) and if it helps report the problem to xen-devel

The actual stubdomain implementation is here:

    https://github.com/marmarek/qubes-vmm-xen-stubdom-linux (branch 
for-upstream)

See readme there for build instructions.

Remaining parts for eliminating dom0's instance of qemu:
 - do not force QDISK backend for CDROM

This patch series is third (fourth?) attempt to get rid of limitation
"if you want to use stubdomain, you're stuck with qemu-traditional", done over
years, by many people.
I think it should be acceptable plan to gradually add features to
qemu-xen+stubdomain configuration - not necessary waiting with committing any
of those patches until full feature set of qemu-xen is supported. After all,
right now "feature set supported by qemu-xen+stubdom" is empty, so wouldn't be
worse.

Changes in v2:
 - apply review comments by Jason Andryuk

Cc: Simon Gaiser <simon@xxxxxxxxxxxxxxxxxxxxxx>
Cc: Eric Shelton <eshelton@xxxxxxxxx>
Cc: Ian Jackson <ian.jackson@xxxxxxxxxxxxx>
Cc: Wei Liu <wei.liu2@xxxxxxxxxx>

Eric Shelton (3):
  libxl: Add "stubdomain_version" to domain_build_info.
  libxl: Handle Linux stubdomain specific QEMU options.
  libxl: Build the domain with a Linux based stubdomain

Marek Marczykowski-Górecki (13):
  libxl: fix qemu-trad cmdline for no sdl/vnc case
  libxl: create vkb device only for guests with graphics output
  libxl: add save/restore support for qemu-xen in stubdomain
  xl: add stubdomain related options to xl config parser
  libxl: use \x1b to separate qemu arguments for linux stubdomain
  xenconsoled: install xenstore watch for all supported consoles
  xenconsoled: add support for consoles using 'state' xenstore entry
  xenconsoled: make console_type->use_gnttab less confusing
  xenconsoled: add support for up to 3 secondary consoles
  xenconsoled: deduplicate error handling
  xenconsoled: add support for non-pty output
  libxl: access QMP socket via console for qemu-in-stubdomain
  libxl: use xenconsoled even for multiple stubdomain's consoles

Simon Gaiser (1):
  libxl: use xenstore for pci hotplug qemu-in-linux-stubdom commands

 docs/man/xl.cfg.pod.5.in        |  23 ++-
 tools/console/daemon/io.c       | 222 ++++++++++++++++++++++++++++-----
 tools/libxl/libxl_create.c      |  83 ++++++++++--
 tools/libxl/libxl_dm.c          | 190 +++++++++++++++++++---------
 tools/libxl/libxl_dom_suspend.c |  10 +-
 tools/libxl/libxl_internal.c    |  22 +++-
 tools/libxl/libxl_internal.h    |   9 +-
 tools/libxl/libxl_mem.c         |   6 +-
 tools/libxl/libxl_pci.c         |  22 ++-
 tools/libxl/libxl_qmp.c         |  52 +++++++-
 tools/libxl/libxl_types.idl     |  10 +-
 tools/xl/xl_parse.c             |   7 +-
 12 files changed, 546 insertions(+), 110 deletions(-)

base-commit: e752f28409678ce3ade49986b39309178fb2a6d6
-- 
git-series 0.9.1

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel