[Xen-devel] [PATCH v7 RFC 0/2] libxl USB prototype and design discussion

[George Dunlap <george.dunlap@xxxxxxxxxxxxx>]

Hey all!  

So here I finally have a rebased the HVM USB hotplug series from last
year.  I went through and addressed all of IanC's technical comments
that I could; so it should be in much better shape than it was.

However, one of the unfinished threads of the conversation at that
time was about the interface.  In particular since Simon Bo is now
working on the PVUSB side of things, I thought it would be good to
review the situation so we can get an interface we're happy with.

In order to really answer the questions, I went through and looked at
the underlying capabilities of QEMU and PVUSB, at the existing libxl
and xend interfaces for those two, and also at the libvirt interface.

I summarize what I've found here; but if you want a TLDR version, just
skip to the bottom, where it says "Draft Design Proposal".  But before
asking any questions or making any criticisms, please go back and read
the appropriate summary section.

References are at the very bottom of the e-mail.

This can be found at the following address:

git://xenbits.xen.org/people/gdunlap/xen.git out/hvm-usb-hotplug-v7-RFC

== Capabilities of QEMU ==

Qemu has the additional ability to not only pass through host devices,
but other kinds of devices -- tablets, mice, keyboards, drives, hubs,
and so on.  As far as I know, PVUSB can only do host device
pass-through at the moment.  (Although you could certainly imagine
that changing; if qemu was taught how to become a PVUSB back-end, for
example.)

qemu has two ways of specifying usb devices.  The "legacy" method can
be specified on the qemu command-line or via the qemu monitor.  It is
limited to USB 1.1.  The new "qdevice" method can be used either on
the command-line or via qmp.

qemu-traditional only supports the legacy method.  qemu-xen still
supports the legacy method via command-line, but we should try to
start using the qdevice method if we can.

qemu-xen has the ability to create virtual USB 2.0 or 3.0 controllers.
These can be named, and it is possible to specify (to a certain level)
which controller to plug a device into.  These can be created either
via the command-line, or hot-plugged via qmp.

qemu also seems to be willing to shift things around behind the scenes
to be able to handle your request; shifting around the USB topology,
for instance.  It will also do "helpful" things; for instance, if you
specify that you want to plug a device into a USB 2.0 controller, and
the device is only 1.1 capable, it will ignore what you asked and plug
it into the 1.1 controller.


== Capabilities of PVUSB ==

PVUSB also requires you to first create a virtual controller, before
attaching host devices to it.  There is some flexibility in how to
create these, and you can create more than one and specify which
virtual bus to attach the host device.  You seem to be able to specify
the USB version number when you specify the controller as well.  You
can also specify the number of ports for a device; I'm not sure what
the maximum number of ports per controller is, or why you would ever
really want to have less than the maximum number of ports.

It looks like in the xm interface, when you create a virtual
controller it is assigned an index, starting at 0; and this is the
index that is used when specifying where to plug in a device.

(Simon, can you please correct this if I'm wrong, and add anything
important that I missed?)


== libxl/xl interface ==

At the moment, libxl/xl only support USB at domain creation time.

For HVM guests, we have two incompatible sets of options.  usb=1 and
usbdevice=[] use the "old-style" qemu interface on the qemu
command-line.  The "old-style" interface can only be used to specify
USB 1.1 devices.  We also have "usbversion=[foo]", which uses the
"new-style" device specification commands.  These new specification
commands *can* be specified either on the qemu command-line or via
qmp; but at the moment libxl specifies them on the command-line.

The patch series attached specifes using the new-style interface via
qmp.  (This seems to work properly with the various controllers no
matter how they were created.)  In theory, we should be able to attach
these devices during domain creation, after qemu has been started but
before the guest is running.

Obviously, we would ideally like for the user not to have to worry
about a lot of this complexity, and just say, "Can you pass this
device through to the guest?  Thanks."

Another thing to consider in the design space is config file and
start-up behavior.


== Libvirt's interface ==

I've had a brief look at libvirt's USB interface, and learned a bit
about libvirt's general approach to things at the Xen Hackathon last
week.  One of the goals of libvirt is to be able to specify the
virtual hardware in enough detail to keep it from changing when you
upgrade the hypervisor, so that certain proprietary operating systems
which are sensitive to this kind of thing continue to work.

Instead of specifying a controller name, you specify a controller
index (which is different than qemu).  But instead of specifying a USB
version number, you specify a model for the USB controller (which
happens to be the exact name that qemu uses): "piix3-uhci",
"piix4-uhci", "ehci", "ich9-ehci1", "ich9-uhci1", "ich9-uhci2",
"ich9-uhci3", "vt82c686b-uhci", "pci-ohci" or "nec-xhci".

When specifying a USB device, libvirt has the concept of an "address"
where you will plug it into.  Here is what the page says about it:  

"USB addresses have the following additional attributes: bus (a hex
value between 0 and 0xfff, inclusive), and port (a dotted notation of
up to four octets, such as 1.2 or 2.1.3.1)."

It's not exactly clear to me what those numbers mean.  But after
chatting with Daniel Berrange at the Hackathon, it looks like the
"bus" in the address corresponds to the "index" in the controller
specification.  It also appears that this "index" is internal to
libvirt and is not exposed to the guest: so it should in theory be
possible to have index 0 be a Xen PVUSB controller, 1 be an emulated
qemu controller, 3 be an emulated PVUSB controller, and so on.


== Open questions ==

Those are things I think I know; there are a couple of pertinent
factual questions which I'm not sure of:

* Is it possible to specify PVUSB controllers and attach USB devices
to them before the guest is up and running (i.e., before the frontend
is connected)?  It looks like xend had a syntax for specifying virtual
controllers and attaching virtual devices, so it seems like it should
be possible.

* Is it possible to connect a USB 1.1 device to a PVUSB controller
which has been specified 2.0, or would there have to be a separate
virtual controller for each?

* Is it possible for the toolstack to tell if dom0 (or whatever the
specified backend domain) has PVUSB support before starting the guest?

* Is it possible for the toolstack to tell if domU has a working and
connected PVUSB front-end?

* Do we want to be able to create virtual hubs for qemu-backed
controllers at some point in the future?  Is there any groundwork we
want to lay for that?


== Design questions ==

Then based on that, we have several design questions.

* How much of the "controller" thing should be exposed via libxl?  Via xl?

* This series only allows you to specify the "protocol", either PV or
DEVICE_MODEL.  Would it be better, for instance, to get rid of that,
and instead allow the user to specify the creation of USB controllers,
allow them to have a type of "HCI (or emulated)" or "PV", and allow
the user to specify which controller to attach a specific device to?

* How much "smarts" should we have in the libxl / xl about creating
emulated/virtual controllers and of what kinds?

* How much / what kind of "smarts" should be in libxl / xl about
choosing a controller to plug a device into?

* What about config file syntax?  Should we try to reuse / extend the
current config syntax, or create a new one?  Should the new one allow
the specification of controllers?  Should it perhaps *require* the
specification of controllers?


== Draft design proposal ==

I've given it some thought, and based on the below is a suggested
design for people to have a go at.

Basic idea: Specify named controllers.  It's controllers that are PV
or emulated, and may have a backend domain.  When adding devices,
specify which controller to attach it to.  Allow most things to have
intelligent defaults.

* libxl functions

usb_controller_add
usb_controller_remove/destroy
usb_controller_list
device struct:
  name # If empty, default to hciN/pvN, where N starts at 0
  type = {PV, EMULATED, AUTO}
  backend_{domname,domid} # PV only
  usbversion = {1, 2, 3}
  numports # default 16

If type==AUTO, then it will be PV for a PV domain, EMULATED for an HVM domain.

usb_add
usb_remove/destroy
usb_list
device struct:
  controller_name # If empty, choose one of the controllers.
  type
    hostdev
      hostbus,hostaddr

Note: I've removed backend from the device struct, as that will be
based on the controller.

* Storing information about virtual USB controllers / devices

qemu does not at the moment have a way to query for plugged-in
devices.  I'm not sure if PVUSB does.  Store information about both
USB controllers and USB devices created with libxl in xenstore,
somewhat similar to the system in the attached patches.

* Domain creation

I think what we add to the domain creation libxl calls will be obvious
from how we design the config file interface.

For reference, here are some example config snippets from the current
xl / xm config files:

-- snip --
# HVM USB
usb=1
usbdevice=['tablet','host:4.3']

# HVM USB, not compatible with the above
usbversion=3

# xend's PVUSB config file; this creates one virtual controller, then
# plugs hostdev 1.8 into port 1
vusb=['usbver=2, numports=4, port_1=1-8']
-- snip --

Given that, here is a potential config file format:

-- snip --
# Create two controllers, one pv, one emulated
usbcontroller=['type=pv,name=pv0,usbversion=2,numports=4',
               'type=emul,name=hci0,usbversion=2']

# Create a controller with the defaults; PV for PV guests, emul for HVM guests
usbcontroller=[''] 

# Same as above, but defaulting to version 2
usbversion=2 

# I think we should be able to automatically detect which format to
# use; so I think we should re-use usbdevice.  I could be convinced otherwise.
usbdevice=['type=tablet','type=hostdev,hostaddr=4.3,bus=pv0']
-- snip --


Other ideas: 

* To make the interface closer to libvirt's, instead of specifying PV
/ EMULATED and usbversion, just specify a model.  Then create
"pvusb-v1", "pvusb-v2", &c for PVUSB hubs with the various versions,
and detect automatically whether to use the PV or the qemu path.  (NB
that at the libvirt level, to fit with their syntax, we'd probably end
up creating a "xenpvusbN" model in libvirt that would translate down
to whatever the appropriate thing is in libxl.)

* Rather than having to specify a controller, automatically hot-plug
controllers as-needed.

* Include an optional port number into which we will plug the USB
device.

Thoughts?

 -George

== References ==

Last time this was posted:

http://thread.gmane.org/gmane.comp.emulators.xen.devel/156796

PVUSB references:

http://doc.opensuse.org/products/draft/SLES/SLES-xen_sd_draft/cha.xen.config.html#sec.xen.config.pvusb

Libvirt references:

http://wiki.libvirt.org/page/QEMUSwitchToLibvirt#-usbdevice
https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Administration_Guide/sect-Attaching_and_updating_a_device_with_virsh.html
http://libvirt.org/formatdomain.html
http://libvirt.org/formatdomain.html#elementsControllers

CC: Ian Jackson <ian.jackson@xxxxxxxxxx>
CC: Roger Pau Monne <roger.pau@xxxxxxxxxx>
CC: sstanisi@xxxxxxxxx
CC: Fabio Fantoni <fabio.fantoni@xxxxxxx>
CC: Ian Campbell <ian.campbell@xxxxxxxxxx>
CC: Anthony Perard <anthony.perard@xxxxxxxxxx>
CC: Simon (Bo) Cao <caobosimon@xxxxxxxxx>
CC: LibVirt Development List <libvir-list@xxxxxxxxxx>
CC: Jim Fehlig <jfehlig@xxxxxxxx>
CC: Daniel P. Berrange <berrange@xxxxxxxxxx>


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel