Re: [Xen-devel] [PATCH] docs: document hotplug script protocol

[Ian Campbell <Ian.Campbell@xxxxxxxxxx>]

On Mon, 2012-07-30 at 17:56 +0100, Ian Jackson wrote:

> diff --git a/docs/misc/xl-scripts-block.txt b/docs/misc/xl-scripts-block.txt
> new file mode 100644
> index 0000000..d6a5123
> --- /dev/null
> +++ b/docs/misc/xl-scripts-block.txt

It'd be nice to use markdown.

> @@ -0,0 +1,246 @@
> +                    -----------------------
> +                    DEVICE SCRIPT INTERFACE
> +                    -----------------------
> +
> +Guest devices can be specified in [lib]xl to be provided with the help
> +of a "script".
> +
> +This document describes the interface between the toolstack and device
> +scripts.  This protocol was partially reverse-engineered from the xend
> +infrastructure during the development of libxl; the interface is
> +intended to be compatible between libxl and xend.  This document
> +describes the protocol implemented by libxl.
> +
> +This description is applicable to Linux.
> +
> +Only block and network devices support these scripts.
> +
> +Note that we are consider deprecating this protocol after the release

                    considering

> +of Xen 4.2, or providing a more sophisticated replacement for it.

I'm not a huge fan of these sorts of TODO list statement scattered
around docs, they quickly become out of date or irrelevant.

> +
> +
> +=============
> +BLOCK SCRIPTS
> +=============
> +
> +When a script is specifed, the <target> value from the

                   specified

> +domain configuration (the pdev_path value in the libxl API) is not
> +interpreted by libxl.  Instead, it is passed as an opaque parameter to
> +the script.
> +
> +The block is responsible for taking the pdev_path (<target> from the
> +xl configuration), henceforth known as the "params", and providing a
> +block device in the toolstack domain.

Do you mean backend domain here?

> +
> +Block scripts are optional; if not specified, the pdev_path is used as
> +the path of the intended block device or file in the driver domain.
> +
> +
> +
> +NETWORK SCRIPTS
> +===============
> +
> +When a guest's networking is being set up, it appears in the driver
> +domain as a virtual network device (a "vif" or "tap" device).
> +
> +The network script is responsible for plumbing this virtual network
> +device through to the driver domain's networking arrangements.  For
> +example, enslaving the virtual device to a suitable bridge, creating
> +routes to it, or whatever is necessary.
> +
> +The script is also responsible for renaming the interface as may be
> +desired.

Should we mention here and in the block section that the script is
responsible for tearing stuff down as well as setting it up?

> +=====
> +MODEL
> +=====
> +
> +We will define a "connection" which is the making available or a
> +single block or network device, relating to a single guest, to a
> +single facility (in the guest or toolstack or service domain).
> +
> +It may be necessary to make multiple connections for one guest device;
> +even multiple connections which overlap in time.
> +
> +Such connection is created by an "add" execution of the script and
> +destroyed by a "remove" execution.  The script is invoked in the
> +driver domain (usually dom0).
> +
> +
> +=========
> +EXECUTION
> +=========
> +
> +Environment
> +-----------
> +
> +The scripts are invoked with the following environment variables:
> +
> + script=<value specified in domain configuration>
> +
> +     This may or may not be a fully qualified path.  Scripts should
> +     not use this information as the form is not reliable.

Looking at the kernel I think this is only supplied by netback in the
traditional udev based model. libxl does indeed supply it in both cases
when it runs the hotplug scripts.

Only the vif-setup script currently makes use of it. AFAIK vif-setup is
only called from the udev based mechanism to shell out to $script, libxl
will call $script itself directly.

We could therefore declare this variable to be internal to the udev
based hotplug script infrastructure and not for use by "users". 

> + XENBUS_TYPE=vbd    [block devices]
> + XENBUS_TYPE=vif    [network devices]
> +
> + XENBUS_PATH=<backend>
> +
> +     This is a path in xenstore which is used extensively to
> +     communicate between the script and the rest of the backend
> +     system.  This may be a relative or absolute xenstore path.
> +
> +     See below for the contents and usage of this value.
> +
> + XENBUS_BASE_PATH=backend
> +
> +     Do not use this value.
> +
> + vif=<devname>             [network vif devices]
> + INTERFACE=<devname>       [network tap devices]
> +
> +     Current name of the virtual device in the toolstack domain.

"Initial name" perhaps, since strictly speaking someone might have
already renamed it (although the scripts are a bi stuffed if someone
does)

> +
> +
> +Arguments
> +---------
> +
> +Block scripts:
> +   The script will get one argument, "add" or "remove"
> +
> +Network scripts:
> +   The script will get two arguments.  They vary according to
> +   whether the script is running with respect to a vif or
> +   a tap virtual device:
> +
> +                arguments for add        arguments for remove
> +   vif device   "online" "type_if=vif"   "offline" "type_if=vif"
> +   tap device   "add" "type_if=tap"      "remove" "type_if=tap"

True, sigh.

> +Exit status and error handling
> +------------------------------
> +
> +The script should exit zero if it is successful.  If it fails it
> +should clean everything up, try write an error message to the
> +hotplug-error path (see below), and and exit 1 or >=126 or die from a
> +signal.
> +
> +Failure of "remove" scripts will be logged and reported but there are
> +no arrangements to re-invoke a failed script.

Is it worth referencing the helper libraries which we supply
(xen-hotplug-common.sh, {block,vif}-common.sh etc).

Although it's true that scripts can just do all this themselves it would
be more consistent and likely to be done correctly if people would use
the supplied functions.

> +
> +
> +Locking, timeouts, concurrency
> +------------------------------
> +
> +Scripts should not block indefinitely.  If a script takes longer than
> +30s to exit it will be sent a SIGKILL which will result in the
> +operation being considered failed.

30s? LIBXL_{INIT,DESTROY,HOTPLUG}_TIMEOUT are all 10s (I'm not sure
which if any actually applies here).

It's also a bit toolstack specific, IIRC xapi has a much longer timeout
(it also has its own hotplug scripts but eventually I suppose we'd like
to converge)

> +
> +The scripts may be invoked concurrently for the same or different
> +resources, and the same or different guests, and must do any necessary
> +locking.

Again point to the helpers in locking.h and *-common.sh?

> +
> +
> +
> +========
> +XENSTORE
> +========
> +
> +The <backend> directory in xenstore contains keys and values used to
> +communicate between the script and the rest of the infrastructure.
> +
> +The contents of this path are shared between the script and other
> +parts of the infrastructure.  The script should therefore not write
> +arbitrary subpaths for its own purposes.
> +
> +The following paths are defined:
> +
> +
> +All devices
> +-----------
> +
> + <backend>/hotplug-error
> +
> +     If a script is about to fail, it may write an error message
> +     string to this path.  The string will be reported to the user.
> +
> +
> +Block devices
> +-------------
> +
> + <backend>/params
> +
> +     The params.  Ie, the <target> string from the configuration file,
> +     which corresponds to the "pdev_path" in the libxl API.
> +
> +     This is not interpreted by the toolstack and the script may
> +     define whatever syntax it likes for it.  Newlines, whitespace,
> +     backslashes, quotes, and so forth, should be avoided as
> +     specifying them from a config file may be difficult.  Consult the
> +     xl configuration API documentation for information about the
> +     exact syntactic transformations in xl.

Reference xl-disk-configuration specifically?

> +
> +     The value is written by the toolstack before the add script is
> +     run and remains present until the remove script has finished.  It
> +     should not be modified by the script.
> +  
> + <backend>/mode = "r" | "w"
> +
> +     The device is to be readonly or read/write, respectively.

Maybe "read-only"? (perhaps it is alloneword, I'm not sure)

> +     Written by the toolstack; should not be modified.
> +
> + <backend>/physical-device
> +
> +     The main result of the add script.  This key does not exist on
> +     entry to add.  The add script should, if successful, write this
> +     key with the major and minor number of the block device
> +     (as accessible in the driver domain) in the format "%x:%x".
> +
> +
> +Other information that may be useful to the script is also present.
> +The block script does not need to pay attention to this if it doesn't
> +want to.  These keys include:
> +
> + <backend>/removable = "0" | "1"       [block devices]
> +
> +     Indicates whether the device is being presented to the guest as a
> +     removable device.
> +
> + <backend>/device-type = "disk" | "cdrom"       [block devices]
> +
> +
> +Network devices
> +---------------
> +
> +All of these values are provided by the toolstack to the script and
> +should not be changed by the script:
> +
> + <backend>/vifname      [network devices]

I guess "[network devices]" (and "[block devices]" above) predate you
splitting them into their own sections?

> +
> +     Name that the interface should be renamed to.
> +
> + <backend>/mac
> +
> +     mac parameter from the libxl API.
> +     Ethernet address which is to be used by the guest.
> +
> + <backend>/ip
> +
> +     ip parameter from the libxl API.
> +     IP address which is to be used by the guest.
> +
> + <backend>/bridge
> +
> +     bridge parameter from the libxl API.
> +     Bridge to which the network should be enslaved (not interpreted
> +     by the libxl toolstack).

I suppose "ip" and "bridge" are something like the block scripts'
"params" node, in so much as we could consider making them more
explicitly generic and opaque to the toolstack in the future.

In practice at the moment bridge is the parameter for vif-bridge and ip
is the parameter for vif-{route,nat}. Or something like that anyway.

> +
> + <backend>/rate = "%"PRIu64",%"PRIu32""
> +
> +     rate_bytes_per_interval and rate_interval_usecs parameters
> +     from the libxl API.
> +     The network script is responsible for enforcing this.

Actually, netback does this itself in practice.

Ian.


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel