[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] Driver domains communication protocol proposal



Hello,

I no longer have the original message, so I'm going to reply in a
copy-paste of xen mailing list archive. Sorry for the inconvenience.

> During some discussions and handwaving, including discussions with
> some experts on the Xenserver/XCP storage architecture, we came up
> with what we think might be a plausible proposal for an architecture
> for communication between toolstack and driver domain, for storage at
> least.
>
> I offered to write it up.  The abstract proposal is as I understand
> the consensus from our conversation.  The concrete protocol is my own
> invention.
>
> Please comments.  After a round of review here we should consider
> whether some of the assumptions need review from the communities
> involved in "other" backends (particularly, the BSDs).
>
> (FAOD the implementation of something like this is not 4.3 material,
> but it may inform some API decisions etc. we take in 4.2.)
>
> Ian.
>
>
> Components
>
>  toolstack
>
>  guest
>     Might be the toolstack domain, or an (intended) guest vm.
>
>  driver domain
>     Responsible for providing the disk service to guests.
>     Consists, internally, of (at least):
>        control plane
>        backend
>     but we avoid exposing this internal implementation detail.
>
>     We permit different driver domains on a single host, serving
>     different guests or the same guests.
>
>     The toolstack is expected to know the domid of the driver domain.
>
>  driver domain kind
>     We permit different "kinds" of driver domain, perhaps implemented
>     by completely different code, which support different facilities.
>
>     Each driver domain kind needs to document what targets (see
>     below) are valid and how they are specified, and what preparatory
>     steps may need to be taken eg at system boot.
>
>     Driver domain kinds do not have a formal presence in the API.
>
> Objects
>
>  target
>      A kind of name.
>
>      Combination of a physical location and data format plus all other
>      information needed by the underlying mechanisms, or relating to
>      the data format, needed to access it.
>
>      These names are assigned by the driver domain kind; the names may
>      be an open class; no facility provided via this API to enumerate
>      these.
>
>      Syntactically, these are key/value pairs, mapping short string
>      keys to shortish string values, suitable for storage in a
>      xenstore directory.
>
>  vdi
>      This host's intent to access a specific target.
>      Non-persistent, created on request by toolstack, enumerable.
>      Possible states: inactive/active.
>      Abstract operations: prepare, activate, deactivate, unprepare.
>
>      (We call the "create" operation for this object "prepare" to
>      avoid confusion with other kinds of "create".)
>
>      The toolstack promises that no two vdis for the same target
>      will simultaneously be active, even if the two vdis are on
>      different hosts.
>
>  vbd
>      Provision of a facility for a guest to access a particular target
>      via a particular vdi.  There may be zero or more of these at any
>      point for a particular vdi.
>
>      Non-persistent, created on request by toolstack, enumerable.
>      Abstract operations: plug, unplug.
>
>      (We call the "create" operation for this object "plug" to avoid
>      confusion with other kinds of "create".)
>
>      vbds may be created/destroyed, and the underlying vdi
>      activated/deactivated, in any other.  However IO is only possible
>      to a vbd when the corresponding vdi is active.  The reason for
>      requiring activation as a separate step is to allow as much of
>      the setup for an incoming migration domain's storage to be done
>      before committing to the migration and entering the "domain is
>      down" stage, during which access is switched from the old to the
>      new host.
>
>      We will consider here the case of a vbd which provides
>      service as a Xen vbd backend.  Other cases (eg, the driver domain
>      is the same as the toolstack domain and the vbd provides a block
>      device in the toolstack domain) can be regarded as
>      optimisations/shortcuts.
>
> Concrete protocol
>
>  The toolstack gives instructions to the driver domain, and receives
>  results, via xenstore, in the path:
>    /local/domain/<driverdomid>/backendctrl/vdi
>  Both driver domain and toolstack have write access to the whole of
>  this area.
>
>  Each vdi which has been requested and/or exists, corresponds to a
>  path .../backendctrl/vdi/<vdi> where <vdi> is a string (of
>  alphanumerics, hyphens and underscores) chosen by the toolstack.
>  Inside this, there are the following nodes:
>
>  /local/domain/<driverdomid>/backendctrl/vdi/<vdi>/
>    state       The current state.  Values are "inactive", "active",
>                or ENOENT meaning the vdi does not exist.
>                Set by the driver domain in response to requests.
>
>    request     Operation requested by the toolstack and currently
>                being performed.  Created by the toolstack, but may
>                then not be modified by the toolstack.  Deleted
>                by the driver domain when the operation has completed.
>
>                The values of "request" are:
>                  prepare
>                  activate
>                  deactivate
>                  unprepare
>                  plug <vbd>
>                  unplug <vbd>
>                <vbd> is an id chosen by the toolstack like <vdi>
>
>    result      errno value (in decimal, Xen error number) best
>                describing the results of the most recently completed
>                operation; 0 means success.  Created or set by the
>                driver domain in the same transaction as it deletes
>                request.  The toolstack may delete this.
>
>    result_msg  Optional UTF-8 string explaining any error; does not
>                exist when result is "0".  Created or deleted by the
>                driver domain whenever the driver domain sets result.
>                The toolstack may delete this.
>
>    t/*         The target name.  Must be written by the toolstack.
>                But may not be removed or changed while either of
>                state or request exist.
>
>    vbd/<vbd>/state
>                The state of a vbd, "ok" or ENOENT.
>                Set or deleted by the driver domain in response to
>                requests.
>
>    vbd/<vbd>/frontend
>                The frontend path (complete path in xenstore) which the
>                xen vbd should be servicing.  Set by the toolstack
>                with the plug request and not modified until after
>                completion of unplug.
>
>    vbd/<vbd>/backend
>                The backend path (complete path in xenstore) which the
>                driver domain has chosen for the vbd.  Set by the
>                driver domain in response to a plug request.
>
>    vbd/<vbd>/b-copy/*
>                The driver domain may request, in response to plug,
>                that the toolstack copy these values to the specified
>                backend directory, in the same transaction as it
>                creates the frontend.  Set by the driver domain in
>                response to a plug request; may be deleted by the
>                toolstack.  DEPRECATED, see below.
>
> The operations:
>
>  prepare
>         Creates a vdi from a target.
>         Preconditions:
>             state ENOENT
>             request ENOENT
>         Request (xenstore writes by toolstack):
>             request = "prepare"
>             t/* as appropriate
>         Results on success (xenstore writes by driver domain):
>             request ENOENT    } applies to success from all operations,
>             result = "0"      }  will not be restated below
>             state = "inactive"
>         Results on error (applies to all operations):  }
>             request ENOENT                             }  applies
>             result = some decimal integer errno value  }   to all
>             result_msg = ENOENT or a string            }    failures
>
>  activate
>         Preconditions:
>             state = "inactive"
>             request ENOENT
>         Request:
>             request = "activate"
>         Results on success:
>             state = "active"
>
>  deactivate
>         Preconditions:
>             state = "active"
>             request ENOENT
>         Request:
>             request = "deactivate"
>         Results on success:
>             state = "inactive"
>
>  unprepare
>         Preconditions:
>             state != ENOENT
>             request ENOENT
>         Request:
>             request = "unprepare"
>         Results on success:
>             state = ENOENT
>
>  removal, modification, etc. of an unprepared vdi:
>         Preconditions:
>             state ENOENT
>             request ENOENT
>         Request:
>             any changes to <vdi> directory which do
>              not create "state" or "request"
>         Results:
>             ignored - no response from driver domain
>
>  plug <vbd>
>         Preconditions:
>             state ENOENT

I'm not sure about this, but shouldn't state = "active" or at least
"prepared"? Maybe I don't understant the protocol correctly, but to be
able to plug a vbd, shouldn't the underlying vdi be prepared first?

Also, as far as I understand, each vdi only has one vbd, why is the
<vbd> parameter needed in both the plug and unplug operations?

>             request ENOENT
>             vbd/<vbd>/state ENOENT
>             <frontend> ENOENT
>         Request:
>             request = "plug <vbd>"
>             vbd/<vbd>/frontend = <frontend> ("/local/domain/<guest>/...")
>         Results on success:
>             vbd/<vbd>/state = "ok"
>             vbd/<vbd>/backend = <rel-backend>
>                 (<rel-backend> is the backend path relative to the
>                  driver domain's home directory in xenstore)
>             vbd/<vbd>/b-copy/*  may be created    } at least one of these
>             <backend>/*  may come into existence  }  must happen
>         Next step (xenstore write) by toolstack:
>             <frontend>  created and populated, specifically
>             <frontend>/backend = <backend>
>                 ("/local/domain/<driverdomid>/<rel-backend>")
>             <backend>    created if necessary
>             <backend>/*  copied from  vbd/<vbd>/b-copy/*  if any
>             <backend>/frontend = <frontend>  unless already set
>
>  unplug <vbd>
>         Preconditions:
>             state ENOENT
>             request ENOENT
>             vbd/<vbd>/state "ok"
>         Request:
>             request = "unplug <vbd>"
>             <frontend> ENOENT
>         Results on success:
>             vbd/<vbd>/state ENOENT
>             <backend> ENOENT

So the flow of the procotol is (if everything return success):

connection: prepare -> activate -> plug
disconnection: unplug -> deactivate -> unprepare

>
>  The toolstack and driver domains should not store state of their own,
>  not required for these communication purposes, in the backendctrl/
>  directory in xenstore.  If the driver domain wishes to make records
>  for its own use in xenstore, it should do so in a different directory
>  of its choice (eg, /local/domain/<driverdomid>/private/<something>.
>
>
> Notes regarding driver domains whose block backend implementation is
> controlled from the actual xenstore backend directory:
>
>  The b-copy/* feature exists for compatibility with some of these.  If
>  such a backend cannot cope with the backend directory coming into
>  existence before the corresponding frontend directory, then it is
>  necessary to create and populate the backend in the same xenstore
>  transaction as the creation of the frontend.  However, such backends
>  should be fixed; the b-copy/* feature is deprecated and will be
>  withdrawn at some point.
>
>  Note that a vbd may be created with the vdi inactive.  In this case

So in this case, the connection may happen with:

connection: prepare -> plug -> activate?

I frankly find this vbd/vdi naming very confusing.

>  the frontend and backend directories will exist, but the information
>  needed to start up the backend properly may be lacking until the vdi
>  is activated.  For example, if the existence of a suitable block
>  device in the driver domain depends on vdi activation, the block
>  device id cannot be made known to the backend until after the backend
>  directory has already been created and perhaps has existed for some
>  time.  It is believed that existing backends cope with this, because
>  they use a "hotplug script" approach - where the backend directory is
>  created without specifying the device node, and this backend directory
>  creation causes the invocation of machinery which establishes the
>  device node, which is subsequently written to xenstore.
>
>
> Question
>
>  What about network interfaces and other kinds of backend ?>

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel


 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.