Re: [Xen-devel] Driver domains and hotplug scripts, redux

[Roger Pau Monnà <roger.pau@xxxxxxxxxxxxx>]

Hello,

This comes from my experience with Xen and hotplug scripts, and it
might be wrong, since I wasn't able to find any document explaining
exactly how hotplug execution works and who does what. I'm gonna try
to list the sequence of events that happens when a device is added (I
really don't want to keep on with the discusion if this is a protocol
or not):

1. Toolstack writes: /local/domain/0/backend/<vbd or vif>/... with "state = 1".
2. Kernel acks xenstore backend device creation, creates the device
and sets backend "state = 2".
3. xenbackendd notices backend device with "state == 2" and launches
hotplug script.
4. Hotplug script executes necessary actions and sets backend
"hotplug-status = connected".
5. Kernel notices "hotplug-status == connected", plugs the device, and
sets xenstore backend device "state = 4".

This is true on NetBSD, because there aren't any userspace hotplug
devices, someone should probably add the missing bits if the device is
implemented in userspace (I'm not really sure of what happens inside
the kernel in #2 and #5, specially when using blktap or qdisk).

Regarding device shutdown/destroy:

1. Guest sets frontend state to 6 (closed)
2. Kernel unplugs the device and sets backend "state = 6".
3. xenbackendd notices device with "state == 6", and performs the
necessary cleanup.
3. Toolstack notices device with "state == 6" and removes xenstore
backend entries.

Notice that I've used two #3, that's where the race condition happens,
because there's no synchronization between toolstack and
hotplug/xenbackendd to know when hotplug scripts have been executed
(however we should be able to synchronize this watching
"hotplug-status" instead of "state", and waiting for it to change to
"disconnected").

Now, we have to decide how to fix the shutdown/destroy race and how to
implement this outside of the Dom0. I'm not really sure if it's a good
idea to try so hard to keep this flow intact, I think it's best to try
to define a flow that solves our current problems, regardless of how
things are now, and then try to map both flows to see what should be
changed and how.

Since the device will be plugged from a Domain different than Dom0,
the toolstack doesn't really (and probably shouldn't) know anything
about which backend type will be used (phy, blktap, qdisk...). Having
that in mind, I don't know how can we write
/local/domain/<driverdom_id>/backend/... from Dom0, instead we should
create something like:

/hotplug/domain/<driverdom_id>/<vbd or vif>/<domu_id>/<device_id>/params
/hotplug/domain/<driverdom_id>/<vbd or vif>/<domu_id>/<device_id>/script
/hotplug/domain/<driverdom_id>/<vbd or vif>/<domu_id>/<device_id>/state
[This seem like the minimum necessary parameters, but probably there
are others, so add what you feel necessary]

With that the driver domain should be able to create
/local/domain/<driverdomain_id>/backend/... and the frontend also.

I'm not sure if we should control the execution of hotplug scripts
from Dom0, or instead let the driver domain decide when it's best to
execute each script. This adds /hotplug to xenstore, but the
plug/unplug sequence could be the same as the one we currently have,
the only change is that each driver domain is in charge of writing
it's own xenstore backend/frontend entries to trigger the plug
sequence.

Hope that helps, Roger.

(xen-devel mailing list was removed at some point during the
conversation, so I'm adding it again)

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel