[Xen-devel] Solution for problems with HyperSCSI and vbds ?

["Sven Kretzschmar" <sven.kretzschmar@xxxxxx>]

I have thought a little bit about Ian's and Keir's proposals
how to make HyperSCSI and vbds work (also in domains > 0).

> 1. NFS-mount HyperSCSI partitions via domain 0 (this will work
>already)

Although this is possible as a temporary workaround, it would take away
the advantages HyperSCSI offers (speed, very low overhead, especially
with the configuration ;-); also it would add another point of failiure or
instability by using NFS for this.
Also, if you want to start more than 15 domains on a server, the
direct mounting of /dev/sd* devices via NFS (vbds don't work yet
with HyperSCSI), would hit the frontier of the maximum of
allowed and mapped sd*X devices (e.g. only sda1 - sda15 work,
the next minor device no. is already for sdb. And my Raid10 Array
used for testing only exports one 160G /dev/sda device).
That means, in order to break this frontier one would definitely need
vbds via vds. Or perhaps a COW device files somewhen in the 
future (But of course this would also hit performance).
So I dont really favour this possibility.

> 2. NFS-mount VBDs which map onto chunks of HyperSCSI disk, via domain
>0 (this might work if you hack DOM0's xl_scsi.c a bit so that DOM0
>VBDs can map onto HyperSCSI block devices).
Much better but still there's NFS involved for no real need, 
only as a workaround.
So, I don't favour this for the same reasons as in 1.)

>>  3. Add proper support for HyperSCSI to Xen. You'd need some scheme
>> for validating transmits which use the HyperSCSI transport, and
>> demusing received frames to the appropriate domain. I don't know
>> anything about teh protocol, so I don't know how easy this would be
>> (e.g. how much state Xen would need to keep lying around).
>
>[Ian:]The main thing would be turning the VFR into more of an L2 switch
>than a router, with each domain having its own MAC[*]. We could then
>add a rule to grant a domain TX permission for a particular 802
>protocol number. HyperSCSI presumably has some high-level
>server-based authentication and privilege verification? If so, it
>should be pretty straightforward. 

This is much better, though more complicated too ;-)

However, I wouldn't do this based on protocols or routing HyperSCSI
ether packets or the need to use HyperSCSI kernel modules in 
domains > 0 (Perhaps too complicated and only a special solution for this
problem).


Here are my first thoughts about a solution:

I try to describe it roughly from the view of domain #1 (not DOM0):

Preconditions: /dev/xvda is a vbd which is attached to a vd which is
mapped to a HyperSCSI partition (e.g. /dev/sda5).
A simpler case would be a direct mapping from a vbd to /dev/sda5; 
without a vd in between. I will only use this case for the sake of
simplicity now...:

Some Application in domain #1 accesses /dev/xvda.
The virtual block device driver mapps this to /dev/sda and forwards
the request to Xen (perhaps it also tags this request as a request
to a "special device" before forwarding the request to Xen).
Xen realizes that there is no physical device connected to /dev/sda
(or registered with Xen ? Maybe it can then also recognize that
the request was marked as targeting a "special device").
Because of that condition, it forwards this block device request
to DOM0 now in which a "request handler" kernel module will listen for 
block device requests which may be forwarded to DOM0 from 
Xen to be handled in DOM0 (It will need to register a callback 
function with Xen in order to do so).
This callback function is now called by Xen to forward the block 
device request to the kernel module loaded in DOM0.
This "block device request handler" kernel module checks the 
data for the block device request (e.g. ioctl, read or write) and just
tries to execute the requested operation on the designated device 
(/dev/sda in our example) in DOM0 and gives back the result and/or 
data to Xen.
If there's no device driver attached to /dev/sda (like, in this example,
the HyperSCSI kernel module) the handler module will return the error
condition to Xen.
Xen in turn hands back the result code and data to the virtual block
device driver in Domain #1 which forwards it to the application who
triggered the block device access request. Done.

Sorry for this prosa above being a little bit unspecific, I currently
don't have that much time to make it shine... ;-)
I have tried to simplify some of the internals of Xen, also because
I don't understand it completly yet ;-)
If I made errors, please feel free to correct me.

This is somewhat similar to the proposal to load "normal" linux
device drivers exclusively in DOM0 to access hardware (also by
Domains > 0) via a lot of already written linux device drivers, instead
of letting Xen handle all access to hardware alone.
But in this case only for block devices.

I would like to do a _little_ restricted case study for that by using the 
HyperSCSI / vbd problem as an example.

However I would need some "small" help and hints from the 
Xen team:


What would be the cleanest way to do the communication
between a kernel module loaded in DOM0 and Xen ?

Does the Xen-API offers functions for registering 
callbacks (to kernel modules) ?

If yes, what are the names and how are they used ?
Where are these functions defined (in which source files) ?

Can these callbacks be done asynchronously somehow
(That is, Xen should be able to call the kernel module at some time
to initiate the block device request in DOM0, and then be called
back later asynchronously with the results by DOM0, because
device accesses can take some time....) ?

Are there any special caveats I have to pay attention at ?


I think all this might also be interesting for accessing other
block devices for which no Xen drivers exists.
A very similar thing could perhaps also be done for
character devices ?

Thanks in advance for any help you could offer me to get
this working... :)

I would of course also like to hear some opinions or concerns
from other members of this list or the Xen team about my
above proposed possible(?) solution ;-)

Regards,

Sven





-------------------------------------------------------
This SF.net email is sponsored by: SF.net Giveback Program.
SourceForge.net hosts over 70,000 Open Source Projects.
See the people who have HELPED US provide better services:
Click here: http://sourceforge.net/supporters.php
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxxx
https://lists.sourceforge.net/lists/listinfo/xen-devel