Re: [Xen-API] VDI import/export with deltas

[Dave Scott <Dave.Scott@xxxxxxxxxx>]

Hi Mate,

On 29 Aug 2014, at 09:58, Mate Lakat <mate.lakat@xxxxxxxxxx> wrote:

> Hi,
> 
> I'm happy to see a prototype for import/export. It seems that this discussion 
> is generating a lot of traffic - do we plan to have some sort of design 
> session about it? I believe the storage team and the stackers would love to 
> be involved in these sessions.
> We also had a discussion which might be releated:
> https://lists.xenserver.org/sympa/arc/xs-devel/2013-08/thrd2.html#00059

We did, thanks for the pointer.

I’ve also done a survey of import/export code in xapi:

http://wiki.xensource.com/wiki/Disk_import/export

The prototype is basically ‘what we already have’: the code was all written for 
various other purposes (e.g. storage motion and CloudStack integration) and all 
I did was to fill in a few missing gaps and add CLI commands on top of existing 
APIs.


> I think the idea of a filesystem interface is very good, however given that 
> the filesystem must provide a consistent view and transactions, as other 
> processes might manipulate files (GC), I think for the moment, let's try to 
> do a bit narrower interface.

I’m definitely in favour of a narrow interface :-)
At the moment our interface is far too wide and exposes far too much (not 
interesting but still important) internal details.


> I also wanted to raise the content ID question - would be nice to generate a 
> content id during the export, and mark it as dirty whenever it has changed, 
> so that the client would know that she should not expect a consistent output.

That would be good. One of the things the current code can’t do — but that I’m 
sure we discussed before — is to support resumable downloads.

Thanks,
Dave

> 
> Cheers,
> Mate
> 
> 
> -----Original Message-----
> From: Dave Scott 
> Sent: Thursday, August 28, 2014 5:10 PM
> To: Bob Ball
> Cc: xen-api@xxxxxxxxxxxxxxxxxxxx; Mate Lakat; John Garbutt; Ant Messerli
> Subject: Re: VDI import/export with deltas
> 
> 
> On 28 Aug 2014, at 15:50, Bob Ball <bob.ball@xxxxxxxxxx> wrote:
> 
>>> http://wiki.xensource.com/wiki/Disk_import/export_APIs
>> 
>> This is great, but I think we're missing the ability to pull a VHD in or 
>> push a VHD out.
>> 
>> The OpenStack case needs to call a vm-import giving a URL to pull a VHD 
>> from, and a vm-export to push a VHD out to.
>> In particular I think we need to set specific headers as well, as 
>> authentication, so I'd like to propose the following as a hopefully simple 
>> extension:
>> 
>> xe vdi-export uuid=$VDI format=vhd dest=http://192.168.0.1:5000/... 
>> headers:arbitrary_string=$AUTH_TOKEN
>> 
>> xe vdi-import uuid=$RESTORE format=vhd 
>> source=http://192.168.0.1:5000/... 
>> headers:arbitrary_string=$AUTH_TOKEN
> 
> This looks sensible to me; plus I'd like to use URLs more in the storage 
> interface in future.
> 
>> 
>> OpenStack also has a case for using bittorrent to download the VDI from 
>> peers rather than the central server, but I don't see how to accomplish that 
>> with the current proposal without requiring double the disk space as we'd 
>> need to download it then push it into XS.
> 
> So at the moment, do you receive the .vhd data by bittorrent in-place?
> 
> When I last spoke to John about this we discussed some kind of import/export 
> plugin extension. I guess to do the most efficient thing possible it would 
> have to have access to (at least a subset of) the physical storage, so it can 
> 'drop in' a vhd (or qcow or vmdk). Ideally the plugin wouldn't see anything 
> else (so it couldn't get confused or make a mistake) and the modified files 
> would be health-checked when the operation is finished.
> 
>> Thoughts?
> 
> Maybe we could make a safe import/export virtual filesystem with FUSE? 
> Perhaps you could even SMB export a virtual folder for XenCenter users on 
> windows to drag 'n drop files into.
> 
> Cheers,
> Dave
> 
>> 
>> Thanks,
>> 
>> Bob
>> 
>>> There's still time before 2.0 to change how these work, so your 
>>> comments and criticism would be appreciated.
>>> 
>>> Thanks!
>>> Dave
>>> 
>>> [1] https://github.com/xapi-project/xapi-project/issues/1
>>> [2] https://github.com/xapi-project/xen-api/issues/1778
>>> [3] https://github.com/xapi-project/vhd-tool/pull/14
>>> _______________________________________________
>>> Xen-api mailing list
>>> Xen-api@xxxxxxxxxxxxx
>>> http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api
> 


_______________________________________________
Xen-api mailing list
Xen-api@xxxxxxxxxxxxx
http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api