Re: [Xen-API] [xs-devel] Backup solutions for XenServer

[Dave Scott <Dave.Scott@xxxxxxxxxx>]

> On 8 Jan 2015, at 14:26, Lorscheider Santiago 
> <lorscheider.santiago@xxxxxxxxx> wrote:
> 
> Hi Tobias,
> 
> Congratulations analysis. Really if used LVMoSCSI or LVMoHBA will not have 
> much gain.

That’s right — our block storage types (LVMoiSCSI and LVMoHBA) are “thick 
provisioned”
and hence VDIs take up a lot of space. As Tobias says below, this shows how 
important
thin provisioning is.

Within the upstream xapi-project (cc:d xen-api) we’ve been looking into thin 
provisioning
options for block storage. I’ve written two draft designs:

1. thin LVHD:

http://xapi-project.github.io/xapi/futures/thin-lvhd/thin-lvhd.html

This proposes to extend the existing LVM-based SRs, allowing LVs to be resized 
at runtime
as more space is needed. Every LV could start out small (like a file) and grow 
as new
blocks are written. The trick is to make sure allocation can still happen when 
there
is a network partition in the pool and avoid timing out VM I/O. The design 
proposes to
cache free blocks locally on each host, allocate from there, and replay an 
allocation
journal against the LVM metadata periodically. [ This is all based on previous 
work by
Germano Percossi and Jon Ludlam, so I can’t claim credit :) ]

2. OCFS2:

http://xapi-project.github.io/xapi/futures/ocfs2/ocfs2.html

This proposes to manage OCFS2 clustered filesystem instances and to store VDIs 
as files
(format flexible) on top. Most of the design so far is to deal with the 
complexity of
managing the configuration and state of the O2CB cluster.

My gut feeling at the moment is that thin LVHD is simpler to implement and will 
cause
less “behaviour churn”, since we don’t have to change how HA behaves, how 
maintenance mode
is used etc.

Hopefully at some point in the future Xapi will be able to use any user-managed 
existing
filesystem, so a Linux expert power-user could use OCFS2 if they wanted.

Cheers,
Dave Scott

> 
> Best Regards,
> 
> Cordialmente,
> 
> Lorscheider Santiago
> Visite meu blog: www.centralcloud.info
> Twitter: @lsantiagos
> Antes de imprimir, pense em sua responsabilidade e compromisso com o meio 
> ambiente
> 
> On Tue, Jan 6, 2015 at 4:40 PM, Tobias Kreidl <tobias.kreidl@xxxxxxx> wrote:
> Thank you very much, Dave, for the updates about this nice addition.
> 
> Here is my one concern, namely that for incremental backups to work properly, 
> you need to continuously keep a base snapshot on the server, hence you will 
> -- even if sparse storage is utilized -- still take up a fair amount of 
> storage as long as you want to maintain the snapshot and backup mechanism.
> 
> This led me to thinking, so, what happens if you have SRs with different 
> underlying properties? What kind of space requirements are being realized?
> 
> As a test, I took both a Linux RHEL 7 and a Windows 8.1 VM and checked how 
> much space was really being used as well as allocated on the SR, as well as 
> in the export files. Please note that here, some extra space savings take 
> place on the NexentaStor NFS mount because of some built-in automatic 
> compression taking place.
> 
> For a Linux VM with 8 GB of allocated storage on a VDI and around 1.63 GB of 
> space in use and a Windows VM with 40 GB of allocated storage and around 20 
> GB in actual use, here are the numbers I came up with:
> 
> 
>                            RHEL 7     RHEL 7        Windows 8.1   Windows 8.1
>  Function                   LVM     NFS+compression    LVM      
> NFS+compression
> 
> base used:                 8.0 GB        1.1 GB       40.0 GB      20.2 GB
> base allocated:            8.0 GB        8.0 GB       40.0 GB      40.0 GB
> 
> pre-snapshot
> virtual-size:            8589923591   8589934592   42049672960   42949672960
> physical-utilisation:    8615100416   5984371200   43041947648   20136346112
> 
> snapshot used:            13.6 GB        1.1 GB       59.0 GB      20.2 GB
> snapshot allocated:       16.0 GB       16.0 GB       80.1 GB      80.0 GB
> 
> full snapshot
> virtual-size:            8589934592   8589934592   42949672960   42949672960
> physical-virtualisation:    8388608     39944704       8388608         88576
> 
> snapshot
> export size (VHD):         5.97 GB      5.97 GB       20.24 GB     20.22 GB
> 
> delta snapshot used:      13.6 GB       1.1 GB        59.1 GB       0.3 GB
> delta snap allocated:     24.0 GB      24.0 GB       120.1 GB     120.0 GB
> 
> delta snapshot
> virtual-size:            8589934592   8589934592   42949672960   42949672960
> physical-virtualistion:     8388608        20992       8388608         88576
> 
> delta snapshot
> export size:              0.126 GB      0.044 GB       0.078 GB    0.074 GB
> 
> 
> Not surprising is that the exports all came our roughly the same (they are, 
> after all, creating VHD files in all cases), with minor difference due to 
> some activity between when the exports were created. However, what is 
> particularly interesting in the NFS-based SR is that the initial snapshot 
> seems to have taken up virtually no discernible space at all. Why? Evidently 
> because it creates it's own initial "difference" disk. Running "xe vdi-list 
> uuid=(UUID_of_VDI) params=all" on the VDI yields details on the space 
> utilization/allocation. This is actually quite interesting, as I had not 
> tried this with an already thinly-provisioned (as opposed to sparely 
> populated) storage before.
> 
> Noting how an NFS-based SR is so much more space efficient, it does however 
> leave the issue that if create both a full and delta snapshot, you will no 
> longer be able to storage Xenmotion the VM until you bring the snapshot count 
> back down to two or lower. One option would be to immediately clean this up 
> after the latest delta vdi-export has taken place; alternatively, you could 
> just defer this until a storage Xenmotion or other action is called for that 
> requires prior cleanup.
> 
> This brings us back to the LVM case. The huge difference here is that you 
> need readily three times the size of the VDI to be allocated and at least 
> double the size to be able to retain a base copy and triple the size to hold 
> in addition a  delta snapshot. Plus, unlike a NFS-based SR, you cannot 
> over-commit your storage allocation on the SR. it would hence be of great 
> benefit in space savings if one could do one of the following: (1) store a 
> thinly-provisioned and compressed version of the initial snapshot similar to 
> what NexentaStor does with the NFS VM, (2) in creating an incremental 
> snapshot, access somehow an off-line VDI file containing the base, (3) had 
> the means to efficiently temporarily pull in an offlined VDI so you'd not 
> have to keep the base snapshot on the SR constantly, (4) be able to store the 
> base snapshot on a different SR than the VDI you want to snapshot, or (5) 
> some other clever, unnamed mechanism.
> 
> The NAU VMbackup mechanism we have used in-house for years is efficient 
> mainly because it creates a full snapshot on the fly for the purpose of a 
> full backup and once completed, deletes it. Hence, you never need more 
> additional space at any given time than that of a copy of the largest storage 
> associated with an particular VM. The disadvantage is that this is of course 
> a sequential operation and hence takes quite some time if you have a lot of 
> VMs. However, being able to snapshot and keep a spare image would still take 
> up a large amount of extra space. While having the means to very efficiently 
> create incremental (delta) snapshots and back them up, this still leaves the 
> large storage requirement issue open for LVI-based storage, should one want 
> to retain baseline snapshots over longer periods of time.
> 
> To me, this is an indication how thin provisioning (in contrast to just 
> sparse storage) can make a huge difference. It would be really interesting to 
> see what other options could be implemented to help address the LVM 
> limitations outlined above.
> 
> Thanks for taking the time to look over these thoughts, and I most certainly 
> welcome feedback, in particular if I have overlooked something blatant.
> 
> -=Tobias
> 
> 
> On 1/5/2015 8:09 AM, Dave Scott wrote:
>>> On 31 Dec 2014, at 11:27, Lorscheider Santiago 
>>> <lorscheider.santiago@xxxxxxxxx>
>>>  wrote:
>>> 
>>> Hi Dave!
>>> 
>>> Very good xapi the project site. Thanks for the tip! 
>>> http://xapi-project.github.io/
>>> 
>>> 
>>> About recuros of "export and import only the blocks Which have changed 
>>> between two snapshots" this is a sensational news! It's been times that I 
>>> miss this feature, which brings tremendous agility to backup. From what you 
>>> wrote, similar operation to VMware CBT.
>>> 
>>> By your tests, you confirmed that this resource is in the build 90383c. We 
>>> can not test since the Release Candidate build is in 90239c. Although he 
>>> had already noticed changes in the snapshot in earlier builds the 90239c.
>>> 
>>> Congratulations for the work and thanks for sharing this information with 
>>> us!
>>> 
>> I’m glad you liked the new site! Credit for the site itself and much of the 
>> content should also go to: (extracted from git history in no particular 
>> order)
>> 
>> Euan Harris
>> John Else
>> Jonathan Davies
>> Rob Hoes
>> Si Beaumont
>> 
>> If you find any problems with it or have suggestions for improvement (or new 
>> content), feel free to make pull requests or make issues on the tracker:
>> 
>> 
>> https://github.com/xapi-project/xapi-project.github.io
>> 
>> 
>> Cheers,
>> Dave
>> 
>> 
>>> Cordialmente,
>>> 
>>> Lorscheider Santiago
>>> Visite meu blog: 
>>> www.centralcloud.info
>>> 
>>> Twitter: @lsantiagos
>>> Antes de imprimir, pense em sua responsabilidade e compromisso com o meio 
>>> ambiente
>>> 
>>> On Mon, Dec 29, 2014 at 7:47 PM, Dave Scott 
>>> <Dave.Scott@xxxxxxxxxx>
>>>  wrote:
>>> 
>>> 
>>>> On 29 Dec 2014, at 15:31, Dennis Duane Booher <Duane.Booher@xxxxxxx>
>>>>  wrote:
>>>> 
>>>> Hi Dave,
>>>> 
>>>> This looks very interesting. Since this is in the xapi-project does this 
>>>> mean it is intended for some future release? Is it in the current 
>>>> Creedence pre-release build?
>>>> 
>>> I believe the code for this is present in the recent creedence builds. Just 
>>> to check I installed build number ‘90383c’ and did a bit of smoke testing— 
>>> the xe commands listed in the xapi-project page ran ok and produced the 
>>> correct output.
>>> 
>>> HTH,
>>> Dave
>>> 
>>> 
>>>> Duane
>>>> 
>>>> -----Original Message-----
>>>> From: 
>>>> xs-devel-request@xxxxxxxxxxxxxxxxxxx 
>>>> [mailto:xs-devel-request@xxxxxxxxxxxxxxxxxxx
>>>> ] On Behalf Of Dave Scott
>>>> Sent: Sunday, December 28, 2014 4:05 AM
>>>> To: 
>>>> xs-devel@xxxxxxxxxxxxxxxxxxx
>>>> 
>>>> Subject: Re: [xs-devel] Backup solutions for XenServer
>>>> 
>>>> Hi,
>>>> 
>>>> 
>>>>> On 27 Dec 2014, at 12:07, Lorscheider Santiago 
>>>>> <lorscheider.santiago@xxxxxxxxx>
>>>>>  wrote:
>>>>> 
>>>>> Hi Duane,
>>>>> 
>>>>> I do not mean backup tools but the improvements that XenServer 6.5 in 
>>>>> your api to facilitate the work of backup tools.
>>>>> 
>>>> I've written up some of the recent XenAPI improvements here:
>>>> 
>>>> 
>>>> http://xapi-project.github.io/features/snapshots/snapshots.html
>>>> 
>>>> 
>>>> The new APIs allow you to
>>>> 
>>>> - export and import disks in .vhd format (previously we only supported 
>>>> raw). This means the files are sparse.
>>>> - export and import only the blocks which have changed between two 
>>>> snapshots. This allows you to avoid re-copying the same data again and 
>>>> again.
>>>> 
>>>> Let me know what you think.
>>>> 
>>>> Cheers,
>>>> Dave
>>>> 
>>>> 
>>>>> Cordialmente,
>>>>> 
>>>>> Lorscheider Santiago
>>>>> Visite meu blog: 
>>>>> www.centralcloud.info
>>>>> 
>>>>> Twitter: @lsantiagos
>>>>> Antes de imprimir, pense em sua responsabilidade e compromisso com o meio 
>>>>> ambiente
>>>>> 
>>>>> On Fri, Dec 19, 2014 at 8:27 PM, Dennis Duane Booher 
>>>>> <Duane.Booher@xxxxxxx>
>>>>>  wrote:
>>>>> Hi Lorscheider,
>>>>> 
>>>>> 
>>>>> 
>>>>> We use 
>>>>> https://github.com/NAUbackup
>>>>>  for all our production Xenserver VM backups. It is freely available if 
>>>>> you would like to give it a try.
>>>>> 
>>>>> 
>>>>> 
>>>>> Duane
>>>>> 
>>>>> 
>>>>> 
>>>>> From: 
>>>>> xs-devel-request@xxxxxxxxxxxxxxxxxxx 
>>>>> [mailto:xs-devel-request@xxxxxxxxxxxxxxxxxxx
>>>>> ] On Behalf Of Lorscheider Santiago
>>>>> Sent: Friday, December 19, 2014 3:33 AM
>>>>> To: xs-devel
>>>>> Subject: [xs-devel] Backup solutions for XenServer
>>>>> 
>>>>> 
>>>>> 
>>>>> Hi All,
>>>>> 
>>>>> 
>>>>> 
>>>>> Looking backup solutions available on the market, solutions with good 
>>>>> support XenServer are limited. The changes made in creedence api will 
>>>>> make it easier and more attractive than more backup siluçoes have support 
>>>>> XenServer?
>>>>> 
>>>>> 
>>>>> 
>>>>> Regards,
>>>>> 
>>>>> Lorscheider Santiago
>>>>> Visite meu blog: 
>>>>> www.centralcloud.info
>>>>> 
>>>>> Twitter: @lsantiagos
>>>>> Antes de imprimir, pense em sua responsabilidade e compromisso com o meio 
>>>>> ambiente
>>>>> 
>>>>> 
>>>>> 
> 
> 
> 


_______________________________________________
Xen-api mailing list
Xen-api@xxxxxxxxxxxxx
http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api