Re: [Xen-API] [xs-devel] Backup solutions for XenServer

[Tobias Kreidl <tobias.kreidl@xxxxxxx>]

As of around CentOS 6.5 (and for sure in RHEL 6.4) that there is a 
thinlvm utility that seems to take care of supporting thinly-provisioned 
LVM volumes. There is an associated snapshot design based on LVM thin 
provisioning that even supports the ability to do snapshots of 
snapshots, etc. down the chain.
Unfortunately, from a cursory look, it doesn't look like a back port to 
CentOS 5 would be that easy or even possible, but that it is integrated 
into both CentOS 6 and CentOS 7 gives some hope for a possibly 
standardized support of it in a future XenServer release, doesn't it? 
The other big question would be how readily something like this could be 
integrated into XenServer.
-=Tobias

On 1/8/2015 9:12 AM, Tobias Kreidl wrote:
> Hello, David:
>
> Thanks very  much for that feedback. Simple often means good with an 
> easier implementation and fewer complexities to worry about, so as 
> suggested some time ago on the Xen Project Web site 
> https://xenorg.uservoice.com/forums/172169-xen-development?filter=hot&page=1 
> the thin provisioning of LVM does appear to be a good contender. The 
> running out of space issue is, of course, endemic in any storage 
> mechanism, so it's more a question of how to prevent corruption and 
> recover in that event. As with ext file systems, perhaps keeping a 
> certain percentage of the volume reserved for emergency issues could 
> help while refusing additional write commits at some point when all 
> storage capacity is exhausted (going into "RO" mode). Of course, being 
> able to copy out and recover lost space would be a useful means of 
> being able to keep VMs running by doing storage Xenmotions 
> periodically and recovering the space through some means from the 
> original disk (if need be though a re-initialization process or an LVM 
> resize operation). With thin provisioning, it's less of an issue for 
> some scenarios like a XenDesktop environment containing a large number 
> of Windows clients because Microsoft comes out with monthly patches 
> and it is way more efficient to just apply those to your golden images 
> and reprovision all your VMs. You're then back to your base storage.
> It would also be nice to se VHDX adopted for Windows VMs at some 
> point. And while on the topic of storage, where do things stand with 
> Ceph? The flexibility looks tremendous, but my understanding is that 
> there are still performance issues. It has the capability of natively 
> supporting several different storage options, addressing your comment 
> about plug-in support for virtually any sort of file system one might 
> want. It would also be interesting to get an update on both NFSv4 
> (which works under Creedence, just isn't officially supported) as well 
> as with potential pNFS support. Some vendors like NetApp showed early 
> on that NFS is still a very viable option for many storage needs and 
> our own tests of iSCSI vs. NFS were convincing enough that with an 
> LACP bond, we have a very solid NFS connection to our NexentaStor SDS 
> and with thin provisioning plus on-the-fly compression/decompression, 
> get anywhere from a 20:1 to 50:1 reduction in utilized storage. And as 
> mentioned already, this is with the equal I/O performance of iSCSI. 
> It's hard to beat that, especially if you can convince those who hold 
> the purse strings that you've just increased your storage capacity by 
> a factor of 20 or more without needing any additional disks.
> Again, your efforts in this area are most appreciated and thank you 
> for continuing to investigate and address these points. Citrix has 
> been really great about being open to suggestions and feedback from 
> the user community, so many kudos to you and your colleagues for 
> lending your ears.
> -=Tobias
>
>
>
> On 1/8/2015 7:57 AM, Dave Scott wrote:
> > > On 8 Jan 2015, at 14:26, Lorscheider Santiago 
> > > <lorscheider.santiago@xxxxxxxxx> wrote:
> > > Hi Tobias,
> > >
> > > Congratulations analysis. Really if used LVMoSCSI or LVMoHBA will 
> > > not have much gain.
> > That’s right — our block storage types (LVMoiSCSI and LVMoHBA) are 
> > “thick provisioned”
> > and hence VDIs take up a lot of space. As Tobias says below, this 
> > shows how important
> > thin provisioning is.
> >
> > Within the upstream xapi-project (cc:d xen-api) we’ve been looking 
> > into thin provisioning
> > options for block storage. I’ve written two draft designs:
> >
> > 1. thin LVHD:
> >
> > http://xapi-project.github.io/xapi/futures/thin-lvhd/thin-lvhd.html
> >
> > This proposes to extend the existing LVM-based SRs, allowing LVs to 
> > be resized at runtime
> > as more space is needed. Every LV could start out small (like a file) 
> > and grow as new
> > blocks are written. The trick is to make sure allocation can still 
> > happen when there
> > is a network partition in the pool and avoid timing out VM I/O. The 
> > design proposes to
> > cache free blocks locally on each host, allocate from there, and 
> > replay an allocation
> > journal against the LVM metadata periodically. [ This is all based on 
> > previous work by
> > Germano Percossi and Jon Ludlam, so I can’t claim credit :) ]
> >
> > 2. OCFS2:
> >
> > http://xapi-project.github.io/xapi/futures/ocfs2/ocfs2.html
> >
> > This proposes to manage OCFS2 clustered filesystem instances and to 
> > store VDIs as files
> > (format flexible) on top. Most of the design so far is to deal with 
> > the complexity of
> > managing the configuration and state of the O2CB cluster.
> >
> > My gut feeling at the moment is that thin LVHD is simpler to 
> > implement and will cause
> > less “behaviour churn”, since we don’t have to change how HA behaves, 
> > how maintenance mode
> > is used etc.
> >
> > Hopefully at some point in the future Xapi will be able to use any 
> > user-managed existing
> > filesystem, so a Linux expert power-user could use OCFS2 if they wanted.
> >
> > Cheers,
> > Dave Scott
> >
> > > Best Regards,
> > >
> > > Cordialmente,
> > >
> > > Lorscheider Santiago
> > > Visite meu blog: www.centralcloud.info
> > > Twitter: @lsantiagos
> > > Antes de imprimir, pense em sua responsabilidade e compromisso com o 
> > > meio ambiente
> > > On Tue, Jan 6, 2015 at 4:40 PM, Tobias Kreidl 
> > > <tobias.kreidl@xxxxxxx> wrote:
> > > Thank you very much, Dave, for the updates about this nice addition.
> > >
> > > Here is my one concern, namely that for incremental backups to work 
> > > properly, you need to continuously keep a base snapshot on the 
> > > server, hence you will -- even if sparse storage is utilized -- 
> > > still take up a fair amount of storage as long as you want to 
> > > maintain the snapshot and backup mechanism.
> > > This led me to thinking, so, what happens if you have SRs with 
> > > different underlying properties? What kind of space requirements are 
> > > being realized?
> > > As a test, I took both a Linux RHEL 7 and a Windows 8.1 VM and 
> > > checked how much space was really being used as well as allocated on 
> > > the SR, as well as in the export files. Please note that here, some 
> > > extra space savings take place on the NexentaStor NFS mount because 
> > > of some built-in automatic compression taking place.
> > > For a Linux VM with 8 GB of allocated storage on a VDI and around 
> > > 1.63 GB of space in use and a Windows VM with 40 GB of allocated 
> > > storage and around 20 GB in actual use, here are the numbers I came 
> > > up with:
> > >
> > >                             RHEL 7     RHEL 7        Windows 8.1   
> > > Windows 8.1
> > >   Function                   LVM     NFS+compression LVM      
> > > NFS+compression
> > > base used:                 8.0 GB        1.1 GB       40.0 GB      
> > > 20.2 GB
> > > base allocated:            8.0 GB        8.0 GB       40.0 GB      
> > > 40.0 GB
> > > pre-snapshot
> > > virtual-size:            8589923591   8589934592 42049672960   
> > > 42949672960
> > > physical-utilisation:    8615100416   5984371200 43041947648   
> > > 20136346112
> > > snapshot used:            13.6 GB        1.1 GB       59.0 GB      
> > > 20.2 GB
> > > snapshot allocated:       16.0 GB       16.0 GB       80.1 GB      
> > > 80.0 GB
> > > full snapshot
> > > virtual-size:            8589934592   8589934592 42949672960   
> > > 42949672960
> > > physical-virtualisation:    8388608     39944704 8388608         88576
> > >
> > > snapshot
> > > export size (VHD):         5.97 GB      5.97 GB       20.24 GB     
> > > 20.22 GB
> > > delta snapshot used:      13.6 GB       1.1 GB        59.1 GB       
> > > 0.3 GB
> > > delta snap allocated:     24.0 GB      24.0 GB       120.1 GB     
> > > 120.0 GB
> > > delta snapshot
> > > virtual-size:            8589934592   8589934592 42949672960   
> > > 42949672960
> > > physical-virtualistion:     8388608        20992 8388608         88576
> > >
> > > delta snapshot
> > > export size:              0.126 GB      0.044 GB       0.078 GB    
> > > 0.074 GB
> > >
> > > Not surprising is that the exports all came our roughly the same 
> > > (they are, after all, creating VHD files in all cases), with minor 
> > > difference due to some activity between when the exports were 
> > > created. However, what is particularly interesting in the NFS-based 
> > > SR is that the initial snapshot seems to have taken up virtually no 
> > > discernible space at all. Why? Evidently because it creates it's own 
> > > initial "difference" disk. Running "xe vdi-list uuid=(UUID_of_VDI) 
> > > params=all" on the VDI yields details on the space 
> > > utilization/allocation. This is actually quite interesting, as I had 
> > > not tried this with an already thinly-provisioned (as opposed to 
> > > sparely populated) storage before.
> > > Noting how an NFS-based SR is so much more space efficient, it does 
> > > however leave the issue that if create both a full and delta 
> > > snapshot, you will no longer be able to storage Xenmotion the VM 
> > > until you bring the snapshot count back down to two or lower. One 
> > > option would be to immediately clean this up after the latest delta 
> > > vdi-export has taken place; alternatively, you could just defer this 
> > > until a storage Xenmotion or other action is called for that 
> > > requires prior cleanup.
> > > This brings us back to the LVM case. The huge difference here is 
> > > that you need readily three times the size of the VDI to be 
> > > allocated and at least double the size to be able to retain a base 
> > > copy and triple the size to hold in addition a  delta snapshot. 
> > > Plus, unlike a NFS-based SR, you cannot over-commit your storage 
> > > allocation on the SR. it would hence be of great benefit in space 
> > > savings if one could do one of the following: (1) store a 
> > > thinly-provisioned and compressed version of the initial snapshot 
> > > similar to what NexentaStor does with the NFS VM, (2) in creating an 
> > > incremental snapshot, access somehow an off-line VDI file containing 
> > > the base, (3) had the means to efficiently temporarily pull in an 
> > > offlined VDI so you'd not have to keep the base snapshot on the SR 
> > > constantly, (4) be able to store the base snapshot on a different SR 
> > > than the VDI you want to snapshot, or (5) some other clever, unnamed 
> > > mechanism.
> > > The NAU VMbackup mechanism we have used in-house for years is 
> > > efficient mainly because it creates a full snapshot on the fly for 
> > > the purpose of a full backup and once completed, deletes it. Hence, 
> > > you never need more additional space at any given time than that of 
> > > a copy of the largest storage associated with an particular VM. The 
> > > disadvantage is that this is of course a sequential operation and 
> > > hence takes quite some time if you have a lot of VMs. However, being 
> > > able to snapshot and keep a spare image would still take up a large 
> > > amount of extra space. While having the means to very efficiently 
> > > create incremental (delta) snapshots and back them up, this still 
> > > leaves the large storage requirement issue open for LVM-based 
> > > storage, should one want to retain baseline snapshots over longer 
> > > periods of time.
> > > To me, this is an indication how thin provisioning (in contrast to 
> > > just sparse storage) can make a huge difference. It would be really 
> > > interesting to see what other options could be implemented to help 
> > > address the LVM limitations outlined above.
> > > Thanks for taking the time to look over these thoughts, and I most 
> > > certainly welcome feedback, in particular if I have overlooked 
> > > something blatant.
> > > -=Tobias
> > >
> > >
> > > On 1/5/2015 8:09 AM, Dave Scott wrote:
> > > > > On 31 Dec 2014, at 11:27, Lorscheider Santiago 
> > > > > <lorscheider.santiago@xxxxxxxxx>
> > > > >   wrote:
> > > > >
> > > > > Hi Dave!
> > > > >
> > > > > Very good xapi the project site. Thanks for the tip!
> > > > > http://xapi-project.github.io/
> > > > >
> > > > >
> > > > > About recuros of "export and import only the blocks Which have 
> > > > > changed between two snapshots" this is a sensational news! It's 
> > > > > been times that I miss this feature, which brings tremendous 
> > > > > agility to backup. From what you wrote, similar operation to 
> > > > > VMware CBT.
> > > > > By your tests, you confirmed that this resource is in the build 
> > > > > 90383c. We can not test since the Release Candidate build is in 
> > > > > 90239c. Although he had already noticed changes in the snapshot in 
> > > > > earlier builds the 90239c.
> > > > > Congratulations for the work and thanks for sharing this 
> > > > > information with us!
> > > > I’m glad you liked the new site! Credit for the site itself and 
> > > > much of the content should also go to: (extracted from git history 
> > > > in no particular order)
> > > > Euan Harris
> > > > John Else
> > > > Jonathan Davies
> > > > Rob Hoes
> > > > Si Beaumont
> > > >
> > > > If you find any problems with it or have suggestions for 
> > > > improvement (or new content), feel free to make pull requests or 
> > > > make issues on the tracker:
> > > >
> > > > https://github.com/xapi-project/xapi-project.github.io
> > > >
> > > >
> > > > Cheers,
> > > > Dave
> > > >
> > > >
> > > > > Cordialmente,
> > > > >
> > > > > Lorscheider Santiago
> > > > > Visite meu blog:
> > > > > www.centralcloud.info
> > > > >
> > > > > Twitter: @lsantiagos
> > > > > Antes de imprimir, pense em sua responsabilidade e compromisso com 
> > > > > o meio ambiente
> > > > > On Mon, Dec 29, 2014 at 7:47 PM, Dave Scott
> > > > > <Dave.Scott@xxxxxxxxxx>
> > > > >   wrote:
> > > > >
> > > > >
> > > > > > On 29 Dec 2014, at 15:31, Dennis Duane Booher <Duane.Booher@xxxxxxx>
> > > > > >   wrote:
> > > > > >
> > > > > > Hi Dave,
> > > > > >
> > > > > > This looks very interesting. Since this is in the xapi-project 
> > > > > > does this mean it is intended for some future release? Is it in 
> > > > > > the current Creedence pre-release build?
> > > > > I believe the code for this is present in the recent creedence 
> > > > > builds. Just to check I installed build number ‘90383c’ and did a 
> > > > > bit of smoke testing— the xe commands listed in the xapi-project 
> > > > > page ran ok and produced the correct output.
> > > > > HTH,
> > > > > Dave
> > > > >
> > > > >
> > > > > > Duane
> > > > > >
> > > > > > -----Original Message-----
> > > > > > From:
> > > > > > xs-devel-request@xxxxxxxxxxxxxxxxxxx 
> > > > > > [mailto:xs-devel-request@xxxxxxxxxxxxxxxxxxx
> > > > > > ] On Behalf Of Dave Scott
> > > > > > Sent: Sunday, December 28, 2014 4:05 AM
> > > > > > To:
> > > > > > xs-devel@xxxxxxxxxxxxxxxxxxx
> > > > > >
> > > > > > Subject: Re: [xs-devel] Backup solutions for XenServer
> > > > > >
> > > > > > Hi,
> > > > > >
> > > > > >
> > > > > > > On 27 Dec 2014, at 12:07, Lorscheider Santiago 
> > > > > > > <lorscheider.santiago@xxxxxxxxx>
> > > > > > >   wrote:
> > > > > > >
> > > > > > > Hi Duane,
> > > > > > >
> > > > > > > I do not mean backup tools but the improvements that XenServer 
> > > > > > > 6.5 in your api to facilitate the work of backup tools.
> > > > > > I've written up some of the recent XenAPI improvements here:
> > > > > >
> > > > > >
> > > > > > http://xapi-project.github.io/features/snapshots/snapshots.html
> > > > > >
> > > > > >
> > > > > > The new APIs allow you to
> > > > > >
> > > > > > - export and import disks in .vhd format (previously we only 
> > > > > > supported raw). This means the files are sparse.
> > > > > > - export and import only the blocks which have changed between 
> > > > > > two snapshots. This allows you to avoid re-copying the same data 
> > > > > > again and again.
> > > > > > Let me know what you think.
> > > > > >
> > > > > > Cheers,
> > > > > > Dave
> > > > > >
> > > > > >
> > > > > > > Cordialmente,
> > > > > > >
> > > > > > > Lorscheider Santiago
> > > > > > > Visite meu blog:
> > > > > > > www.centralcloud.info
> > > > > > >
> > > > > > > Twitter: @lsantiagos
> > > > > > > Antes de imprimir, pense em sua responsabilidade e compromisso 
> > > > > > > com o meio ambiente
> > > > > > > On Fri, Dec 19, 2014 at 8:27 PM, Dennis Duane Booher
> > > > > > > <Duane.Booher@xxxxxxx>
> > > > > > >   wrote:
> > > > > > > Hi Lorscheider,
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > We use
> > > > > > > https://github.com/NAUbackup
> > > > > > >   for all our production Xenserver VM backups. It is freely 
> > > > > > > available if you would like to give it a try.
> > > > > > >
> > > > > > >
> > > > > > > Duane
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > From:
> > > > > > > xs-devel-request@xxxxxxxxxxxxxxxxxxx 
> > > > > > > [mailto:xs-devel-request@xxxxxxxxxxxxxxxxxxx
> > > > > > > ] On Behalf Of Lorscheider Santiago
> > > > > > > Sent: Friday, December 19, 2014 3:33 AM
> > > > > > > To: xs-devel
> > > > > > > Subject: [xs-devel] Backup solutions for XenServer
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Hi All,
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Looking backup solutions available on the market, solutions with 
> > > > > > > good support XenServer are limited. The changes made in 
> > > > > > > creedence api will make it easier and more attractive than more 
> > > > > > > backup siluçoes have support XenServer?
> > > > > > >
> > > > > > >
> > > > > > > Regards,
> > > > > > >
> > > > > > > Lorscheider Santiago
> > > > > > > Visite meu blog:
> > > > > > > www.centralcloud.info
> > > > > > >
> > > > > > > Twitter: @lsantiagos
> > > > > > > Antes de imprimir, pense em sua responsabilidade e compromisso 
> > > > > > > com o meio ambiente


_______________________________________________
Xen-api mailing list
Xen-api@xxxxxxxxxxxxx
http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api