[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Xen-API] [xs-devel] Backup solutions for XenServer
Hello, David:
Thanks very much for that feedback. Simple often means good with an
easier implementation and fewer complexities to worry about, so as
suggested some time ago on the Xen Project Web site
https://xenorg.uservoice.com/forums/172169-xen-development?filter=hot&page=1
the thin provisioning of LVM does appear to be a good contender. The
running out of space issue is, of course, endemic in any storage
mechanism, so it's more a question of how to prevent corruption and
recover in that event. As with ext file systems, perhaps keeping a
certain percentage of the volume reserved for emergency issues could
help while refusing additional write commits at some point when all
storage capacity is exhausted (going into "RO" mode). Of course, being
able to copy out and recover lost space would be a useful means of being
able to keep VMs running by doing storage Xenmotions periodically and
recovering the space through some means from the original disk (if need
be though a re-initialization process or an LVM resize operation). With
thin provisioning, it's less of an issue for some scenarios like a
XenDesktop environment containing a large number of Windows clients
because Microsoft comes out with monthly patches and it is way more
efficient to just apply those to your golden images and reprovision all
your VMs. You're then back to your base storage.
It would also be nice to se VHDX adopted for Windows VMs at some point.
And while on the topic of storage, where do things stand with Ceph? The
flexibility looks tremendous, but my understanding is that there are
still performance issues. It has the capability of natively supporting
several different storage options, addressing your comment about plug-in
support for virtually any sort of file system one might want. It would
also be interesting to get an update on both NFSv4 (which works under
Creedence, just isn't officially supported) as well as with potential
pNFS support. Some vendors like NetApp showed early on that NFS is still
a very viable option for many storage needs and our own tests of iSCSI
vs. NFS were convincing enough that with an LACP bond, we have a very
solid NFS connection to our NexentaStor SDS and with thin provisioning
plus on-the-fly compression/decompression, get anywhere from a 20:1 to
50:1 reduction in utilized storage. And as mentioned already, this is
with the equal I/O performance of iSCSI. It's hard to beat that,
especially if you can convince those who hold the purse strings that
you've just increased your storage capacity by a factor of 20 or more
without needing any additional disks.
Again, your efforts in this area are most appreciated and thank you for
continuing to investigate and address these points. Citrix has been
really great about being open to suggestions and feedback from the user
community, so many kudos to you and your colleagues for lending your ears.
-=Tobias
On 1/8/2015 7:57 AM, Dave Scott wrote:
On 8 Jan 2015, at 14:26, Lorscheider Santiago <lorscheider.santiago@xxxxxxxxx>
wrote:
Hi Tobias,
Congratulations analysis. Really if used LVMoSCSI or LVMoHBA will not have much
gain.
That’s right — our block storage types (LVMoiSCSI and LVMoHBA) are “thick
provisioned”
and hence VDIs take up a lot of space. As Tobias says below, this shows how
important
thin provisioning is.
Within the upstream xapi-project (cc:d xen-api) we’ve been looking into thin
provisioning
options for block storage. I’ve written two draft designs:
1. thin LVHD:
http://xapi-project.github.io/xapi/futures/thin-lvhd/thin-lvhd.html
This proposes to extend the existing LVM-based SRs, allowing LVs to be resized
at runtime
as more space is needed. Every LV could start out small (like a file) and grow
as new
blocks are written. The trick is to make sure allocation can still happen when
there
is a network partition in the pool and avoid timing out VM I/O. The design
proposes to
cache free blocks locally on each host, allocate from there, and replay an
allocation
journal against the LVM metadata periodically. [ This is all based on previous
work by
Germano Percossi and Jon Ludlam, so I can’t claim credit :) ]
2. OCFS2:
http://xapi-project.github.io/xapi/futures/ocfs2/ocfs2.html
This proposes to manage OCFS2 clustered filesystem instances and to store VDIs
as files
(format flexible) on top. Most of the design so far is to deal with the
complexity of
managing the configuration and state of the O2CB cluster.
My gut feeling at the moment is that thin LVHD is simpler to implement and will
cause
less “behaviour churn”, since we don’t have to change how HA behaves, how
maintenance mode
is used etc.
Hopefully at some point in the future Xapi will be able to use any user-managed
existing
filesystem, so a Linux expert power-user could use OCFS2 if they wanted.
Cheers,
Dave Scott
Best Regards,
Cordialmente,
Lorscheider Santiago
Visite meu blog: www.centralcloud.info
Twitter: @lsantiagos
Antes de imprimir, pense em sua responsabilidade e compromisso com o meio
ambiente
On Tue, Jan 6, 2015 at 4:40 PM, Tobias Kreidl <tobias.kreidl@xxxxxxx> wrote:
Thank you very much, Dave, for the updates about this nice addition.
Here is my one concern, namely that for incremental backups to work properly,
you need to continuously keep a base snapshot on the server, hence you will --
even if sparse storage is utilized -- still take up a fair amount of storage as
long as you want to maintain the snapshot and backup mechanism.
This led me to thinking, so, what happens if you have SRs with different
underlying properties? What kind of space requirements are being realized?
As a test, I took both a Linux RHEL 7 and a Windows 8.1 VM and checked how much
space was really being used as well as allocated on the SR, as well as in the
export files. Please note that here, some extra space savings take place on the
NexentaStor NFS mount because of some built-in automatic compression taking
place.
For a Linux VM with 8 GB of allocated storage on a VDI and around 1.63 GB of
space in use and a Windows VM with 40 GB of allocated storage and around 20 GB
in actual use, here are the numbers I came up with:
RHEL 7 RHEL 7 Windows 8.1 Windows 8.1
Function LVM NFS+compression LVM NFS+compression
base used: 8.0 GB 1.1 GB 40.0 GB 20.2 GB
base allocated: 8.0 GB 8.0 GB 40.0 GB 40.0 GB
pre-snapshot
virtual-size: 8589923591 8589934592 42049672960 42949672960
physical-utilisation: 8615100416 5984371200 43041947648 20136346112
snapshot used: 13.6 GB 1.1 GB 59.0 GB 20.2 GB
snapshot allocated: 16.0 GB 16.0 GB 80.1 GB 80.0 GB
full snapshot
virtual-size: 8589934592 8589934592 42949672960 42949672960
physical-virtualisation: 8388608 39944704 8388608 88576
snapshot
export size (VHD): 5.97 GB 5.97 GB 20.24 GB 20.22 GB
delta snapshot used: 13.6 GB 1.1 GB 59.1 GB 0.3 GB
delta snap allocated: 24.0 GB 24.0 GB 120.1 GB 120.0 GB
delta snapshot
virtual-size: 8589934592 8589934592 42949672960 42949672960
physical-virtualistion: 8388608 20992 8388608 88576
delta snapshot
export size: 0.126 GB 0.044 GB 0.078 GB 0.074 GB
Not surprising is that the exports all came our roughly the same (they are, after all, creating VHD
files in all cases), with minor difference due to some activity between when the exports were
created. However, what is particularly interesting in the NFS-based SR is that the initial snapshot
seems to have taken up virtually no discernible space at all. Why? Evidently because it creates
it's own initial "difference" disk. Running "xe vdi-list uuid=(UUID_of_VDI)
params=all" on the VDI yields details on the space utilization/allocation. This is actually
quite interesting, as I had not tried this with an already thinly-provisioned (as opposed to
sparely populated) storage before.
Noting how an NFS-based SR is so much more space efficient, it does however
leave the issue that if create both a full and delta snapshot, you will no
longer be able to storage Xenmotion the VM until you bring the snapshot count
back down to two or lower. One option would be to immediately clean this up
after the latest delta vdi-export has taken place; alternatively, you could
just defer this until a storage Xenmotion or other action is called for that
requires prior cleanup.
This brings us back to the LVM case. The huge difference here is that you need
readily three times the size of the VDI to be allocated and at least double the
size to be able to retain a base copy and triple the size to hold in addition a
delta snapshot. Plus, unlike a NFS-based SR, you cannot over-commit your
storage allocation on the SR. it would hence be of great benefit in space
savings if one could do one of the following: (1) store a thinly-provisioned
and compressed version of the initial snapshot similar to what NexentaStor does
with the NFS VM, (2) in creating an incremental snapshot, access somehow an
off-line VDI file containing the base, (3) had the means to efficiently
temporarily pull in an offlined VDI so you'd not have to keep the base snapshot
on the SR constantly, (4) be able to store the base snapshot on a different SR
than the VDI you want to snapshot, or (5) some other clever, unnamed mechanism.
The NAU VMbackup mechanism we have used in-house for years is efficient mainly
because it creates a full snapshot on the fly for the purpose of a full backup
and once completed, deletes it. Hence, you never need more additional space at
any given time than that of a copy of the largest storage associated with an
particular VM. The disadvantage is that this is of course a sequential
operation and hence takes quite some time if you have a lot of VMs. However,
being able to snapshot and keep a spare image would still take up a large
amount of extra space. While having the means to very efficiently create
incremental (delta) snapshots and back them up, this still leaves the large
storage requirement issue open for LVM-based storage, should one want to retain
baseline snapshots over longer periods of time.
To me, this is an indication how thin provisioning (in contrast to just sparse
storage) can make a huge difference. It would be really interesting to see what
other options could be implemented to help address the LVM limitations outlined
above.
Thanks for taking the time to look over these thoughts, and I most certainly
welcome feedback, in particular if I have overlooked something blatant.
-=Tobias
On 1/5/2015 8:09 AM, Dave Scott wrote:
On 31 Dec 2014, at 11:27, Lorscheider Santiago <lorscheider.santiago@xxxxxxxxx>
wrote:
Hi Dave!
Very good xapi the project site. Thanks for the tip!
http://xapi-project.github.io/
About recuros of "export and import only the blocks Which have changed between two
snapshots" this is a sensational news! It's been times that I miss this feature,
which brings tremendous agility to backup. From what you wrote, similar operation to
VMware CBT.
By your tests, you confirmed that this resource is in the build 90383c. We can
not test since the Release Candidate build is in 90239c. Although he had
already noticed changes in the snapshot in earlier builds the 90239c.
Congratulations for the work and thanks for sharing this information with us!
I’m glad you liked the new site! Credit for the site itself and much of the
content should also go to: (extracted from git history in no particular order)
Euan Harris
John Else
Jonathan Davies
Rob Hoes
Si Beaumont
If you find any problems with it or have suggestions for improvement (or new
content), feel free to make pull requests or make issues on the tracker:
https://github.com/xapi-project/xapi-project.github.io
Cheers,
Dave
Cordialmente,
Lorscheider Santiago
Visite meu blog:
www.centralcloud.info
Twitter: @lsantiagos
Antes de imprimir, pense em sua responsabilidade e compromisso com o meio
ambiente
On Mon, Dec 29, 2014 at 7:47 PM, Dave Scott
<Dave.Scott@xxxxxxxxxx>
wrote:
On 29 Dec 2014, at 15:31, Dennis Duane Booher <Duane.Booher@xxxxxxx>
wrote:
Hi Dave,
This looks very interesting. Since this is in the xapi-project does this mean
it is intended for some future release? Is it in the current Creedence
pre-release build?
I believe the code for this is present in the recent creedence builds. Just to check I installed
build number ‘90383c’ and did a bit of smoke testing— the xe commands listed in
the xapi-project page ran ok and produced the correct output.
HTH,
Dave
Duane
-----Original Message-----
From:
xs-devel-request@xxxxxxxxxxxxxxxxxxx
[mailto:xs-devel-request@xxxxxxxxxxxxxxxxxxx
] On Behalf Of Dave Scott
Sent: Sunday, December 28, 2014 4:05 AM
To:
xs-devel@xxxxxxxxxxxxxxxxxxx
Subject: Re: [xs-devel] Backup solutions for XenServer
Hi,
On 27 Dec 2014, at 12:07, Lorscheider Santiago <lorscheider.santiago@xxxxxxxxx>
wrote:
Hi Duane,
I do not mean backup tools but the improvements that XenServer 6.5 in your api
to facilitate the work of backup tools.
I've written up some of the recent XenAPI improvements here:
http://xapi-project.github.io/features/snapshots/snapshots.html
The new APIs allow you to
- export and import disks in .vhd format (previously we only supported raw).
This means the files are sparse.
- export and import only the blocks which have changed between two snapshots.
This allows you to avoid re-copying the same data again and again.
Let me know what you think.
Cheers,
Dave
Cordialmente,
Lorscheider Santiago
Visite meu blog:
www.centralcloud.info
Twitter: @lsantiagos
Antes de imprimir, pense em sua responsabilidade e compromisso com o meio
ambiente
On Fri, Dec 19, 2014 at 8:27 PM, Dennis Duane Booher
<Duane.Booher@xxxxxxx>
wrote:
Hi Lorscheider,
We use
https://github.com/NAUbackup
for all our production Xenserver VM backups. It is freely available if you
would like to give it a try.
Duane
From:
xs-devel-request@xxxxxxxxxxxxxxxxxxx
[mailto:xs-devel-request@xxxxxxxxxxxxxxxxxxx
] On Behalf Of Lorscheider Santiago
Sent: Friday, December 19, 2014 3:33 AM
To: xs-devel
Subject: [xs-devel] Backup solutions for XenServer
Hi All,
Looking backup solutions available on the market, solutions with good support
XenServer are limited. The changes made in creedence api will make it easier and
more attractive than more backup siluçoes have support XenServer?
Regards,
Lorscheider Santiago
Visite meu blog:
www.centralcloud.info
Twitter: @lsantiagos
Antes de imprimir, pense em sua responsabilidade e compromisso com o meio
ambiente
_______________________________________________
Xen-api mailing list
Xen-api@xxxxxxxxxxxxx
http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api
|
