Re: [Xen-API] xcp SR and drbd

[Tim <tim@xxxxxxxxxx>]

Hi Denis,

> I'm wondering if it could be possible to extend lvm SR to integrate a 
> DRBD primary/secondary redundancy for a two nodes installation.
> Let me explains the context and the a possible solution.
>
> we've been using drbd for some years on iscsi SAN with Xen/XCP, having 
> 2 redundant iscsi SAN and 2 Xen/XCP nodes.
> This kind of setup works great, however for smaller setups it is kind 
> of overkill.
We used to do exactly the same, using two switches and multi-path to 
deal with the switch redundancy. Feeling it was overkill, we switched to 
a similar setup to you..
> So we started to integrate drbd directly on XCP nodes (there are some 
> docs at linbits about it). However being a little bit paranoid about 
> split brain scenario, we have always been using primary/secondary 
> setups (SR1 primary on XCP1 and SR2 primary on XCP2).
We are using a similar setup, but with dual primary. The two servers are 
connected directly via a cable so there is little possibility of the two 
being disconnected. We have only run into a problem with split-brain 
once where we were doing something rather silly - using vdi-copy to 
migrate virtual disks to a local software raid5 array while running VMs 
on the same raid5 array. In short the whole machine stopped responding 
to the network for minutes at a time while the copy took place. 
Fortunately we were able to recover pretty easily - lesson learned: stay 
away from software raid5 on the hypervisors.
> This kind of setup has a big drawback : VMs with VDI on SR1 have to 
> run on XCP1 and VMs with VDI on SR2 have to run on XCP2. There is a 
> loss of flexibility and a loss of transparancy for XCP admins.
Generally, we have not had a problem with dual primary - we use 
live-migrate and run VMs on either node. When the hypervisors boot, we 
make sure everything is connected, switch both to primary and plug in 
the PBDs.
Our plan for recovering from split-brain it to pick the host with the 
most changed VDIs (the most data) as the new primary. Use DRBD with an 
external meta-disk to replicate the changed VDIs to the new primary. 
Invalidate the data on the junked host and run a full re-sync. Not fun, 
but recoverable as long as you know which VMs have been running on which 
host and you keep an eye out for split-brain (there are hooks you can 
use to notify you when things happen to drbd).
> So I'd want to extend the lvm SR to integrate DRBD primary secondary, 
> and I'd like to have some input on this kind of scenario :
> * one each lvm, create a drbd resource, and when a vbd is brough up 
> the drbd resource is switch to primary.
> * when migrating a vm to the second node, turn drbd on first node 
> secondary, turn drbd on the second node primary, and get on with 
> resuming the VM.
> * when a VMs is brought down, pbd is brought down and drbd resource is 
> switch to secondary.
My understanding is the following: An LVM SR is effectively an LVM 
Volume group attached via a PBD. If you unplug the PBD when shutting 
down the VM, you are taking the whole SR offline. If you had one PBD per 
VM, you would need one SR per VDI therefore one VG per VDI which 
wouldn't work... You wouldn't be able to do snapshots, resize VDIs 
etc.......
What you might do (which is what I think you're angling at) is the 
following:
Both hosts handle their own LVM VG and corresponding LVs. Every time you 
create a VDI, each host would create the LV locally, set up a DRBD 
resource and start syncing. Both VDIs could sit in "Secondary" mode 
while the VM was down, and would only switch to primary while the VM was 
running on that particular host.
Provisions would have to be made for making sure the DRBD/LVM config was 
kept concurrent across the two hosts. Most of the DRBD config (sync 
rates, protcol, passwords, data integrity algorithm etc.....) could be 
stored in the SR config, but one would need to be able to re-build a 
whole SR should a disk on one of the hosts fail.
How would you deal with split-brain? If XCP2 appeared down, would XCP1 
put a VDI into primary without knowing the state of the VDI on XCP2?
Please feel to correct me should my understanding be at fault in any way.

> That would make a lot of drbd resource when accounting for snapshot 
> and all, but if it could be possible to be done, it would be a 
> tremendous addition for smaller setups for SMBs.
So far, we have only thought about a two node setup, but I suppose you 
could go further. You would need a system of keeping track of which VDI 
was mirrored on which which host, automatic re-distribution of VDIs to 
another host in the pool should a another fail (or be removed from the 
pool), each VM would be limited to two hosts in the pool (so you may run 
into problems with multiple host failures), each host would have to have 
a large amount of local storage. There are some limitations, but it 
might be workable......
My two cents!

> I'd be glad to have some input from the dev if possible. By the way, 
> kudos to the devs for XCP 1.6, it really rocks.
Good to hear - thanks to the devs from this corner too. I'm looking 
forward to playing with the new features soon.
Regards,

Tim


_______________________________________________
Xen-api mailing list
Xen-api@xxxxxxxxxxxxx
http://lists.xen.org/cgi-bin/mailman/listinfo/xen-api