[Xen-users] Re: Re: Redundant server setup

[Matthew Palmer <mpalmer@xxxxxxxxxxx>]

On Fri, May 12, 2006 at 09:38:09AM +0100, Matthew Wild wrote:
> What I've been working on is using a separate box to provide the storage for 
> the domUs through iSCSI. So at the moment I have two machines set up as dom0s 
> and one NAS box using iscsitarget to provide storage. As long as I use 
> the /dev/disk/by-id names for the disks, I can run xm migrate --live and 
> everything switches over imperceptibly.

Shared storage is the best if you can swing it.  Not necessarily the
cheapest option in the world, though if you want real reliability -- I'd
never just use another box as my NAS, because that's a new and possibly even
more dangerous single point of failure.  And fully-redundant channel-bonded
everything can get costly and complex quickly.  DRBD has it's quirks, but
it's nicely redundant and *cheap*.

> All we need now is to use heartbeat to check if domUs or a dom0 has failed 
> and 
> start up on the other as appropriate.

That's easy enough to do -- you just specify domU::<hostname> in the
haresources line for each of your domUs, and write a quick domU script for
resources.d to up/down the domains.  Bonus points are available for checking
if we're doing a gentle move (hb_takeover instead of a DR event) and use
live migration to minimise downtime.

> It would also be useful to consider some load-balancing, though that's a
> longer term thought.

You mean automatically distribute domUs across the cluster to maintain a
consistent(ish) load average?  That's non-trivial to do, but with live
migration you can have a fairly minimal amount of downtime while the hosts
move around.

> The other thing that I'm thinking hard about is how to decide which domUs
> come up automatically on which dom0, though heartbeat may be able to help
> with that.

With the simple two-node heartbeat, yeah, you just list the domUs in the
haresources file and heartbeat manages the start/stop.  For the simplest
case -- that of one active, one reserve -- dom0, you simply list all your
domUs against the "primary" node (a fairly arbitrary distinction, of
course).

For a slightly more interesting configuration, you can list half your domUs
as primarying against each of the two dom0s, so you can get maximum
performance in the general case.  auto_failback is probably a given in this
situation -- and here the live migration support in your domU script will
really shine.

The problem with both of these configurations, by default, is that you'll
only ever be able to allocate half your cluster-available RAM, so there's
enough RAM available to have all the domains on one machine during failover. 
The coolest configuration is when you have the failover scripts
automatically balloon down all of the running domains to half their RAM when
you're bringing the other node's domUs across (and then ballooning them back
up afterwards when all the other domUs go away), so you can normally have
all your domains running at full memory, for maximum memory usage.  Of
course, you want to make sure you give all your domUs a little extra swap
for those occasional DR moments.

- Matt

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users