Re: [Xen-users] Dom0 on USB Stick

[Florian Heigl <florian.heigl@xxxxxxxxx>]

Hi,

tl;dr:
- It's a good idea and works great once you got it right
- Don't waste time doing it with a normal distro: instead help make
Xen work nicely on Alpine Linux which is designed for robustness / usb
boot
- check you're not using sched-credit2, Then use sched-credit -d
Domain-0 -w 5120 -c 130 on your dom0. You can still pin it to specific
cpus. Note you can use xenpm to find out which cpus are real ones and
which are just HT.


2011/12/13 Jonathan Tripathy <jonnyt@xxxxxxxxxxx>:> Hi Everyone,
>
> I'm currently mulling over the idea of running our Dom0 on a USB flash disk.
> Currently, we have the Dom0 and DomU running of the same disk array. I'd
> like to know the list's opinion on doing this. It seems like a very nice
> idea to split the single critical Dom0 and the tens of DomUs.
>
> I'd like to stop the DomUs from affecting the disk IOPS available for the
> Dom0 as I'm sure you'll appreciate the critical nature of the it.

Most people, surprisingly, don't.

> Any opinions are very much appreciated

we've done this in a large ha production system.
HA not in the standard "we run important online shop" but in the older
"yes, sure make a cluster. but btw: it's simply not allowed to fail,
period" sense.


The good:
Different fault domains.
dom0 IO not impaired by heavy IO domUs, like you described. This means
dom0 is fully operable even if the host is running multiple domUs
pushing many K IOPS / manu 100 MB/s
all RAID disks are strictly for "payload".

The bad:
USB sticks fail more easily than Raid disks, hotplug is a nightmare
and udev is a failure.

The ugly:
Linux block device persistence in 2.16.XX is a mere joke, old CentOS
kernels will re-enumerate a usb stick that comes back from sleep after
being idle too long, and while they don't have command that allows to
disable powersaving, they seem to have it enabled...
You can of course setup persistent device names, BUT udev fails anyway
since it's all running via callout shell scripts, and udev fails even
more since it does not really change the kernel device names, only the
stuff in /dev/ is affected.
Even if you have rules that name your "replugged" root device
"/dev/myUSBwonder" it will not help at all since the kernel device has
still changed to "the usb thing right next to where the old usb thing
was". That means you lose the root device node and thats it.
In case anybody missed it: Yes, I think doing a kernels job with
shellscripts in user space is not the smartest thing on earth.

Most CentOS code proved to be quite fragile, and not even a reboot
works if you have a disk issue of any notable scale.
Why would anybody test what happens if things go wrong, right?

The upside:
We wanted stuff to be really robust.
Using /etc/sysconfig/readonlyroot + rwtab tweakimg you can actually
load 90% of the system into RAM and still keep 10% on readonly USB
(i.e. stuff in /etc/ that you want to be change-able. We now have a
system that *will* work for months even if the boot media fails. This
allows for the OS to feel "unmodified" in case you need to apply some
post-install updates w/o a reboot.
Now on a failure of root usb thing very little bad can happen, and any
failure on the Raid is something we can almost laugh at.

So, if you need to build a highly robust embedded solution on basis of
a standard distro system, and bring along a few dozen hours, then go
for it. I'm quite happy now that things ended up pretty solid.

Re: PXE booting:
Well, if you want all your hosts to be in a reboot loop after a
massive power failure until the DHCP is back?
Otherwise, try PXE, then rather boot from some solid media (unless
that is not installed via PXE yet. then... usual)



Now for something better:
But, alternatively, if you *wanna* spend time on such a thing, it
would be even cooler if you joined #alpine-linux on irc at some time
and help polishing the Xen port. Because Alpine has all the
embedded-ultra-robustness stuff builtin. It was designed by network
engineers and is much closer to what you want.


Florian


-- 
the purpose of libvirt is to provide an abstraction layer hiding all
xen features added since 2006 until they were finally understood and
copied by the kvm devs.

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users