Re: [Xen-users] hardware recommendations please

[Luke S Crawford <lsc@xxxxxxxxx>]

Chris Fanning <christopher.fanning@xxxxxxxxx> writes:

> I'm about to by a server that will run Xen. DomU's a mostly linux but
> possibly windows.
> I'd like to make sure I buy 'working' hardware.

First, note, while I do have a lot of servers in production and 
a lot of users, most of my experience is with paravirtualized 
guests.  I have not used HVM outside of testing or compiling 
kernels for weird platforms.  paravirtualization is *vastly* superior
to HVM in terms of both speed and reliability, from what I've seen of
HVM mode.  

prgmr.com is almost entirely opteron with the nvidia MCP55-V Pro  You save 
a whole lot of money going with registered ecc ddr2 vs. registered ecc ddr3,
and I really question if a <2.0Ghz nehalam can compete with a 2.2 or 2.4Ghz
shanghai.  (I know it can if you are running intel-designed benchmarks,
but I imagine that doesn't map much to reality.)   Besides, it's a
rare day that I get a complaint about CPU speed that I can't show to be 
actually caused by I/O.   (I give each guest 1vcpu, and dedicate
one hard to the Dom0, so there is almost always a full cpu idle on my system.
I'm thinking about adding 2 vcpus to the higher-ram DomUs.)  

I do have a few single-socket core2quad motherboards with 8GiB unbuffered
ecc.  they work nicely, but 8GiB ram isn't really enough to bother with
for a virtualization server, especially as they eat about 2/3rds as much
power as my low-power 8 core/32GiB ram servers.  

I use the SR1530AHLX  barebone for that, see:
http://serverconfigurator.intel.com/details.aspx?id=1334

but it doesn't have hot-swap drives, and never again will I put something
in production that doesn't have hot-swap drives.  Even without hot swap 
drivers, swapping a disk if it has hot-swap carriers is so much less of
a pain in the ass.  totally worth the cost of the backplanes.

The nvidia disk drivers distributed with the 2.6.18.8-xen xen.org kernel
don't hot-swap, but I put in a sata_sil card to solve that problem,
and everything else seems to work just fine.  both sata_sil and the
nvidia drivers benchmark about as well as the same drives on the 
intel/AHCI hardware.  (back porting the nvidia drivers from RHEL would
be nice, but I'm too dumb to do it myself, and too broke to pay 
someone else to do it, and the sata_sil cards work well and are pretty
cheap.) 

I really like the supermicro kit.  Has anyone tried the 
AMD SR5690 + SP5100 chipsets that Supermicro has just started shipping
a bit ago?  I've been thinking about buying one of these:
http://supermicro.com/Aplus/motherboard/Opteron2000/SR56x0/H8DAi+.cfm

the only socket F motherboard I've seen that wasn't tainted by Nvidia 
shit.  

right now, I have many of the SuperMicro 2 in 1u kits:
http://supermicro.com/Aplus/system/1U/1021/AS-1021TM-T+.cfm

and they work well, though they have the limitation of 2 disks, so you
definitely need to put a sata card in that supports hot-swap

I also found a deal on the 3u 8 bay SuperMicro chassis, so I bought
a few h8dme-2 boards:  

http://supermicro.com/Aplus/motherboard/Opteron2000/MCP55/H8DME-2.cfm

and those work pretty well;  again, if you are in a service provider
position and can't do scheduled reboots you either need to put in a hot
spare or two when you provision the thing, or put in sata cards that 
support hot swap on the 2.6.18.8-xen kernel.   

Uh I also have some tyan thunder n3600r boards, but I can't recommend them 
unless you get them for super cheap and your time is free.  If you re-flash
them to the latest bios from tyan, they support quad-core CPUs, but 
out of 10 boards, 3 of them I never got working, and tyan won't help you
as they are only specced for dual-core.  (Note, I've not had one 
fail in production in a year and a half, other than the 'luke is stupid
and didn't plan on a hot spare drive or add a controller that supports
hot swap'  I had on some of the early ones, which was really a hard
drive problem.  in fact, most of them have not been rebooted during that 
time, it's just 30% of them never get out of burn-in, which is ok. You
are allowed to waste my time in the lab.  You are not allowed to 
crash my customers in production.)  

Speaking of which, if anyone is into the coreboot stuff, the n3600r
(s2912) I have is supposed to be supported, I could probably be talked into
giving away (or selling very cheaply) some of the boards that only
work with dual-core, if you wanna play with coreboot.  

They are the exact same chipset as the supermicros and 99% of the 
socketF kit, so they have the same lack of hot-swap problem, but other
than that work okay. 

Also note, I used to be pretty big on buying motherboards with 16 ram slots,
as 2GiB sticks were a lot cheaper than 4GiB;  but that has changed lately;
with the recent ram prices I'm paying as much or more per gigabyte for
2GiB registered ecc ddr2 as 4GiB registered ecc ddr2.   

I'm watching the price of registered ecc ddr3 every month or so to 
see when (or if)  I will switch to the nehalam platform.  

Also note, I'm on the budget side of things;  if you've got a lot of 
money sloshing around, the nehalam kit is pretty nice;  I haven't 
run it with the xen.org kernel, but I have run it with the RHEL 5.3
xen kernel and it ran quite well.  Only complaint I have with it is the
expensive ram, which is a big thing for me.  (really, it was a big
thing for my client who was buying the nehalam, too-  they'd try to
get by with 16GiB ram.  Hah!  they would have been many times better
off going with the slightly slower opterons and getting 32GiB or more
at the same price.   Having enough ram is enormously more important
than having fast ram.  Obviously, if you can afford enough fast ram,
that's best of all, but finding out you didn't buy enough ram makes your 
SysAdmins very frustrated.)  

-- 
Luke S. Crawford
http://prgmr.com/xen/       -   Hosting for the technically adept
http://nostarch.com/xen.htm -   We don't assume you are stupid.  


_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-users