|
On 09/24/2012 08:49 AM, Robin Axelsson
wrote:
On 2012-09-24 05:45, ShadesOfGrey wrote:
Sorry
for the late response, I've had a lot to digest.
<snip>
The lack of current information about Xen (and KVM) online has
been frustrating — especially finding the many proof of concept
videos that demonstrated possibilities but offered no real
specifics. Looking for specifics, I sought information from
gaming and enthusiast sites; I figured finding confirmation of
VT-d and AMD-Vi support on such sites would be more likely.
However, I found that wasn't often the case. I did determine
that ASRock motherboards seem to be the most likely to support
VT-d, ASUS least likely (unless equipped with an Intel
'sanctioned' VT-d chipset). I had narrowed my choices to two
motherboards that appear to offer VT-d support and was intending
to contact the manufacturer before purchase. Both choices are a
bit pricey and I've been reconsidering whether I should look to
other motherboards to reduce costs.
Some motherboards support IOMMU even though it is not found in the
user manual or specified on the website. Your best bet is to ask
customer support. A guy posted here that he got it working on an
Intel motherboard that doesn't even have options for it in the
BIOS, so it seems that in some cases it is only up to the CPU.
This is not the case with AMD though as I stated before. I have
bought a couple of Gigabyte GA990FX-UD7 myself, they are stable
and have a good layout. They have support for IOMMU but I haven't
tested it thoroughly enough to fully confirm this although I don't
believe there would be any problem.
I'm aware of this. In fact, I only have anecdotal evidence that the
Gigabyte G1.Sniper 3 has IOMMU (VT-d) support. I intend to query
the manufacturers, seeking confirmation of IOMMU support, of every
motherboard that ends up on my short list. I may include other Z77
motherboards from MSI and Gigabyte, since there is concrete evidence
that the Z77 chipset does support IOMMU. I did focus, however, on
those motherboards I had some inkling could support IOMMU. Now I'm
trying to expand my selection process to include less expensive
options.
It surprises me that ASRock and ASUS are so
different. ASRock is, or at least used to be a subsidiary of ASUS
so there shouldn't be that much difference between them.
Same here. But I guess things changed after ASRock was spun-off.
<snip>
This is precisely the kind of information I was looking for from
the threads I started on Ars Technica. It's just unfortunate
that FLR and D3 D0 support aren't often found in the tech specs
of must expansion hardware. However, now that I know what to
ask, I'll try contacting hardware manufacturers prior to
purchasing any expansion hardware. Thank you!
D3 and D0 are power states defined for devices in the ACPI
specification and can be used to control the supply voltage (Vcc)
to PCI and PCIe devices. You can find more information about it
here for example:
http://en.wikipedia.org/wiki/Advanced_Configuration_and_Power_Interface
---------------
Device states
The device states D0-D3 are device-dependent:
- D0 Fully On is the operating state.
- D1 and D2 are intermediate power-states whose
definition varies by device.
- D3 Off has the device powered off and
unresponsive to its bus.
---------------
So, either it works for a certain type of hardware or it doesn't
and I wouldn't expect a vendor to state this "support" in the
specifications since it isn't a "feature" in and of itself if you
get me. But maybe this will change and maybe FLR support will
become more widespread.
I did read that Wikipedia entry after your referenced excerpt. To
my mind, if a manufacturer claims support for the ACPI or PCIe spec
and doesn't implement certain portions of those specs(some are
optional after all), said manufacturer has an obligation to make
that clear to your customers and users. But then, that's just me.
Anyway, this information gives me what I need to ask the right
questions of tech support before purchasing.
<snip>
From everything I've read, solutions that rely on any form of
remote display protocols would be limited to a subset of
Direct3D functions. Furthermore, these would vary from one
implementation to another, thus making them far less attractive
for gaming than VGA passthrough... Well, in my opinion anyway.
VirtualBox's seamless mode is pretty nifty. But it's a Type 2
Hypervisor and relies on paravirtualized drivers that also
suffer from the same limitations as remote display protocols.
It's great for most things, but gaming is not one of them. And
I'm speaking from personal experience. Though I haven't used
them myself, the same would seem to hold true of Parallel's and
VMWare's 'Workstation' offerings. At least, as far as I've
gathered.
FYI, the Type 1 Hypervisors from Parallel's and VMWare* are
priced waaayyy outside my budget.
I understand that you want full 3D functionality for Windows
gaming but maybe you'll find the subset of 3D functionality for
the Linux machine acceptable. I have looked into VirtualGL and
with TurboVNC, you might get a pretty decent desktop environment
and it seems like most of the features are there already. It
appears that the 3D is rendered by hardware/GPU before it is
streamed through VNC or Spice. So it seems that you would need
another GPU for that. You can find more info on VirtualGL here:
http://www.virtualgl.org/
Upon further investigation, the only option that is available to
remotely translate Windows 3D apps is Microsoft's own RemoteFX. In
which case, Microsoft products would be the foundation of my
software stack... Something I'm trying to avoid.
Also the line between a type 1 and type 2 hypervisor tend to get a
bit blurry. The point with type 1 is that it has access to ring-0
so that it can get access directly to the hardware to be passed
through to the guests (I did confuse 'host' and 'guest' in my
prior post). It also doesn't need to ask the host OS for
permission in the same way as a type 2 hypervisor which is likely
to give performance advantages in some cases.
However, even a type 2 hypervisor, although it is run as an
application inside the OS can get "type 1" like privileges. By
patching into the kernel and/or using special "dummy drivers" for
hardware to be shared with VMs you can achieve pretty much the
same thing, ergo it is no longer clear whether the hypervisor is a
type 1 or type 2.
There is an article about it from the old IBM Mainframe days but I
can't seem to find it.
You might be right, the performance of a Type 2 Hypervisor may be
sufficient. Regardless, I'd still have roughly the same hardware
requirements as if I were going to use a Type 1 Hypervisor. I'd
still need the same CPU, RAM, GPU, and storage requirements as I've
already put forth. I could omit the potentially necessary
additional components for use with a Type 1 Hypervisor. Things like
USB controller, NIC, or sound card. But those costs would be
replaced with the cost of licensing the Type2 Hypervisor.
In which case, I don't really gain or lose anything by experimenting
with Type 1 Hypervisors... At any rate, I could compare and
contrast the performance of Type 1 and Type 2 Hypervisors using
Paralles' and VMWare's trialware. It might take a good long while,
but it would be an adventure.
*I only found out about VMWare's 'free' vSphere after I'd
written this response.
<snip>
Also, it is highly recommended that you
use ECC RAM for such applications and it doesn't hurt to
dedicate a few gigs of it to the ZFS as RAM is used for cache.
The good news is that most motherboards with good chipsets
support ECC RAM even though you might not find anything about
it in the user manuals.
Again, thanks for the thorough explanation. This gives me a
great deal to think about. The more I learn about ZFS, the less
appealing it becomes. And by that I mean the confusion over
which version of ZFS is in what OS? And just how well maintained
the OSes supporting ZFS are? Now I have additional hardware
considerations to keep in mind that may (or may not) make the
cost of ZFS RAID-Z pool comparable to a hardware RAID5/6
solution anyway. Do you have any suggestions as to which of LSI
HBAs I should be considering? I haven't found an HCL for ZFS in
my searches.
Out of curiosity — and if you would happen to know — do you
think what you suggest about the HBA and SAS drives for ZFS also
applies to Btrfs? I'm assuming it would, but I'd appreciate some
confirmation.
It's funny how the "I" in RAID never really seems to apply...
Especially since it looks more and more like using ZFS or Btrfs
will require I commit myself, from the start, to one or the
other and a discrete HBA. Transitioning from an integrated SATA
controller(s) and mdadm seems rather impractical. If I
understand what's involved in doing so correctly. It may turn
out that anything other than mdadm is price prohibitive.
I don't think you will have a problem with getting ZFS to run and
if that's your only goal then you don't need to be very picky with
your choice of hardware. I find ZFS pretty easy and handy to use.
I has really great functionality and I don't have many bad things
to say about it so far. ZFS is a filesystem (along with a couple
of software tools to administrate it) just like EXT4 or NTFS so
hardware support depends on the platform it runs on.
But the point with using ZFS is to get maximum protection against
data corruption and that's where the selection of hardware gets
limited and there are "best practices" set up to achieve that. I
have not tested ZFS on any other platform than on OpenSolaris and
OpenIndiana but I do know that it is well implemented on that
platform and more mature there than on any other (non-solaris)
platform. Another advantage with the OSOL/OI platform is that the
CIFS functionality is implemented in the kernel space and not in
the userland which will give advantages performance wise if you
intend to share files with other windows computers. (I don't deny
that Samba is pretty good on Linux too. There are some benchmarks
on the phoronix website comparing samba with NFS and they are in
favor of Samba on those benchmarks...) The second best
implementation is found with FreeBSD and it is probably fairly
mature but I haven't tested it myself and some people have run
into problems with it in the past. The Linux version is probably
merely at infancy stage and likely not yet mature enough for
regular use. It is probably not as "bad" as btrfs though. There is
quite a bit of information about it on the phoronix.com website
(and probably also at lwn.net):
http://www.phoronix.com/scan.php?page=news_item&px=MTE4Nzc
My goal wasn't just to experiment with Btrfs, mdamd, or ZFS, if
that's what you meant by, "with getting ZFS to run and if that's
your only goal". I actually intend to use it 'in production'. In
fact, the secondary role (Linux desktop being primary) for my
proposed Virtualization rig is as file server (incl. httpd).
Windows gaming is a tertiary or quaternary concern for me. So,
finding out that the "best practices" for implementing ZFS include
hardware I hadn't anticipated, is a bit off-putting.
The diversity in ZFS implementations is what I meant about being
confused. I'm not as familiar with the underlying platforms that
utilize ZFS (other than Linux, and not with ZFS in use). Without
that familiarity, it makes it more difficult to gauge which of those
platforms would suit my purposes. For example, I had read a little
bit about the degraded CIFS performance on FreeBSD and Linux due to
their reliance on Samba (it residing in user-space being the issue).
BTW, I meant to ask how you came to the conclusion that ECC RAM is
supported on desktop motherboards? It's always been my
understanding that you could use ECC RAM on such hardware, but there
was no added benefit.
A search there on ZFS will give more articles. The
latest official version of ZFS is 28 and is probably implemented
in both Linux, and FreeBSD by now. Later versions have been
released since Oracle killed the OpenSolaris project and can be
found with the commercial closed-source Solaris platform that is
supplied by Oracle. Things have happened since Oracle pulled the
plug on OSOL project and leading developers behind the ZFS project
such as Jeff Bonwick left Sun (after the acquisition by Oracle)
and joined up with the Illumos team instead. So you cannot
determine the stability of ZFS and ZPOOL merely by looking into
the version number unfortunately and I wouldn't expect the FreeBSD
implementation to be as stable as the Solaris implementation. It
just takes time for the implementation to mature and the bugs to
be weeded out and it just happens to have been around for
Solaris/OpenSolaris/Illumos for much longer than the other
platforms and the Solaris/Illumos version also happens to get
first dibs on the features. Among the Illumos people there is an
ambition to drop the version numbering altogether and instead talk
about available features.
One of the two ZFS implementations on Linux has reached version 28,
the other is at version 23 and seems to be abandoned or stagnant
(last release was May 2011). I'm not quite sure what the status of
ZFS on FreeBSD is. From this table
on Wikipedia, ZFS is at version 28. However the table notes that
there is no CIFS or iSCSI support, which I'll try to independently
confirm. And, as you say, the fact that ZFS (and Solaris as a
whole) has essentially been forked, with the re-consolidation of
Solaris as closed-source, just adds to the confusion. It's possible
any given build of Illumos' version 28 of ZFS could have features or
bug fixes no present in Oracle's version 33.
The recommendation to use SAS hard drives is not so
much about the quality of the hard drives themselves as it is
about the SAS protocol. The SAS protocol simply handles SCSI
transport commands in a better and more reliable manner than do
SATA. I believe any decent SAS drive would do. As for HBAs I wrote
a list with LSI based hardware a while ago here:
https://www.illumos.org/boards/1/topics/572
the thing is that a lot of OEMs such as IBM, HP, Cisco,
Fujitsu-Siemens, Dell, ... supply their branded HBAs with LSI
circuitry on them. What hardware to choose depends on what you're
looking for. If you want an 8-port controller I would go for Intel
SASUC8I or LSI SAS3801E-R. If you want SAS/SATA3 with 6.0 Gb/s
then LSI's LSI SAS3801E-R series cards would be a better choice. I
don't know what OEMs have come up with in the SATA3 department
since I wrote that list but the chips to look for in that case are
the LSI MegaRAID 2004/2008/2016e depending on how many ports you
want.
If you want to read a further discussion about reliability of
different RAID setups I made a post about this in the following
thread (last post):
http://communities.intel.com/thread/25945
The cost of the drives is marginally higher than the drives I had
budgeted, but including the cost of an SAS HBA is problematic. I
hadn't expected purchasing an HBA from the very start. I'd hoped to
deffer such a purchase for a bit, but you are saying that would be
ill-advised. How likely is it I would encounter problems using SATA
drives in the interim?
Have you seen any price/feature advantage to reseller versions of
LSI OEM products? I would think the offerings from Cisco, Dell, HP,
and IBM would come at a premium as opposed to purchasing LSI branded
hardware. Anyway, I thought there might be specific models from
particular manufacturers to meet the "best practices" for ZFS.
However, from your suggestions, I gather it really doesn't matter as
long as the HBA is Intel or LSI?
I have seen discussions in support of file system level redundancy
like that of ZFS, precisely for the reason you cited. I guess I can
add your link to the pro-FS pile of reference material.
I wonder, should we perhaps move the discussion of ZFS to private
email? It is a bit off-topic.
|
