Re: [Xen-users] Recommendations for Virtulization Hardware

[ShadesOfGrey <shades_of_grey@xxxxxxxxxxxxx>]

Sorry for the late response, I've had a lot to digest.

On 09/21/2012 11:22 AM, Robin Axelsson wrote:
> If you want to be able to use PCI and VGA passthrough you basically 
> need to make sure that your hardware supports either AMD-Vi (formerly 
> known as AMD-IOMMU) or Intel VT-d extensions. In the Intel case it 
> limits your choice of Motherboard (it must be supported in the BIOS) 
> and CPU. In the AMD case it limits only your choice of motherboard. A 
> good start is to check out one of these pages:
> http://wiki.xensource.com/xenwiki/VTdHowTo
> http://wiki.xen.org/wiki/VTd_HowTo
>
> A word of warning here is that parts of the documentation is somewhat 
> dated. You can also communicate with e.g. Gigabyte, Asus or ASRock 
> customer support and ask them if a particular motherboard supports 
> these extensions. Most motherboards also have downloadable user 
> manuals, if the BIOS settings in those pages shows options to 
> enable/disable VT-d or AMD-Vi/IOMMU extensions then you will be ok 
> with that motherboard.
The lack of current information about Xen (and KVM) online has been 
frustrating — especially finding the many proof of concept videos that 
demonstrated possibilities but offered no real specifics. Looking for 
specifics, I sought information from gaming and enthusiast sites; I 
figured finding confirmation of VT-d and AMD-Vi support on such sites 
would be more likely. However, I found that wasn't often the case. I did 
determine that ASRock motherboards seem to be the most likely to support 
VT-d, ASUS least likely (unless equipped with an Intel 'sanctioned' VT-d 
chipset). I had narrowed my choices to two motherboards that appear to 
offer VT-d support and was intending to contact the manufacturer before 
purchase. Both choices are a bit pricey and I've been reconsidering 
whether I should look to other motherboards to reduce costs.
> The other thing is choice of GPU for VGA passthrough and it is 
> preferable that the GPU supports FLR or Function Level Reset as it is 
> called. Thing is that the hardware needs to be reset somehow as it is 
> passed through to the host. This is best done with FLR and nVidia is 
> known to supply firmware patches for some of their Geforce cards with 
> this support and it is said to be supported by default with their 
> Quadro cards. FLR is not the only way to reset a PCI device, a reset 
> could be trigged through the ACPI power management framework by 
> temporarily cutting power to the affected PCI slot. These reset 
> methods are called d3d0 and bus reset. The question however, is if 
> this works on PCI cards that use auxiliary power directly from the 
> PSU. There is a pdf document on the VMWare website 
> (http://www.vmware.com/files/pdf/techpaper/vsp_4_vmdirectpath_host.pdf) about 
> this:
> -----------------------
> Reset Method
>
> Possible values for the reset method include flr, d3d0, link, bridge, 
> or default.
> The default setting is described as follows. If a device supports 
> function level reset (FLR), ESX always uses FLR. If the device does 
> not support FLR, ESX next defaults to link reset and bus reset in that 
> order. Link reset and bus reset might prevent some devices from being 
> assigned to different virtual machines, or from being assigned between 
> the VMkernel and virtual machines. In the absence of FLR, it is 
> possible to use PCI Power Management capability (D3 to D0 transitions) 
> to trigger a reset. Most of the Intel NICs and various other HBAs 
> support this mode.
> -----------------------
>
>
> There are indications from people that d3d0 also work with PCI cards 
> that take power from auxiliary inputs. I suggest that you take a look 
> at the following youtube clip and read the comments there:
> http://www.youtube.com/watch?v=Gtmwnx-k2qg
>
> So it seems that it works although it may be a bit more quirky. It 
> doesn't hurt to take that discussion (particularly about FLR support) 
> with nVidia and/or AMD.
This is precisely the kind of information I was looking for from the 
threads I started on Ars Technica. It's just unfortunate that FLR and D3 
D0 support aren't often found in the tech specs of must expansion 
hardware. However, now that I know what to ask, I'll try contacting 
hardware manufacturers prior to purchasing any expansion hardware. Thank 
you!
> When it comes to virtualization, the technology has come very far, but 
> it is still lacking considerably when it comes to sharing GPUs and 
> also to some degree when it comes to sharing I/O devices (especially 
> when you intend to run many virtual machines on a single system). The 
> GPU today consists of three types of components; the processing unit, 
> graphics memory and the video output unit/adapter and it is not clear 
> as to how to share these components seamlessly between the host and 
> virtual machines with minimal overhead. Whereas there are VT-x 
> extensions that allows you to pretty seamlessly share CPU cores 
> between VMs and the host there are currently none for the processing 
> unit. It is also not clear how the hardware can assist with sharing 
> TV/monitor screen estate between machines with all 3D effects such as 
> Aqua for Win7 and the whatnot enabled for all machines. Especially 
> when considering the dynamics of plugging and unplugging computer 
> monitors to multiport/eyefinity graphics cards and the ability to 
> change screen resolution. Things are improving for sure and a lot of 
> research is likely going into this. I don't know what's happening in 
> the GPU frontline but I know that the next thing with passthrough is 
> the SR-IOV that allows PCI units to present several virtual instances 
> of oneself to several virtual machines. It's a cool thing, I recommend 
> further reading about this here:
> http://www.intel.com/content/www/us/en/pci-express/pci-sig-sr-iov-primer-sr-iov-technology-paper.html 
>
> http://blog.scottlowe.org/2009/12/02/what-is-sr-iov/
That is fascinating. Extending virtualization to expansion hardware via 
SR-IOV, sure would make the kind of setup I'm attempting a lot easier. 
However, if I can replicate what I've seen in proof of concept videos 
(namely Casey DeLorme's), I think that will meet my needs for now. As it 
stands, I initially intend to reserve any discrete GPU(s) for Windows 
and rely on an integrated GPU for all other VMs using PV drivers 
(wherever possible). Afterward, I want to experiment with re-assigning 
the whatever discrete GPU(s) for GPGPU functions under a Linux VM 
whenever the GPU is not going to used for gaming (if at all possible).
> It is likely to take a few years before something useful will come out 
> of it. In the meanwhile, unless you want to use several GPUs which 
> might not be a bad thing as a lot of monitors these days have several 
> inputs, you can resort to using a remote desktop client to integrate 
> one machine with another. Virtualbox for example use RDP through which 
> you can interact with your virtual machine. In a similar manner you 
> can set up a VNC server on your Linux host and establish a connection 
> to it through your Windows VM. You will not get full 3D functionality 
> (such as Aqua) through the client although there is a growing support 
> for it through VirtualGL extensions that are coming to VNC and perhaps 
> the Spice protocol. But some clients might even allow for seamless 
> mode that lets you mix Linux and Windows windows on the same desktop 
> like this for example:
> http://i.techrepublic.com.com/blogs/seamless.png
> http://www.youtube.com/watch?v=eQr8iI0yZH4
>
>
> Just keep in mind that this is still a little bit of uncharted 
> territory so there may be a few bumps on the way and it may not work 
> as smooth as you would desire.
From everything I've read, solutions that rely on any form of remote 
display protocols would be limited to a subset of Direct3D functions. 
Furthermore, these would vary from one implementation to another, thus 
making them far less attractive for gaming than VGA passthrough... Well, 
in my opinion anyway.
VirtualBox's seamless mode is pretty nifty. But it's a Type 2 Hypervisor 
and relies on paravirtualized drivers that also suffer from the same 
limitations as remote display protocols. It's great for most things, but 
gaming is not one of them. And I'm speaking from personal experience. 
Though I haven't used them myself, the same would seem to hold true of 
Parallel's and VMWare's 'Workstation' offerings. At least, as far as 
I've gathered.
FYI, the Type 1 Hypervisors from Parallel's and VMWare* are priced 
waaayyy outside my budget.

*I only found out about VMWare's 'free' vSphere after I'd written this 
response.
> I see that your demands are somewhat multifaceted. I believe that you 
> also want to use diffent services such as using your machine as a file 
> server with the possible intention of using filesystems such as ZFS. 
> If you do, you should be careful with your selection of hardware for 
> these particular purposes. If you want to get full protection against 
> data corruption from ZFS, your choice of hardware gets rather limited 
> when it comes to choice of hard drives, host bus adapter and network 
> controller. The most stable implementation of ZFS is found with 
> Illumos based operating systems (such as OpenIndiana, SmartOS, OmniOS, 
> Belenix etc) or Solaris if you choose to download it from Oracle's 
> website. With these operating systems you are most likely to want to 
> use hardware that has certified drivers for it. That way you are less 
> likely to run into problems later on. That implies that you will be 
> limited to choosing Intel based network adapters and LSI based SAS 
> controllers. There should be _no_ hardware RAID functionality in the 
> SAS controller that merely should be run in IT mode (or 
> Initiator-Target mode). That requires the LSI controller to be flashed 
> with IT firmware in most cases. The objective here is to make sure 
> that _all_ errors that might occur with the hard drives are reported 
> all the way to the software level and that nothing is concealed of 
> obfuscated by internal error handling in the hardware. It is therefore 
> recommended to use SAS hard drives instead of S-ATA (which also are 
> fully compatible with SAS controllers). SAS hard drives are not much 
> more expensive than similar SATA drives and you get a higher 
> reliability out of them. It is also recommended to have at least two 
> drive redundancy simply because if one drive is dead and you swap it, 
> it is not uncommon that another drive dies in the rebuild process of 
> the RAID cluster because of the added strain the rebuild process (or 
> 'resilvering' as it is called in Solaris terms) put on the drives. Of 
> course, the system should communicate directly to the hard drive 
> hardware and not be obfuscated by some virtual abstraction layer in 
> between which means that you either run ZFS on the metal or through 
> PCI passthrough of the SAS (and perhaps also network) adapters. Also, 
> it is highly recommended that you use ECC RAM for such applications 
> and it doesn't hurt to dedicate a few gigs of it to the ZFS as RAM is 
> used for cache. The good news is that most motherboards with good 
> chipsets support ECC RAM even though you might not find anything about 
> it in the user manuals.
Again, thanks for the thorough explanation. This gives me a great deal 
to think about. The more I learn about ZFS, the less appealing it 
becomes. And by that I mean the confusion over which version of ZFS is 
in what OS? And just how well maintained the OSes supporting ZFS are? 
Now I have additional hardware considerations to keep in mind that may 
(or may not) make the cost of ZFS RAID-Z pool comparable to a hardware 
RAID5/6 solution anyway. Do you have any suggestions as to which of LSI 
HBAs I should be considering? I haven't found an HCL for ZFS in my searches.
Out of curiosity — and if you would happen to know — do you think what 
you suggest about the HBA and SAS drives for ZFS also applies to Btrfs? 
I'm assuming it would, but I'd appreciate some confirmation.
It's funny how the "I" in RAID never really seems to apply... Especially 
since it looks more and more like using ZFS or Btrfs will require I 
commit myself, from the start, to one or the other and a discrete HBA. 
Transitioning from an integrated SATA controller(s) and mdadm seems 
rather impractical. If I understand what's involved in doing so 
correctly. It may turn out that anything other than mdadm is price 
prohibitive.
> I admire your persistence with pursuing this undertaking and wish you 
> the best of luck with it!
> Robin.
Thanks. I've invested too much time in research to not at least make the 
attempt. Besides, if all else fails, I can fallback to a two box 
solution. That is, if I can get my hypothetical virtualization box to 
fit in my budget envelope...

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users