[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Xen-users] gaming on multiple OS of the same machine?
Sorry for the delay, I meant to reply yesterday but had a lot of trouble writing up an appropriate answer, as I am no specialist when it comes to hardware.
I have respect for anyone willing to invest in experimental technology, but that said I would never recommend something if I didn't know it would work, and work better than an existing alternative.
It is possible to get four HVM's with GPU Passthrough on one machine, but I don't believe it is feasible. I would think it easier to try out a dual setup first, especially if you are new to Xen, and if that works you can reproduce it easily.
It would be a tragedy to invest in amazing hardware equivalent to the cost of two machines only to encounter a fatal problem (whether mentioned here or brand new).
I was new to Xen as of March, it took me two months of tinkering to get a firm enough grasp of Xen that I managed GPU passthrough to one machine. After a month more, I can safely say I could probably produce two to three HVM's with passthrough on my current hardware if I wanted to go that route.
As Andrew stated, there is no "perfect" board, I looked just a couple of months ago and failed and I looked at hundreds of boards from at least a dozen manufacturers. It's clearly possible to setup a four machine passthrough system, but I wouldn't consider it feasible.
If you want to save time, sanity, and possible an expensive investment, I would stick with a more feasible setup and duplicate it for the same cost.
Given that Andrew has successfully setup what you are looking to, it would be wise to take him up on his offer and get hardware & setup details.
*********
Intel Core i7 processors have hyperthreading, a 4 core i7 is the equivalent of an 8-core AMD, so if Andrew has had success with an 8-core AMD CPU and no resource issues, you might be able to get by with a 4-core i7 (a 6 core i7 would be 12 vcores), but I would be hesitant to assign all the cores to HVM's.
*********
Andrew is correct, stay away from NF200, it's not compatible. There are certain required features it doesn't have, which can be checked with "lspci -vvv", but I don't recall what the flags you would be looking for are.
PLX is a great product, I don't deny that in the least, and it is IOMMU compatible which is awesome. However, I believe the flaw comes by design or implementation of PLX on the board.
In my case the Extreme7 shared two PCIe slots with other components using the PLX bridge. While NOT the case with my Extreme4, when I passed one of those "slots" with the Extreme7 it treated the entire bridge as a single PCI device and all components on it were passed via xen-pciback.hide. It was an awful experience, I lost internet, various input devices, and due to the lack of documentation on reversing xen-pciback.hide I was unable to fix it without reinstalling.
I found lspci to be of little help identifying it, as a bridge it just acts as a layer, so you can use tree view (lspci -tv) for details, but it won't say whether the devices are all tied together. My guess is the PLX was bridged to the PCIe slot, and they shared the same space, the tree on the Extreme4 has tons of unused bridge ports in contrast to the Extreme4 which was completely filled.
This experience also leads me to question how the Dual GPU cards will be handled, as a single merged device or separately. Depending on PLX "implementation" it could go either way.
In conclusion, if you get a board with PLX it will pass devices, but be prepared (by which I mean keep regular backups as you progress) to hit a wall or two during testing.
*********
For five machines (4x Windows and Dom0) each to have a keyboard and mouse you need 10 USB ports. If you choose to use PCI Passthrough you will also need five USB PCI Buses.
My Extreme4 comes with four USB PCI Buses onboard (2x USB 3.0, and 2x. USB 2.0), so in that sense you might need a spare PCI slot for a USB Bus Expansion card.
If you choose to use USB Passthrough you won't need as many PCI Buses, but you may not have the ability to pass complex devices that require drivers. For example, I could not get an XBox 360 Wireless Adapter working with USB Passthrough (PCI Passthrough it works fine).
*********
On that same note, I encountered problems with USB 3.0, I believe they were just USB incompatibilities and not Xen or PCI Passthrough related, you won't have any trouble with normal input devices.
My board also has a Marvel SATA controller which did not work with VT-d (DMA READ operation fails), so I had to disable it in UEFI to prevent lengthy and verbose errors at boot time. For me loosing two SATA ports was not a big deal, for others this could be a deal breaker.
*********
While not a limitation per say, you may want to consider buying more hard drives to run the virtual machines on, four machines making IO requests to partitions on the same drive could get sluggish even with PV drivers.
*********
In line with what Andrew said, manufacturers don't list the information you would need to verify any of thus stuff, you essentially have to buy it and try it, or look for other peoples reported successes.
*********
In contrast to what Andrew has reported, my problem is OpenGL for Accelerated "Graphics", oddly Accelerated Video works great. Youtube works with the box checked just fine, and I play a large number of media files and formats without any problems (except for h.264 10-bit video playback, had to change to an alternative codec since Microsoft does not support 10-bit yet).
Direct3D works, I have played Borderlands and The Last Remnant at max settings without lag. To test compatibility I also installed Final Fantasy 7 & 8 which work with related patch files. I am pretty sure the patch files open the game and run it through an OpenGL filter of some kind, so I have to imagine that works. While I am using an AMD/ATI card, Borderlands installed nVidia PhysX which is also working.
The error messages I have encountered appear to be with specific applications, more specifically anything that has to do with image editing software. Photoshop and Blender 3D Sculpting software are the problem, I think Blender uses a Java library. Photoshop runs but says no GPU Acceleration, Blender doesn't open at all. Search results all say it's because they can't be told to use a secondary graphics card.
When I searched for details about either message, they both led me to problems with the software using a Secondary Graphics card. Perhaps I should find it funny that games I can buy for $30 are more compatible with secondary graphics than $400 professional image editing software.
Do keep us updated on your progress with the system, it's always nice to have more people giving this a try.
~Casey
On Tue, May 15, 2012 at 9:48 AM, Andrew Bobulsky <rulerof@xxxxxxxxx> wrote:
Hello Peter,
Answering some of your questions inline below:
On Mon, May 14, 2012 at 5:52 PM, Peter Vandendriessche
< peter.vandendriessche@xxxxxxxxx> wrote:
> Hi Casey,
>
> Thanks a lot for the bunch of information. Some further questions on the PLX
> and the VGA passthrough though.
>
>
> On Sun, May 13, 2012 at 8:42 PM, Casey DeLorme < cdelorme@xxxxxxxxx> wrote:
>>
>> I checked every major manufacturers high end boards, searching for one
>> with the best features to price and was aiming for the most PCIe slots I
>> could get. Pretty much Every single board with more than 3 PCIe (x16) slots
>> came with some form of PCI switch. Most of these PCI switches break IOMMU
>> in one way or another.
>
>
> Yes, indeed, all of the 4-GPU-motherboards I know, have 2 PCI-e x16 slots
> which split to 2 x8 each (so 4 x8 in total). Is this always a fatal problem?
> Is there any easy way to find out if it will be a problem (like from the
> lspci info)?
would be a problem and what wouldn't. I don't know too much in the
way of the nitty-gritty details, but the problem with NF200 PCIe
switched boards, in particular, has something do with the fact that
they route data to the southbridge, rather than the northbridge.
Something about it makes it so that it's not quite "real" PCIe
switching, it's more analogous to PCIe "routing," if that's a fair
term to use. Instead of switching traffic on the existing PCIe bus
that's natively part of the northbridge, they "bolt on" a second PCIe
bus through some kind of trickery that prevents a solid line of
"native PCIe" from being visible to the IOMMU. IIRC, the entire
contents of the NF200 bus itself is a valid passthrough device, but
the individual devices on it are not.... which kinda defeats the
purpose, IMHO. This is all just my recollection of when I looked into
it over a year ago, so I could be wrong about why it doesn't work...
but Casey's advice is sound: stay away from NF200. :)
Which brings me to PLX...
>> The PLX was an entirely different form of problem, it merges device
>> "function" with device identifiers. If you run "lspci" you get a list of
>> your devices, they are identified by "bus:device.function".
>> [...]
>>
>> Armed with this knowledge, here is where you may run into problems:
>> - Finding a board with 4x PCIe x16 Slots not tied to a PCI Switch
>
>
> So, do I understand correct that it will work if and only if there is 1
> "bus" per PCI-e slot?
In my experience, PLX PCIe switches are *fantastic*. They speak
native PCIe, and all the devices that they hook onto the PCIe bus have
a clear logical path to the IOMMU. The PLX switch built into the
Radeon 6990 is the reason you could attach each GPU to a different VM
(though, as I said, the architecture of the card PAST that point is
the reason this wouldn't be of any use to you), and the PLX chip on
the HighPoint RU1144A is the reason that each port on the card can be
assigned to a different VM. Granted, while the purpose of the RU1144A
is to provide four full-bandwidth USB3 ports to a host machine, the
way that HighPoint ended up architecting that goal provided me with
exactly the piece of hardware I needed to work out my USB connectivity
crisis, a purpose I highly doubt the engineers who designed it had
even thought of... but I thank them nonetheless :)
>> - Sparing enough USB ports for all machines input devices
>
>
> What do you mean by sparing here? On a different bus than the PCI-e slots
> for the GPUs?
Specifically, what I think he's getting at is partly what I addressed
above; In order to supply your VMs with USB, the best bet here is to
use PCIe passthrough to hook different controllers to different VMs.
It's highly likely that the architecture of your mainboard will
prohibit pulling this off when you're going to have more than two
different guest OSes which each need their own PCI -> USB adapter.
Normally, because this kind of thing basically *never* matters,
especially seeing as how cost-prohibitive it would be to facilitate it
from the standpoint of motherboard manufacturers, you'll probably not
be able to find a board that will allow you to work out the problem.
Even if you did, I highly doubt it would have four 16x PCIe slots!
Also, since this kind of information is damn near impossible to come
by without getting the hardware in hand and checking it out
yourself... it's probably a lost cause to look for the "perfect"
motherboard.
>> - Limited to around 3GB of RAM per HVM unless you buy 8GB RAM Chips
>
>
> Neither seems a problem (3GB RAM per machine or 8GB RAM chips). The price of
> RAM is fairly linear in its size.
I'll second your notion. 3.5 GB per DomU is more than enough for
pretty much any game. It'd be a different story if you were
virtualizing SQL databases or something, but games are designed to run
on systems that would probably make most enthusiasts on Xen-Users list
cry if they had to use such a machine on a daily basis :D
>> - Will need a 6-core i7 to power all systems without potential resource
>> conflicts
>
> Okay. That rises the price a bit, but it'd still be well worth it.
At LEAST a 6 core chip. If you build the AMD route, just shovel out
for the 8 core chip instead, so you know you're golden. Dom0 won't be
doing too much, and that leaves two cores for each VM that they can
max out without even being capable of trampling eachother.
>> - Encountering bugs nobody else has when you reach that 3rd or 4th HVM
>
>
> This would probably be the real problem, given that I'm new to Xen.
I've seen odd behavior myself. Sometimes, one of the VMs USB ports
won't work. I switch the PCIe assignments around, and it's just a
particular VM that's being fishy... nothing wrong with the actual
controller. It's been weird, but I attribute it to the fact that my
USB controller is plugged into a riser cable. I've seen devices get
funny when you do that. However, it was the only way to plug in four
GPUs AND a PCIe USB controller :P
>> I have never run more than one GPU in my computers before, so I don't know
>> if there is some special magic that happens when you have two or more that
>> they suddenly get even hotter, but I have to imagine that not to be the case
>> unless you're doing some serious overclocking.
>
>
> Depends on their size. Most GPUs are 2 PCI-e slots high (and occupy two of
> those metal plates on the back) and hence plugging 4 of them in leaves no
> space inbetween them, which hinders their air intake. Hence the need for
> watercooling the GPUs in this case.
That would probably raise your costs significantly, but it would
greatly increase your odds for success, I'd say. My biggest
limitations and largest source of problems is that my GPUs eat all 8
slots. Severely limited my routes for solving problems, and I had to
cut the plasic off of a portion of a GPU fan shroud to accomodate a
riser cable. It did end up working, though :)
>> The ASRock Extreme4 Gen3 does have enough PCIe slots that I could connect
>> three GPU's and still have space for a single-slot PCIe device, but I only
>> have a 650W power supply, and have no need for more than one Windows
>> instance.
>
>
> ... and it has a PLX chip. Right? Or is a PLX chip not a fatal problem?
I can't speak to an Intel setup, but for a board, I'd suggest the
Gigabyte GA-990FXA-UD7. I'm testing it right now, and I'm fairly
confident it will work for what you want to do. The tested and
working setup that I've built is on the MSI 890FXA-GD70. That board,
I can 100% guarantee WILL WORK for this purpose (and might require a
BIOS downgrade, YMMV). I'd recommend the newer Gigabyte board though,
just because it's newer, I suppose. Probably supports more CPUs or
something.
>> Secondary pass through works great for gaming, shows the display after the
>> machine boots without any problems, and takes literally no extra effort to
>> setup on your part.
>>
>> Primary passthrough requires custom ATI patching, and what exists may not
>> work for all cards.
>>
>> I began looking into Primary passthrough very recently, because I use my
>> machine for more than just games and ran into a problem. Software like CAD,
>> Photoshop, and 3D Sculpting tools use OpenGL and only work with the primary
>> GPU, which means they either don't run or run without GPU acceleration
>> (slowly).
>
>
> Hmm? I'm not following. Most games also use OpenGL, right? And why would
> OpenGL not support non-primary cards? I know that OpenCL can run on any
> number of GPUs, so it'd surprise me if OpenGL was different. Do you have any
> link where I can read more background on this?
Not sure where to read more on this, but I can share a big caveat:
GPU-Accelerated video decoding DOES NOT WORK. It crashes stuff.
Hardcore. Youtube = fail... until you disable hardware decoding. Not
sure about the OpenGL thing Casey mentions, but I can confirm Direct3D
works like a charm :)
>>
>> A lot to take in, but I hope my answers help a bit. If you have more
>> questions I'll be happy to share what knowledge I can.
>
>
> Certainly, they're of great help. Thanks a lot!
>
> Best regards,
> Peter
I have to admit I'm excited by the prospect of someone else venturing
down this path. I just hope that I can save you some repeated bouts
of banging your head against a wall ;)
If you'd like, I could send you some photos of my setup, or more
detailed hardware specs. I had to do a lot of research to build this
on a reasonable budget... because if it wouldn't have been cheaper
than multiple systems, I know I wouldn't have done it!
Cheers,
Andrew Bobulsky
_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users
|
