Re: [Xen-users] gaming on multiple OS of the same machine?

[Andrew Bobulsky <rulerof@xxxxxxxxx>]

Hello Peter,

Answering some of your questions inline below:

On Mon, May 14, 2012 at 5:52 PM, Peter Vandendriessche
<peter.vandendriessche@xxxxxxxxx> wrote:
> Hi Casey,
>
> Thanks a lot for the bunch of information. Some further questions on the PLX
> and the VGA passthrough though.
>
>
> On Sun, May 13, 2012 at 8:42 PM, Casey DeLorme <cdelorme@xxxxxxxxx> wrote:
>>
>> I checked every major manufacturers high end boards, searching for one
>> with the best features to price and was aiming for the most PCIe slots I
>> could get.  Pretty much Every single board with more than 3 PCIe (x16) slots
>> came with some form of PCI switch.  Most of these PCI switches break IOMMU
>> in one way or another.
>
>
> Yes, indeed, all of the 4-GPU-motherboards I know, have 2 PCI-e x16 slots
> which split to 2 x8 each (so 4 x8 in total). Is this always a fatal problem?
> Is there any easy way to find out if it will be a problem (like from the
> lspci info)?

I couldn't say for sure if there's any great way to identify what
would be a problem and what wouldn't.  I don't know too much in the
way of the nitty-gritty details, but the problem with NF200 PCIe
switched boards, in particular, has something do with the fact that
they route data to the southbridge, rather than the northbridge.
Something about it makes it so that it's not quite "real" PCIe
switching, it's more analogous to PCIe "routing," if that's a fair
term to use.  Instead of switching traffic on the existing PCIe bus
that's natively part of the northbridge,  they "bolt on" a second PCIe
bus through some kind of trickery that prevents a solid line of
"native PCIe" from being visible to the IOMMU.  IIRC, the entire
contents of the NF200 bus itself is a valid passthrough device, but
the individual devices on it are not.... which kinda defeats the
purpose, IMHO.  This is all just my recollection of when I looked into
it over a year ago, so I could be wrong about why it doesn't work...
but Casey's advice is sound: stay away from NF200. :)

Which brings me to PLX...

>> The PLX was an entirely different form of problem, it merges device
>> "function" with device identifiers.  If you run "lspci" you get a list of
>> your devices, they are identified by "bus:device.function".
>> [...]
>>
>> Armed with this knowledge, here is where you may run into problems:
>> -  Finding a board with 4x PCIe x16 Slots not tied to a PCI Switch
>
>
> So, do I understand correct that it will work if and only if there is 1
> "bus" per PCI-e slot?

In my experience, PLX PCIe switches are *fantastic*.  They speak
native PCIe, and all the devices that they hook onto the PCIe bus have
a clear logical path to the IOMMU.  The PLX switch built into the
Radeon 6990 is the reason you could attach each GPU to a different VM
(though, as I said, the architecture of the card PAST that point is
the reason this wouldn't be of any use to you), and the PLX chip on
the HighPoint RU1144A is the reason that each port on the card can be
assigned to a different VM.  Granted, while the purpose of the RU1144A
is to provide four full-bandwidth USB3 ports to a host machine, the
way that HighPoint ended up architecting that goal provided me with
exactly the piece of hardware I needed to work out my USB connectivity
crisis, a purpose I highly doubt the engineers who designed it had
even thought of... but I thank them nonetheless :)


>> -  Sparing enough USB ports for all machines input devices
>
>
> What do you mean by sparing here? On a different bus than the PCI-e slots
> for the GPUs?

Specifically, what I think he's getting at is partly what I addressed
above; In order to supply your VMs with USB, the best bet here is to
use PCIe passthrough to hook different controllers to different VMs.
It's highly likely that the architecture of your mainboard will
prohibit pulling this off when you're going to have more than two
different guest OSes which each need their own PCI -> USB adapter.
Normally, because this kind of thing basically *never* matters,
especially seeing as how cost-prohibitive it would be to facilitate it
from the standpoint of motherboard manufacturers, you'll probably not
be able to find a board that will allow you to work out the problem.
Even if you did, I highly doubt it would have four 16x PCIe slots!
Also, since this kind of information is damn near impossible to come
by without getting the hardware in hand and checking it out
yourself... it's probably a lost cause to look for the "perfect"
motherboard.

>> -  Limited to around 3GB of RAM per HVM unless you buy 8GB RAM Chips
>
>
> Neither seems a problem (3GB RAM per machine or 8GB RAM chips). The price of
> RAM is fairly linear in its size.

I'll second your notion.  3.5 GB per DomU is more than enough for
pretty much any game.  It'd be a different story if you were
virtualizing SQL databases or something, but games are designed to run
on systems that would probably make most enthusiasts on Xen-Users list
cry if they had to use such a machine on a daily basis :D

>> -  Will need a 6-core i7 to power all systems without potential resource
>> conflicts
>
> Okay. That rises the price a bit, but it'd still be well worth it.

At LEAST a 6 core chip.  If you build the AMD route, just shovel out
for the 8 core chip instead, so you know you're golden.  Dom0 won't be
doing too much, and that leaves two cores for each VM that they can
max out without even being capable of trampling eachother.

>> -  Encountering bugs nobody else has when you reach that 3rd or 4th HVM
>
>
> This would probably be the real problem, given that I'm new to Xen.

I've seen odd behavior myself.  Sometimes, one of the VMs USB ports
won't work.  I switch the PCIe assignments around, and it's just a
particular VM that's being fishy... nothing wrong with the actual
controller.  It's been weird, but I attribute it to the fact that my
USB controller is plugged into a riser cable.  I've seen devices get
funny when you do that.  However, it was the only way to plug in four
GPUs AND a PCIe USB controller :P

>> I have never run more than one GPU in my computers before, so I don't know
>> if there is some special magic that happens when you have two or more that
>> they suddenly get even hotter, but I have to imagine that not to be the case
>> unless you're doing some serious overclocking.
>
>
> Depends on their size. Most GPUs are 2 PCI-e slots high (and occupy two of
> those metal plates on the back) and hence plugging 4 of them in leaves no
> space inbetween them, which hinders their air intake. Hence the need for
> watercooling the GPUs in this case.

That would probably raise your costs significantly, but it would
greatly increase your odds for success, I'd say.  My biggest
limitations and largest source of problems is that my GPUs eat all 8
slots.  Severely limited my routes for solving problems, and I had to
cut the plasic off of a portion of a GPU fan shroud to accomodate a
riser cable.  It did end up working, though :)


>> The ASRock Extreme4 Gen3 does have enough PCIe slots that I could connect
>> three GPU's and still have space for a single-slot PCIe device, but I only
>> have a 650W power supply, and have no need for more than one Windows
>> instance.
>
>
> ... and it has a PLX chip. Right? Or is a PLX chip not a fatal problem?

I can't speak to an Intel setup, but for a board, I'd suggest the
Gigabyte GA-990FXA-UD7.  I'm testing it right now, and I'm fairly
confident it will work for what you want to do.  The tested and
working setup that I've built is on the MSI 890FXA-GD70.  That board,
I can 100% guarantee WILL WORK for this purpose (and might require a
BIOS downgrade, YMMV).  I'd recommend the newer Gigabyte board though,
just because it's newer, I suppose.  Probably supports more CPUs or
something.

>> Secondary pass through works great for gaming, shows the display after the
>> machine boots without any problems, and takes literally no extra effort to
>> setup on your part.
>>
>> Primary passthrough requires custom ATI patching, and what exists may not
>> work for all cards.
>>
>> I began looking into Primary passthrough very recently, because I use my
>> machine for more than just games and ran into a problem.  Software like CAD,
>> Photoshop, and 3D Sculpting tools use OpenGL and only work with the primary
>> GPU, which means they either don't run or run without GPU acceleration
>> (slowly).
>
>
> Hmm? I'm not following. Most games also use OpenGL, right? And why would
> OpenGL not support non-primary cards? I know that OpenCL can run on any
> number of GPUs, so it'd surprise me if OpenGL was different. Do you have any
> link where I can read more background on this?

Not sure where to read more on this, but I can share a big caveat:
GPU-Accelerated video decoding DOES NOT WORK.  It crashes stuff.
Hardcore.  Youtube = fail... until you disable hardware decoding.  Not
sure about the OpenGL thing Casey mentions, but I can confirm Direct3D
works like a charm :)

>>
>> A lot to take in, but I hope my answers help a bit.  If you have more
>> questions I'll be happy to share what knowledge I can.
>
>
> Certainly, they're of great help. Thanks a lot!
>
> Best regards,
> Peter

I have to admit I'm excited by the prospect of someone else venturing
down this path.  I just hope that I can save you some repeated bouts
of banging your head against a wall ;)

If you'd like, I could send you some photos of my setup, or more
detailed hardware specs.  I had to do a lot of research to build this
on a reasonable budget... because if it wouldn't have been cheaper
than multiple systems, I know I wouldn't have done it!

Cheers,
Andrew Bobulsky

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users