Re: [Xen-users] VGA passthrough

[Matthias <matthias.kannenberg@xxxxxxxxxxxxxx>]

Hi,

okay, thank you for the additional information!

First of all some advices on your configs (these might not help you
with the vga passthrough but should eliminate other problems so you
have a good foundation):

1) You used mixed hdd types and not in any sequential order.. just use
either xvda or hda (xvda should be preferred) and increment step by
step a,b,c,d,e etc..
2) You try to pass the /dev/sr0 to both domUs at the same time.. don't
know it that is supported.. (actually i never got the DVD-Rom running
perfectly, so you maybe just want to leave that out entirely if you
don't need it)
3) You use ioemu for the network device. Since you have a hvm anyway
you can leave that out and might also see a performance gain on that..
4) You might consider switching to LVM instead of diskimages.. should
give you a I/O performance boost

Okay, now to the kernel:

Is this a vanilla one? Config looks good from what i can tell from the
tutorial you linked. I actually had problems getting the vanilla
kernel to run with vga passthrough at all and use an openSuse one ever
since cause they ship some xen patches with it.. but if vga
passthrough works for you (at least for one card) you should be fine,
i guess..

Xen-Version:

I think xen causes your problem here, and as i said before, dual vga
passthrough is currently broken for me, too. But first of all: You use
gfx_passthrough=1 in your config. Have you tried it with =0 so the
domUs only get the card as secondary? In my experiance this is much
more gracefully and easier to get running..

Now to what i think is your actual problem: Using passthrough with xl
as toolstack requires you to calculate some RAM-Ranges (Point 5 in the
tutorial you've referenced). My guess is that this formula / method
only works for one vga card. You basically (as far as I understand it,
I haven't really looked into it, i explain why in a moment) tell xen /
the kernel that if you are doing vga passthrough use THESE specific
ranges for the PCI devices and since this is now hardcoded, when you
try the second card you use the ranges from the first one, which isn't
good for the first domU because their graphic memory is overwritten
from something other then the domU, and it is not good for the second
domU because pci device and pci associated memory do not fit
together.. so in the end everything goes haywire..

What I would suggest to you is that you switch toolstacks from xl to
xm. On the one hand, xm is much more stable, especially with
experimental features like vga passthrough, and on the other hand it
does not require that you hardcode your VGA Bioses into xen or
calculated some adress ranges. The last thing I tested dual vga
passthrough successfully with is rev 23190 in xen-testing. I want to
start searching for the broken changeset in xen-testing next week, but
since then unfortunatly i can only advice to use this ~3 month old
change set. So if you want:

1) uninstall your current xen so there are no leftovers
2) cd /somewhere/you/want/to/compile/xen
3) hg clone -r 23190:5a00ccfc6391 http://xenbits.xen.org/hg/xen-4.1-testing.hg/
4) cd xen-4.1-testing.hg
5) make -j1 xen && make -j1 tools && make -j1 stubdom
6) make install-xen
7) make install-tools PYTHON_PREFIX_ARG=
8) make install-stubdom
9) update your grub entry and reboot so the new xen is loaded
10) if it is not started yet:
10a) /etc/init.d/xencommons start
10b) /etc/init.d/xend start
11) now you can start your domU with 'xm start domU'.. basically
everything should work the same but instead of using xl as the xen
command, use xm

P.S: with xm you can throw out the 00:01 device from pci too as well
as ignore the start script from the tutorial

Hope this works for you :)



2012/6/28 joe harper <jharper.research@xxxxxxxxx>:
> Since this is being cc'd to xen's mailing list I will include a few more
> details.
>
> I am trying to assign 1 card per domU.  Essentially I am trying to recreate
> Ubisoft's experiment. http://www.youtube.com/watch?v=Gtmwnx-k2qg
>
> Here is my configuration:
>
> I followed David Techer's tutorial for VGA passthrough with Nvidia graphics
> cards.
> http://www.davidgis.fr/blog/index.php?2011/12/07/860-xen-42unstable-patches-for-vga-pass-through
>
> One difference between my setup and his it that I am using Fedora 14 as my
> dom0.
>
> I am using a tyan s7025 motherboard with two Nvidia gtx 460 video cards.
>
> My dom0 kernel = 3.1.0-rc10
>
> Xen = xen-unstable.hg-rev-24407
>
> #xl info:
>
> host : vise
> release : 3.1.0-rc10
> version : #2 SMP Fri Jun 15 12:11:10 PDT 2012
> machine : x86_64
> nr_cpus : 4
> nr_nodes : 1
> cores_per_socket : 4
> threads_per_core : 1
> cpu_mhz : 1600
> hw_caps :
> bfebfbff:2c100800:00000000:00003f40:029ee3ff:00000000:00000001:00000000
> virt_caps : hvm hvm_directio
> total_memory : 12279
> free_memory : 11086
> free_cpus : 0
> xen_major : 4
> xen_minor : 2
> xen_extra : -unstable
> xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p
> hvm-3.0-x86_64
> xen_scheduler : credit
> xen_pagesize : 4096
> platform_params : virt_start=0xffff800000000000
> xen_changeset : Fri Dec 09 16:19:36 2011 +0000 24407:03138a08366b
> xen_commandline : dom0_mem=1024M loglvl=all guest_loglvl=all iommu=1
> cc_compiler : gcc version 4.5.1 20100924 (Red Hat 4.5.1-4) (GCC)
> cc_compile_by : root
> cc_compile_domain :
> cc_compile_date : Fri Jun 22 15:59:26 PDT 2012
> xend_config_format : 4
>
> Grub Conf:
>
> title Fedora Xen 4.2-unstable with Linux 3.1.0-rc10 pvops dom0
> root (hd0,0)
> kernel /xen.gz dom0_mem=1024M loglvl=all guest_loglvl=all iommu=1
> module /vmlinuz-3.1.0-rc10 ro root=/dev/mapper/vg_vise-lg_root nomodeset
> xen-pciback.hide=(06:00.0)(06:00.1)(08:00.0)(08:00.1)
>
>
> DomU (hvm_8):
>
> #----------------------------------------------------------------------------
> builder='hvm'
> memory = 1024
> vif = [ 'type=ioemu, bridge=virbr0' ]
> monitor = 1
> name = "hvm_8"
> disk = [ 'file: /var/lib/libvirt/images/centos5.3_8.img,hda,w',
> 'phy:/dev/sr0,xvdc:cdrom,r',]
> boot = 'c'
> vnc = 1
> vncdisplay = 1
> usb = 1
> usbdevice = 'tablet'
> localtime = 1
> vcpus = 1
> xen_platform_pci=1
> gfx_passthru=1
> pci = [ '08:00.0','08:00.1','00:1a.2' ]
> pci_msitranslate = 1
> pci_power_mgmt = 1
> #----------------------------------------------------------------------------
>
>
> DomU(hvm_6):
> builder='hvm'
> memory = 1024
> vif = [ 'type=ioemu, bridge=virbr0' ]
> monitor = 1
> name = "hvm_6"
> disk = [ 'file: /var/lib/libvirt/images/centos5.3.img,hda,w',
> 'phy:/dev/sr0,xvdc:cdrom,r',]
> boot = 'c'
> vnc = 1
> vncdisplay = 1
> usb = 1
> usbdevice = 'tablet'
> localtime = 1
> vcpus = 1
> xen_platform_pci=1
> gfx_passthru=1
> pci = [ '06:00.0','06:00.1','00:1d.0' ]
> pci_msitranslate = 1
> pci_power_mgmt = 1
> #----------------------------------------------------------------------------
>
>
>
> Currently Video card 1's BusId = "08:0.0" and video card 2's BusId =
> "06:0.0"
>
> So far I have successfully configured VGA passthough for card 1.
>
>  I have two domU guest named hvm_8 and hvm_6 accordingly. When dom0 starts
> it displays output on card 1. After dom0 comes up I start domU hvm_8. At
> this point Card 1 resets and displays the console for hvm_8. This all works
> as I would expect.
>
> Next I start hvm_6. What I expect to see is Card 2 display hvm_6's video
> output. Instead Card 1 goes blank.
>
> Both hmv_6 and hvm_8 show up in “#xl list” but neither display output on
> either card 1 or card 2. Both monitors connected to card 1 and card 2 are
> black and appear to be off.
>
> It seams like Xen is sending a reset to card 1 when I start domU associated
> with card 2.
>
> Joe,
>
>
> On Wed, Jun 27, 2012 at 11:03 AM, Matthias
> <matthias.kannenberg@xxxxxxxxxxxxxx> wrote:
>>
>> Hi
>>
>> A short question just to be save:
>>
>> Are you trying to assign one seperate vga card per domU or are you trying
>> to assign the same vga card to different domUs and trying to run them at the
>> same time?
>>
>> Also, is it okay for you if i keep adding the xen mailing list in CC? One
>> benefit is that another user might have the same issue in the future and can
>> find your post and the other is that others might see your post and help
>> you, too..
>>
>> Am 27.06.2012 18:31 schrieb "joe harper" <jharper.research@xxxxxxxxx>:
>>
>>> Thank you for your response.
>>>
>>> I have been trying to utilize primary passthrough(gfx-passthrough-1) with
>>> nvidia GTX 460 cards. I am able to pass 1 card through.
>>>
>>> My goal is to have 1 card for each domU guest. The problem that I see is
>>> when I start the second domU the card associated with the first domain
>>> resets and goes blank. At this point neither card shows output and then the
>>> domu that I Just started crashes.
>>>
>>> It seems to me that the problem lies within the power managment / reset
>>> for the graphics cards.
>>>
>>> Thanks,
>>>
>>> On Tue, Jun 26, 2012 at 12:59 AM, Matthias
>>> <matthias.kannenberg@xxxxxxxxxxxxxx> wrote:
>>>>
>>>> Hi,
>>>>
>>>> sure I can give you some info, but unfortunately as explained here
>>>> (http://lists.xen.org/archives/html/xen-users/2012-06/msg00392.html),
>>>> dual vga is broken for me currently, too. But i guess this is only a
>>>> matter of finding the currupted changeset.
>>>>
>>>> In general, I was never aware that this is even a problem. I used to
>>>> simply hide both graphic cards from the dom0 at boot time via the
>>>> pciback.hide and than assigned them as secondary (= without
>>>> gfx_passthrough=1) to differnet domUs, one to a Windows 7 for Work /
>>>> Gaming and one to a Linux as Mediacenter. The dom0 boots just fine
>>>> without graphic cards and the management (starting xen / starting
>>>> domUs etc.) is done via SSH from either a Smartphone or a laptop.
>>>>
>>>> My dom0 is a Debian Wheezy, but with an open Suse Kernel and their
>>>> patches, which help a lot, and my xen is a testing from their git
>>>> repository. I'm not compiling any vga bios into kernel or doing some
>>>> other patching to xen (like the adress calculation) to get it work.
>>>>
>>>> I think the main difference to most other is that I am still using xm
>>>> over xl because it seems much more stable to me and vga passthrough
>>>> works like a charm.
>>>>
>>>> I'm Using an AMD Phenom 2 X1090T on a Asus Crosshair IV, 16GB
>>>> DDR3-1600 RAM, an AMD HD5700 and an AMD HD5450 (passive cooling) card.
>>>>
>>>> Oh, and a an advice: If you want to seperate domUs with VGA you should
>>>> keep in mind that you have to think of somethink concerning your sound
>>>> because the emulated sound kind of sucks. I solved this for me with
>>>> passing my onboard sound to the windows VM and using the AudiOverHDMI
>>>> with my mediacenter because it is connected to my TV. If you want to
>>>> connect both domUs to Monitors you might run into problems here.. In
>>>> general it might be possible to just pass a dedicated audio card to
>>>> the other domU, but at least for my, my PCI slots are full with
>>>> graphic cards and their coolers ;) . Same issue with sound goes with
>>>> input devices. Either user two seperated keyboard / mouse setups, or
>>>> you can use xen's vnc server. With the usb=tablet input option, this
>>>> is pretty decent.
>>>>
>>>> Well, I hope i could give you some pointers. These were the genereal
>>>> information i could think of. If you want / need some more detailed
>>>> information on a topic, just ask.
>>>>
>>>>
>>>>
>>>> 2012/6/26 joe harper <jharper.research@xxxxxxxxx>:
>>>> > Hello Matthias,
>>>> >
>>>> > I came across a recent post of yours that mentioned success with Dual
>>>> > VGA
>>>> > passthrough in Xen. I am trying to do something similiar. Would you
>>>> > please
>>>> > inform me of your set up? Which video cards are you using and are you
>>>> > using
>>>> > primary or secondary passthrough?
>>>> >
>>>> >
>>>> > Thank you
>>>> >
>>>> > Joe.
>>>> >
>>>> >
>>>
>>>
>

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users