Xen project Mailing List

FOR XEN 4.3 UNSTABLE WE CAN NOW START A WINDOWS 7 64 (ATI/VGA PASSTHROUGH) WITH MORE THAN 3GB OF RAM.

For the current xen-unstable 4.3, applying this patch ftp://ftp.enjellic.com/pub/xen/xen-4.2.0.ati-passthrough.patch

Linux: I was able to start a Ubuntu domU with more than 3GB.
Windows: The only issue was for a Windows 7 domU with more than 3GB

The good new is that if this patch http://marc.info/?l=qemu-devel&m=136177475215360&q=raw is used too then we can have more than 3GB of RAM for Windows 7 64 Bits

1) Download Xen sources
=======================
rev=267773
hg clone -r $rev http://xenbits.xensource.com/staging/xen-unstable.hg/ xen-unstable.hg-rev-XXXXX
cd xen-unstable.hg-rev-XXXXX

2) Configure
============
CURL=$(which curl-config) XML=$(which xml2-config) ./configure

3) Make a first build for tools and cleanup the folder
======================================================

cd tools
make -j4

# Clean up the folder
make clean

4) Download and apply patches
=============================

# Download and apply the 1st patch
cd qemu-xen-dir-remote/
wget "http://marc.info/?l=qemu-devel&m=136177475215360&q=raw" -O - | patch -p1

# Download and apply the 2nd patch
cd ../..
wget ftp://ftp.enjellic.com/pub/xen/xen-4.2.0.ati-passthrough.patch -O - | sed -e "s:qemu-xen-traditional:qemu-xen-traditional-dir-remote:g" | patch -p1

5) Build xen and tools
=======================
cd ..
make -j4 xen && make -j4 tools

6) Build stubdom
================
make stubdom

You shoul have this error

/opt/tmp/xen-unstable.hg-rev-XXXXX/stubdom/mini-os-x86_64-ioemu/mini-os.o: In function `ati_hw_out':
/opt/tmp/xen-unstable.hg-rev-XXXXX/stubdom/ioemu/hw/pt-graphics.c:82: undefined reference to `iopl'
/opt/tmp/xen-unstable.hg-rev-XXXXX/stubdom/ioemu/hw/pt-graphics.c:84: undefined reference to `iopl'
/opt/tmp/xen-unstable.hg-rev-XXXXX/stubdom/mini-os-x86_64-ioemu/mini-os.o: In function `ati_hw_in':
/opt/tmp/xen-unstable.hg-rev-XXXXX/stubdom/ioemu/hw/pt-graphics.c:72: undefined reference to `iopl'
/opt/tmp/xen-unstable.hg-rev-XXXXX/stubdom/ioemu/hw/pt-graphics.c:74: undefined reference to `iopl'

I don't know if commenting iopl(3) and iopl(0) is the good solution but it works for me

sed -i "s: iopl(: //iopl(:g" ./tools/qemu-xen-traditional-dir-remote/hw/pt-graphics.c

Then rebuild

make stubdom

7) Install all
==============

rm -rf stubdom/gmp-x86_64/

make install-xen && make install-tools PYTHON_PREFIX_ARG= && make install-stubdom

8) Linux script to manage domU
==============================
Download the Linux shell script ftp://ftp.enjellic.com/pub/xen/run-passthrough and updated it to your own needs

De : David TECHER <davidtecher@xxxxxxxx>
À : Eric Shelton <eshelton@xxxxxxxxx>
Cc : "xen-users@xxxxxxxxxxxxx" <xen-users@xxxxxxxxxxxxx>; Patrick Proniewski <patpro@xxxxxxxxxx>
Envoyé le : Jeudi 4 avril 2013 3h00
Objet : Re: [Xen-users] PCI/VGA passthrough: differences between Xen and ESXi?

I did it myself :)

Indeed this patch seems to be very interesting.

rev=26771
hg clone -r $rev http://xenbits.xensource.com/staging/xen-unstable.hg/ xen-unstable.hg-rev-${rev}

./configure
cd tools
make
make clean

cd ..

wget ftp://ftp.enjellic.com/pub/xen/xen-4.2.0.ati-passthrough.patch -O - | sed -e "s:qemu-xen-traditional:qemu-xen-traditional-dir-remote:g" | patch -p1

make xen && make install-xen

make tools && make install-tools

Thanks for your help about iopl(). Agreed that iopl(3) and iopl(0) for stubdom are a bit complicated. I commented out these functions so I can build stubdom (shame on me!)

make stubdom

make install-stubdom

Using the script to download I was able (after testing/updating only the start process)

- to start my domU (Linux Ubuntu 12.04 32). I was able to test it for RAM over than 3GB :). 6GB and 6 vcpus. All my games with Wine works perfectly, VLC, smplayer are OK :)

- to start my domU (Windows 7). I didn't test it for RAM over than 3GB. I will do a full test by reinstalling a Windows 7 64 Bits.

Thanks for helping.

De : David TECHER <davidtecher@xxxxxxxx>
À : Eric Shelton <eshelton@xxxxxxxxx>
Cc : "xen-users@xxxxxxxxxxxxx" <xen-users@xxxxxxxxxxxxx>; Patrick Proniewski <patpro@xxxxxxxxxx>
Envoyé le : Mercredi 3 avril 2013 21h49
Objet : Re: [Xen-users] PCI/VGA passthrough: differences between Xen and ESXi?

Thanks for this wonderful story but

I was asking

1) for your revision number/changeset for Xen 4.3 (xl info should return something)

hg clone -r ????

./configure

make ???

patch -p1 <????

make ???

2) any domU configuration file ?

De : Eric Shelton <eshelton@xxxxxxxxx>
À : David TECHER <davidtecher@xxxxxxxx>
Cc : "xen-users@xxxxxxxxxxxxx" <xen-users@xxxxxxxxxxxxx>; Patrick Proniewski <patpro@xxxxxxxxxx>
Envoyé le : Mercredi 3 avril 2013 18h01
Objet : Re: [Xen-users] PCI/VGA passthrough: differences between Xen and ESXi?

1) I applied the patch to 4.3 unstable (back in February). I recall
it applying cleanly (thanks to Dr. Wettstein's work in updating the
original patches). However, there is a build issue when the patched
files are used to build minios, as its libraries do not provide
iopl(). On my initial pass, I may have #ifdef-ed out the iopl() calls
during minios build time, on the presumption that this code path is
not exercised by minios. However, not being fully confident in that
presumption, in some later work I identified a more appropriate
mechanism: an existing hypervisor call for iopl. I think revised code
was written, but remains untested.

2) I am using what I saw as the core aspects of that script:
unbind_devices(), re-enabling the devices, and the use of vbetool to
reinitialize the display (which returns the text console in my case).
Although I am doing passthrough of a USB3 controller as well, I did
not bother with rebinding the USB driver, as I simply surrendered use
of those ports for Windows. I also am not waiting for the VM to exit;
instead, I split it the single script into before and after scripts
that were manually invoked.

I think the runtime unbind, bind, and vbetool, versus giving the PCI
devices over to pciback at boot time, makes a difference in terms of
whether you can stop & restart the passthrough VM. With this config,
it is no problem for me. I heard many reports of people starting up
the Windows VM once, but not again without a reboot.

3+) Tonight I can pull together what I am using and post it up here.
Very little modification of Dr. Wettstein's patch was done. However,
at this stage, it seems to be the little things that make the
difference between GPU passthrough working or not.

My understanding is that there are many intentional quirks in GPUs to
be in compliance with certain HDCP requirements. Unfortunately, I
think the universe of possible secret register access sequences, etc.
will make it impossible, even with 1:1 mapping in addition, to 100%
convince the display driver, which would be needed to get to
virtualized HDCP compliance (I am interested in a virtualized HTPC
environment). It sounds like, from what I have heard of your work,
that NVidia's quirks are especially difficult to deal with (not
necessarily intentionally). It was enough to make me abandon by
existing NVidia GPU in favor of an AMD solution.

Also, as I think I noted in an earlier post, one of the qemu
developers has, perhaps independently, implemented the necessary
quirks for AMD cards. His code effects pretty much the same things as
the patch I used. I think there is at least a patch against mainline
qemu in qemu-devel, if it has not been incorporated already. He has
also been doing a fair amount of PCIe work, which I think may be
important for NVidia, assuming their use of extended config space is
causing some of the difficulties.

It would be nice to see enough code get into Xen 4.3 to get AMD cards
working, but with the switch away from qemu-traditional, which is what
the patch is for, this may now be more reliant on the work already
being done in qemu-devel being mainlined into qemu.

- Eric

On Wed, Apr 3, 2013 at 4:45 AM, David TECHER <davidtecher@xxxxxxxx> wrote:
> Thanks for clarifying.
>
> I am applying the original patches for nvidia. That's the reason why I've
> got the 3GM limitation.
>
> Question 1:
>
> In your link
>
> http://lists.xen.org/archives/html/xen-users/2013-02/msg00410.html
>
> Did you apply the following patch
>
> ftp://ftp.enjellic.com/pub/xen/xen-4.2.0.ati-passthrough.patch
>
> on Xen 4.3 unstable or Xen 4.2 stable?
>
> Question 2:
>
> Do you use this script
>
> ftp://ftp.enjellic.com/pub/xen/run-passthrough
>
> to start/stop your domU
>
> Question 3
>
> Can you show a domU config file?
>
> Thanks for letting me know.
>
>
>
> ________________________________
> De : Eric Shelton <eshelton@xxxxxxxxx>
> À : "xen-users@xxxxxxxxxxxxx" <xen-users@xxxxxxxxxxxxx>; Patrick Proniewski
> <patpro@xxxxxxxxxx>; davidtecher@xxxxxxxx
> Envoyé le : Mercredi 3 avril 2013 1h26
> Objet : Re: [Xen-users] PCI/VGA passthrough: differences between Xen and
> ESXi?
>
> David:
> What are the circumstances in which your comment about "Xen - limited
> to 3GB for RAM" applies? Is it in the case in which one uses multiple
> GPU cards?
>
> There are a number of HVM tutorial examples with well over 3GB:
> http://wiki.xen.org/wiki/Secondary_GPU_Passthrough (6GB Win7 HVM with
> passthrough of ATI 6970 and USB controller)
> http://wiki.xen.org/wiki/Comprehensive_Xen_Debian_Wheezy_PCI_Passthrough_Tutorial
> (6GB Win7 HVM with passthrough of ATI 68XX and multiple USB
> controllers)
> None of these tutorials mention a 3GB limitation.
>
> The only recent reference to a 3GB HVM issue during passthrough is here:
> http://comments.gmane.org/gmane.comp.emulators.xen.user/77214
> which appears to link the issue with use of nvidia patches from here:
> http://www.davidgis.fr/download/xen-4.2_rev25240_gfx-passthrough-patchs.tar.bz2
> From what I gather, nvidia cards perform a bit of undocumented voodoo
> (by design) that has not been fully accounted for.
>
> Elsewhere:
> http://new-wiki.xen.org/old-wiki/xenwiki/XenPCIpassthrough.html
> it sounds like there was a PV domain issue arising from e820 for which
> a reliable workaround was incorporated into the mainline code some
> time ago.
>
>
> Patrick:
> From the above examples, it seems like >3GB may be fine if you use
> ATI/AMD GPUs. I have had fairly good experiences with passthrough of
> a single AMD 6570 with the patch mentioned here:
> http://lists.xen.org/archives/html/xen-users/2013-02/msg00410.html
> although I did not explore how much or how little memory I could
> assign to it. Note, however, that patch may make certain assumptions,
> such as that only one GPU is being passed through. I'm not sure there
> is any unpatched release version (for example, 4.2.1) that reliably
> does GPU passthrough (PCI passthrough of less wonky devices has worked
> for some time).
>
> I think the proposed multi-GPU setup may be putting you out on the
> bleeding edge (on the other hand, it sounds like it is absolutely no
> available under ESXi). It's probably not impossible. Maybe someone
> has already done it. However, how interested and/or comfortable are
> you with code & configuration tweaking, debug, and experimentation?
>
> - Eric
>
>>> Patrick,
>>>
>>> 1) Xen's vocabulary
>>>
>>> dom0 = the main (first) hypervisor, hosting all virtual machines -- like
>>> your Mac OS X
>>> domU = any virtual machine.
>>>
>>> This is not the real meaning but here I try to stay "understandable". But
>>> for better understanding please refer to http://en.wikipedia.org/wiki/Xen
>>>
>>> My understanding is that your dom0 has to be replaced by Linux so you can
>>> run Xen. But for this part I am not a Mac OS X expert.
>>>
>>> 2) More info - details listed beow are only available for a Xen dom0
>>> running on Linux.
>>>
>>> [Extracted]:....I've ended my testing with ESXi, because I was not able
>>> to passthrough sound device, and USB....
>>> [Comment]: With Xen you can :)
>>>
>>> [Extracted]: ...with stock ATI Radeon..;
>>> [Comment]...Xen offers VGA PassThrough feature for ATI Radeon.
>>>
>>> Here is a link to a Youtube video to run Crysis 3 on a domU Windows 7 64
>>> Bits with HD 7970
>>>
>>> http://www.youtube.com/watch?v=GTnchEG4YtI&feature=player_embedded
>>>
>>> As you can see
>>>
>>> - I can use PCI Passthrough to add a XBOX 360 Paddle while playing Crysis
>>> 3 (see at 09:12 in the Video)
>>>
>>> - I have sound too :)
>>>
>>> But it works on Linux domU too. Here is another Youtube video link
>>>
>>> http://www.youtube.com/watch?feature=player_embedded&v=KzqOIMaBgX0
>>>
>>> But you need to apply a few patches to able this feature with Xen 4.2 and
>>> over. Not very complicated to do.
>>> Here are my instructions to do it but I know that there are currently
>>> better way to test it
>>>
>>>
>>> http://www.davidgis.fr/blog/index.php?2013/03/13/935-xen-43-vga-passthrough-ati-card-hd-7970-changeset-26706
>>>
>>> [Extracted]...memory allocation
>>> [Comment]...Xen - limited to 3GB for RAM. A few developers are working on
>>> overpassing this limitation.
>>>
>>> [Extracted]...no snaphost possible
>>> [Comment]...If you used Xen with LVM is then snahshotting is easier to
>>> manage ;)
>>>
>>> [Extracted] no sleep
>>> [Comment] works on Xen for both Linux and Windows. The main issue is that
>>> you can reboot a domU without restarting the dom0.
>>>
>>>
>>> [Extracted] Is it possible to create a virtualized desktop with VGA and
>>> PCI passthrough for 2 or 3 VM's running simultaneously (ie. each one with
>>> its own video card)?
>>> [Comment] Yes it is doable. Never tested but from experience share with
>>> other Xen users I know that it should be run fine.
>>>
>>> Hope it helps.
>>>
>>>
>>> Kind regards.
>>>
>>> David.
>
>

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users