[Xen-devel] GPU passthrough with Xen 4.2 on Ubuntu 12.04

[Matthew Dean <mcd40@xxxxxxxxx>]

I have been trying to setup GPU passthrough for a couple of days now with little luck.  I'm hoping someone can shed some light as to where I may be going wrong or at least identify some genuine bugs.  Essentially pci passthrough works for me but gpu passthrough doesn't.

My system is currently configured as follows (please ask if you need further details)

Asrock Z77 e-Itx (stock bios, not sure the version but I can find out.  vt-d is enabled)
i7 3770S
AMD Radeon HD 7750

Dom0 - Ubuntu server 12.04 with desktop environment and the following extra packages installed before the build

build-essential zlib1g-dev python-dev libncurses5-dev libssl-dev libx11-dev uuid-dev libyajl-dev libaio-dev libglib2.0-dev pkg-config bridge-utils iproute udev bison flex gettext bin86 bcc iasl markdown ocaml-nox ocaml-findlib git gcc-multilib texinfo pciutils-dev gawk libcurl3 libcurl4-openssl-dev bzip2 module-init-tools transfig tgif texlive-latex-base texlive-latex-recommended texlive-fonts-extra texlive-fonts-recommended mercurial make gcc libc6-dev python python-twisted patch libsdl-dev libjpeg62-dev libbz2-dev e2fslibs-dev git-core xz-utils liblzma-dev liblzo2-dev libvncserver-dev

I've used the stock Ubuntu provided kernel which is version 3.2.0-27-generic.  Dom0 is connected to a display via the intel integrated graphics.

xen 4.2 has been build from xen unstable tag 4.2.0-rc1 changeset 25689 using the following commands

./configure
make world
make install

I had to manually add a line to /etc/fstab to get /proc/xen to mount on startup
Modules xen-evtchn, xen-gntdev and xen-pciback were setup to load on boot in /etc/modules
I've then setup xencommons to start on boot: "update-rc.d xencommons defaults 19 18"

Grub2 has then been setup to automatically boot the xen kernel.  I've also had to add the option xsave=0 to the xen boot command line to get things to boot

On restart everything looks good. "xl list" returns the Dom0 domain only.  "cat /proc/xen/capabilities" returns control_d.  I've created a windows 7 domU without any pci passthrough and successfully installed windows 7 ultimate.  The config file looks like:

builder='hvm'
vcpus = 4
memory = 8192
shadow_memory = 48
name = "xenwin7"
vif = [ 'bridge=br0' ]
acpi = 1
apic = 1
disk = [ 'file:/usr/local/xenImages/xenwin7.img,hda,w']
boot="c"
sdl=0
vnc=1
vncconsole=1
vncpasswd=''
viridian=1
usb=1
serial='pty'
usbdevice='tablet'




I would like to be able to pass through the HD 7750 and a USB controller to the VM.  I see from lspci there are three devices I need to pass through, two for the gpu (the card itseft and the hd audio for hdmi device) and one for the USB controller.  The bus/device ids are:

0000:01:00.0
0000:01:00.1
0000:00:14.0

There are other usb controllers but I would like to leave them with dom0.  I can successfully configure the devices for passthrough using "xl pci-assignable-add".  In the config I then add the line:

 pci=["01:00.0","01:00.1","00:14.0"]
 
When I boot the VM everything seems to work fine.  Windows picks up all three devices and I can install drivers for them.  The USB controller works flawlessly and I can use an attached mouse and keyboard.  The radeon card requires a restart after which an attached display comes to life and I have 3D acceleration.  Restarting the VM seems to work OK.  If however I shut down the VM I have no end of problems.  On any subsequent startup the vm struggles to get past the windows splash screen, waiting much longer than usual.  During this period dom0 is sluggish regarding mouse and keyboard input even though cpu and memory usage are very low.  When windows finally loads I have no active display and I have to view the VM via VNC.  In device manage I find that windows has disabled the GPU saying there are not enough resources to run the card.  From this point onwards I can do nothing to get the gpu working again aside from removing the device, manually deleting the drivers, and starting again.

Note that at this point I am only trying to do secondary passthrough though I would ideally like to get to the point of doing primary passthrough.  Adding the line gfx_passthru=1 at any point in all this to the machine config however just prevents the VM from booting entirely; when I VNC in all I get is a qemu prompt and I never get any ouput to the real display.

Any suggestions as to how to get this to work would be greatly appreciated as I've hit a bit of a brick wall.  I should also say that I have managed to get secondary passthrough working using Debian Wheezy and the repository version of xen 4.1.  In that case though Dom0 didn't boot reliably.

Sincerely,

Matthew Dean

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel