[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-users] VGA Passthrough / Xen 4.2 / Linux 3.9.2



On 05/15/2013 02:33 AM, Alex Karaoui wrote:
Aloha!

Gordan, I've been following this thread reading your very detailed and
thorough analysis.  Like you I too am using an AMD Radeon (I have a 5870
and a 5750).  I'm also using a dual socket Xeon (5440 on Supermicro
X7DWA-N).  I haven't gotten as far as you have though, not even close.
  I never video to work, not even once.  I had to stop working on
getting Xen to work because I'm involved in several development projects
at the moment and just didn't have the time, but I hope to get back to
it soon.

Kindly, would it be possible for you to answer _some_ the following:
  (it's a lot, so pick and choose, but I would like best if I could
reproduce your environment).

1) Could you post your grub.cfg, kernel configuration (from /boot),

My machine is diskless using NFS root and booting via PXE, but my pxe boot config file entry is here:

label xen
        kernel mboot.c32
append xen.gz noreboot dom0_vcpus_pin --- vmlinuz-3.9.2-1.el6xen.x86_64 rd_NO_LUKS rd_NO_LVM rd_NO_MD rd_NO_DM root=nfs:10.11.12.13:/nfsroot/normandy,rw,proto=tcp,noatime,nolock,nocto,actimeo=300 ip=eth0:dhcp selinux=0 intel_iommu=on elevator=deadline iomem=relaxed noirqbalance --- initramfs-3.9.2-1.el6xen.x86_64.img

(sorry about the line wraps, I'm sure you can figure it out)

You may or may not require noirqbalance - I found that without it the domU would reliably BSOD every time.

and your xm's domU configuration file?

It is in xenstore, so I cannot easily get to it. You should be able to reverse engineer most of it from the attached xend.log.

I created the VM using virt-manager.

2) What distro and kernel version are your running?

EL6 (Scientific Linux) with the kernel and xen packages from here:
http://xen.crc.id.au/support/guides/install/

Note: /boot/xen.gz I use is from xen-hypervisor-4.2.1-6 package.
The rest of the xen stack (using xm) is 4.2.2. This is because of this bug that is not yet resolved:

http://xen.crc.id.au/bugs/view.php?id=5

Versions of /boot/xen.gz newer than 4.2.1-6 don't work for me, and the problem has been tracked down to XSA46.

3) Did you compile the Xen kernel or the dom0 kernel with any patches or
are they stock?

Unmodified packages from the source mentioned above.

4) Did you compile qemu with any patches?

All packages are as pulled down from EL6 repositories or from the Xen EL6 kernels site.

5) What version of Xen are you using?  Still using the xm toolchain, yes?

Yes, xm.

6) Are you using 'radeon' OSS driver with Kernel-Mode-Swtiching or the
AMD proprietary driver?

My dom0 GPU is nvidia, using the Nvidia binary driver (modified 295.75 due to this:

http://www.altechnative.net/2013/04/14/wquxga-a-k-a-omgwtf-ibm-t221-3840x2400-204dpi-monitor-part-6-regressing-drivers-and-xen/

I have not yet had a chance to write a LD_PRELOAD library that fakes out RandR geometry the same way the fakexinerama library does to enable sensible use of my T221.

7) Are you using pciback compiled into the kernel or as a module?  How
are you invoking it?

It's a module.
# cat /etc/modprobe.d/xen-pciback.conf
options xen-pciback permissive=1 hide=(00:1a.0)(00:1b.0)(00:1d.2)(02:00.0)(07:00.0)(07:00.1)

02:00.0 is a Marvell NIC (one of the two on-board ones)
07:00.? is the ATI card
the rest is USB conrollers.

The radeon driver is blacklisted.

Before I start the VM, I detach the devices from the host:

====
# cat usr/local/sbin/detach.sh
#!/bin/bash

modprobe xen-pciback

virsh nodedev-detach pci_0000_00_1a_0
virsh nodedev-detach pci_0000_00_1b_0
virsh nodedev-detach pci_0000_00_1d_2
virsh nodedev-detach pci_0000_02_00_0
virsh nodedev-detach pci_0000_07_00_0
virsh nodedev-detach pci_0000_07_00_1
====

If you are using radeon cards for both dom0 and domU you will have to do some additional magic because you can't just blacklist the radeon driver.

(Note: Modified from my nvidia based config when I was trying Nvidia 8800GT for dom0 and Quadro 2000 for domU, both using the same binary driver)

# cat etc/modprobe.d/radeon.conf
install radeon /usr/local/sbin/detach-radeon.sh; insmod /lib/modules/$(/bin/uname -r)/kernel/drivers/gpu/drm/radeon/radeon.ko

====
# cat usr/local/sbin/detach-radeon.sh
#!/bin/bash

modprobe xen-pciback

virsh nodedev-detach pci_0000_07_00_0
virsh nodedev-detach pci_0000_07_00_1
====

In this example, the passed through card is 07:00:?, adjust the IDs accordingly for your setup. What this does is when the radeon driver gets probed, it will first invoke the script above, which will detach the passthrough device from the host and bind it to xen-pciback driver. This will make the radeon driver unable to bind the device, and it will thus only bind to the dom0 primary device, which will leave the secondary free for passthrough.

You can omit the duplicated lines in detach.sh since there is no point in detaching the GPU twice (although I don't think there's any harm in doing so).

8) What version of Windows and what version of Catalyst are you using?
  Are you also using CCC?

Windows 7 in this test. Latest 13.4 ATI drivers. No CCC, just the bare driver.

9) Is the dom0 running Xorg?  Is Xorg running on your primary (BIOS) or
secondary card?

Yes, running on the BIOS primary GPU (Nvidia 8800GT).

10) Which card did you pass through the primary or secondary?

Secondary, Radeon 6450.

What I would ultimately like to do is create a LiveCD for anyone
attempting this in the future that's configured to some extent to run
AMD cards using PCI passthrough.

I don't think this is possible without at least some manual configuration. Specifically, the device PCI IDs to be detached and passed through will differ on every system. Thankfully, when swapping cards around, the PCI ID is based on the slot the card is plugged into so when replacing GPUs, as long as you put it in the same slot, you don't have to change any IDs in your configs.

But having followed your posts, I'm just amazed that you got it working,
and now, seemingly flawlessly.  I just wish I could understand how.

Me too - I am more frustrated by the fact that it is working now than I was by the fact that it wasn't working before.

The only definitive thing I can narrow it down to at the moment is ejecting the card to reset it just prior to starting a game. Then again, I half expect that tonight when I try it again, it will just BSOD on me all over the place, even though the configuration can't have changed since powering the machine off last night.

Gordan

Attachment: xend.log
Description: Text document

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users

 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.