[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-users] HVM VGA Passthrough to Win7 64 with Xen 4.3


  • To: xen-users@xxxxxxxxxxxxx
  • From: Gordan Bobic <gordan@xxxxxxxxxx>
  • Date: Sun, 01 Sep 2013 00:55:04 +0100
  • Delivery-date: Sat, 31 Aug 2013 23:56:18 +0000
  • List-id: Xen user discussion <xen-users.lists.xen.org>

I have no prior experience with that specific motherboard, so cannot be 100% sure (BIOS/firmware/hardware bugs are disappointingly common these days). The rest sounds just fine.

Things to check/try in no particular order:

1) Make sure that the GPU is never touched during dom0 initialization. Blacklist the radeon, fglrx or any other drivers that might bind to the ATI card(s). If your xen-pciback module is built into the kernel, you can ensure this using something like the following in your kernel boot options:
xen-pciback.hide=(02:00.0)(02.00.1)

If the dom0 driver has ever bound to the device, it will have initialized it, and this is known to wreak havoc afterwards since the card is no longer in the state the domU driver expects to find it in, it gets double-initialized and either runs cripplingly slowly or not at all. The ATI BIOS and drivers aren't quite clever enough when it comes to this.

If your xen-pciback is a module, you'll have to blacklist the radeon/fglrx/whatever driver. Xorg will still end up loading it explicitly when it starts in dom0, so you may need to use an "install" option on the module (don't know how Ubuntu configs are laid out, on EL you'd put it in /etc/modprobe.d/$modulename.conf and possible add stuff to /etc/sysconfig/modules/$modulename.modules. The point of this is that you pre-load xen-pciback before the radeon driver is loaded. You need to tell xen-pciback (in a similar config) which devices to bind to. The net result is that xen-pciback loads before any other drivers for the device, thus ensuring other drivers don't taint the device before it is made available to a domU.

2) Depending on BIOS bugs, check where your PCI memory hole begins, using something like the following in dom0:

# lspci -vvv | grep "Memory at " | sed -e 's/.*Memory at //' | sort

Until you get it working, set your domU RAM to less than the lowest value the above command returns.

3) Disable the generic VGA device in domU (not in the xen config, in domU device manager). I had a similar problem where my quasi-Quadro wouldn't initialize in domU with the same error, because the driver was trying to map legacy VGA ports which were already claimed by the emulated VGA device.

4) Try to figure out (lspci -t helps) your layout WRT PCIe bridges and slots. If you can, put your secondary (passthrough) card on a PCIe bridge of it's own. This can help with some buggy hardware.

Also, you probably don't want to hear this, but I've had slightly better luck (although by no means perfect) with quadrified GeForce cards than ATI cards, despite AMD's recent efforts to make this work. Then again, I have what is probably the buggiest and most problematic motherboard ever made commercially available (EVGA SR-2).

Gordan

On 08/31/2013 10:43 PM, Dan Schmitt wrote:
Hardware:
16gb Ram
AMD FX-8320
MSI 970A-G43
Primary GPU Radeon HD6570 01:00.0,01:00.1
Secondary GPU Radeon HD7950 02:00.0,02:00.1

Dom0 Ubuntu 12.04
Kernel 3.5.0-39
xen 4.3
xl toolset
DomU Win7 64bit

It took some shuffling to hide the HD7950 from
Ubuntu, it's boot process, and X, but pciback seems
to own it now (xl pci-assingable-list shows the 02:00.0
and 02:00.1 when the Win7 DomU isn't running.)

Win7 runs/installs with the emulated video card, and
networking works.
With the drivers installed Win7 seems to see the 7950,
the hardware manager shows it with a yellow warning
triangle/excalmation point. I'm seeing

This device cannot find enough free resources that it can
use (Code 12)

If you want to use this device, you will need to disable
one of the other devices on this system.

In the /var/log/xen/xen-hotplug.log I'm seeing lots of

RTNETLINK answers: Operation not supported
RTNETLINK answers: Operation not supported

but I think they are harmless.
In /var/log/xen/qemu-dm-Win7.hvm.log I'm seeing:

xen_ram_alloc: do not alloc 10f800000 bytes of ram at 0 when runstate
is INMIGRATE
xen_ram_alloc: do not alloc 800000 bytes of ram at 10f800000 when
runstate is INMIGRATE
xen_ram_alloc: do not alloc 10000 bytes of ram at 110000000 when
runstate is INMIGRATE
xen_ram_alloc: do not alloc 10000 bytes of ram at 110010000 when
runstate is INMIGRATE

in snippets of dmesg from boot that may be of interest:

Grub command line (I think I have all the magic to hide 02:00.0):

[    0.000000] Command line: placeholder
root=UUID=14de2da2-d40b-48b8-a0ce-a43f71ddd612 ro apparmor=0 quiet
splash blacklist.nouveau=1 xen-pciback.permissive
xen-pciback.hide=(02:00.0)(02:00.2)
pci=resource_alignment=02:00.0,02.00.2

We can see it on the PCI bus:

[    3.793517] pci 0000:02:00.0: [1002:679a] type 00 class 0x030000
[    3.793539] pci 0000:02:00.0: reg 10: [mem 0xc0000000-0xcfffffff 64bit pref]
[    3.793557] pci 0000:02:00.0: reg 18: [mem 0xfe900000-0xfe93ffff 64bit]
[    3.793569] pci 0000:02:00.0: reg 20: [io  0xd000-0xd0ff]
[    3.793592] pci 0000:02:00.0: reg 30: [mem 0xfe940000-0xfe95ffff pref]
[    3.793598] pci 0000:02:00.0: Disabling memory decoding and
releasing memory resources.
[    3.793668] pci 0000:02:00.0: supports D1 D2
[    3.793670] pci 0000:02:00.0: PME# supported from D1 D2 D3hot
[    3.793718] pci 0000:02:00.1: [1002:aaa0] type 00 class 0x040300
[    3.793740] pci 0000:02:00.1: reg 10: [mem 0xfe9

I don't know why vgaarb grabs it (or if this is ok):

[    3.835681] xen-balloon: Initialising balloon driver.
[    3.835716] xen/balloon: Xen selfballooning driver disabled for domain0.
[    3.835820] vgaarb: device added:
PCI:0000:01:00.0,decodes=io+mem,owns=io+mem,locks=none
[    3.835826] vgaarb: device added:
PCI:0000:02:00.0,decodes=io+mem,owns=none,locks=none
[    3.835830] vgaarb: loaded
[    3.835831] vgaarb: bridge control possible 0000:02:00.0
[    3.835832] vgaarb: bridge control possible 0000:01:00.0

[    3.884755] pci 0000:02:00.0: BAR 0: assigned [mem
0xc0000000-0xcfffffff 64bit pref]
[    3.884764] pci 0000:02:00.0: BAR 2: assigned [mem
0xfe900000-0xfe93ffff 64bit]
[    3.884772] pci 0000:02:00.0: BAR 6: assigned [mem
0xfe940000-0xfe95ffff pref]

The kernel grabs the 6570:
The kernel grabs the 6570:

[    4.680418] [drm] initializing kernel modesetting (TURKS
0x1002:0x6759 0x1462:0x2502).
[    4.680483] [drm] register mmio base: 0xFEA20000
[    4.680485] [drm] register mmio size: 131072
[    4.680692] ATOM BIOS: 113
[    4.680727] radeon 0000:01:00.0: VRAM: 1024M 0x0000000000000000 -
0x000000003FFFFFFF (1024M used)
[    4.680730] radeon 0000:01:00.0: GTT: 512M 0x0000000040000000 -
0x000000005FFFFFFF
[    4.680732] [drm] Detected VRAM RAM=1024M, BAR=256M
[    4.680734] [drm] RAM width 128bits DDR
[    4.681001] [TTM] Zone  kernel: Available graphics memory: 7982182 kiB
[    4.681007] [TTM] Zone   dma32: Available graphics memory: 2097152 kiB
[    4.681009] [TTM] Initializing pool allocator
[    4.681018] [TTM] Initializing DMA pool allocator
[    4.681044] [drm] radeon: 1024M of VRAM memory ready
[    4.681046] [drm] radeon: 512M of GTT memory ready.
[    4.681077] [drm] GART: num cpu pages 131072, num gpu pages 131072
[    4.681670] [drm] Loading TURKS Microcode

I don't know why it goes after the 7950 here (or how to stop it):

[    5.034399] Console: switching to colour frame buffer device 240x67
[    5.051586] fb0: radeondrmfb frame buffer device
[    5.051587] drm: registered panic notifier
[    5.051592] [drm] Initialized radeon 2.18.0 20080528 for
0000:01:00.0 on minor 0
[    5.051622] radeon 0000:02:00.0: enabling device (0000 -> 0003)
[    5.300587] sd 2:0:0:0: [sdb] Write cache: enabled, read cache:
enabled, doesn't support DPO or FUA
[    5.317630] radeon 0000:02:00.0: VRAM: 3072M 0x0000000000000000 -
0x00000000BFFFFFFF (3072M used)
[    5.317633] radeon 0000:02:00.0: GTT: 512M 0x00000000C0000000 -
0x00000000DFFFFFFF
[    5.317635] [drm] Detected VRAM RAM=3072M, BAR=256M
[    5.317637] [drm] RAM width 384bits DDR
[    5.317645] [drm] radeon: 3072M of VRAM memory ready
[    5.317647] [drm] radeon: 512M of GTT memory ready.
[    5.317661] [drm] GART: num cpu pages 131072, num gpu pages 131072
[    5.318173] [drm] Loading TAHITI Microcode
[    5.696909] radeon 0000:02:00.0: WB enabled
[    5.696912] radeon 0000:02:00.0: fence driver on ring 0 use gpu
addr 0x00000000c0000c00 and cpu addr 0xffff880407f99c00
[    5.696915] radeon 0000:02:00.0: fence driver on ring 1 use gpu
addr 0x00000000c0000c04 and cpu addr 0xffff880407f99c04
[    5.696917] radeon 0000:02:00.0: fence driver on ring 2 use gpu
addr 0x00000000c0000c08 and cpu addr 0xffff880407f99c08
[    5.696919] [drm] Supports vblank timestamp caching Rev 1 (10.10.2010).
[    5.696920] [drm] Driver supports precise vblank timestamp query.
[    5.697011] radeon 0000:02:00.0: radeon: using MSI.
[    5.697081] [drm] radeon: irq initialized.
[    5.716856] [drm] ring test on 0 succeeded in 1 usecs
[    5.716861] [drm] ring test on 1 succeeded in 1 usecs
[    5.716866] [drm] ring test on 2 succeeded in 1 usecs
[    5.717408] [drm] ib test on ring 0 succeeded in 0 usecs
[    5.717443] [drm] ib test on ring 1 succeeded in 0 usecs
[    5.717472] [drm] ib test on ring 2 succeeded in 0 usecs
[    5.770027] [drm] Radeon Display Connectors

[    5.837219] fb1: radeondrmfb frame buffer device
[    5.837224] [drm] Initialized radeon 2.18.0 20080528 for
0000:02:00.0 on minor 1

It seems like the drm stuff lets go of it and picback gets
it:

[   12.475932] [drm] radeon: finishing device.
[   12.481586] radeon 0000:02:00.0: ffff880409aa7800 unpin not necessary
[   12.481858] [drm] radeon: ttm finalized
[   12.482371] pciback 0000:02:00.0: seizing device
[   12.584854] pciback 0000:02:00.1: seizing device

I don't know if this is good:

[   12.603755] vgaarb: device changed decodes:
PCI:0000:01:00.0,olddecodes=io+mem,decodes=none:owns=io+mem
[   12.603758] vgaarb: transferring owner from PCI:0000:01:00.0 to
PCI:0000:02:00.0

This seems like what I want to happen when the Win7 hvm starts up:

[   40.821498] xen-pciback: vpci: 0000:02:00.0: assign to virtual slot 0
[   40.822125] xen-pciback: vpci: 0000:02:00.1: assign to virtual slot 0 func 1

My xorg is set up to only look at the 6750
with BusId "PCI:1:0:0"

So, qeustions.

Do I have hardware that will work for hvm
Win7 (HD6570 for Dom0, and HD7950 for DomU,
970 bridge chip Mobo and VT-d enabled?)

Do I have the right software to "work out
of the box"?  (4.3 xen release and 12.04
ubuntu)

Am I doint something silly or are there more things
I should look for?

My next explorations will be trying to pass the 7950 to
an ubuntu HVM to see what happens, and see if booting
in text mode (and keeping the kernel/frame buffer out
of it) does anything.

Thanks for your time.

          Dan S

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users



_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users


 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.