Re: [Xen-devel] Available PCIe lanes for VGA passthrough

[Sandi Romih <romihs.forums@xxxxxxxxx>]

Hello everyone,

Looks like my initial email to xen-devel did not get delivered or something...

Please see it below this email.

Okay, here is some info about my system:

lspci | grep -i vga

03:00.0 VGA compatible controller: nVidia Corporation GF100 [GeForce GTX 480] (rev a3)

06:04.0 VGA compatible controller: Matrox Graphics, Inc. MGA G200eW WPCM450 (rev 0a)

lspci | grep -i 03:00

03:00.0 VGA compatible controller: nVidia Corporation GF100 [GeForce GTX 480] (rev a3)

03:00.1 Audio device: nVidia Corporation GF100 High Definition Audio Controller (rev a1)

dmesg | grep -i 03:00

[    0.000000] Command line: placeholder root=UUID=d5f5207b-d2aa-4f19-b51d-bc2c727b9e8f ro nomodeset xen-pciback.passthrough=1 xen-pciback.permissive xen-pciback.hide=(03:00.0)(03:00.1) quiet

[    5.114443] Kernel command line: placeholder root=UUID=d5f5207b-d2aa-4f19-b51d-bc2c727b9e8f ro nomodeset xen-pciback.passthrough=1 xen-pciback.permissive xen-pciback.hide=(03:00.0)(03:00.1) quiet

[    5.356290] pci 0000:03:00.0: [10de:06c0] type 0 class 0x000300

[    5.356310] pci 0000:03:00.0: reg 10: [mem 0xf8000000-0xf9ffffff]

[    5.356332] pci 0000:03:00.0: reg 14: [mem 0xd8000000-0xdfffffff 64bit pref]

[    5.356353] pci 0000:03:00.0: reg 1c: [mem 0xd4000000-0xd7ffffff 64bit pref]

[    5.356368] pci 0000:03:00.0: reg 24: [io  0xdc00-0xdc7f]

[    5.356383] pci 0000:03:00.0: reg 30: [mem 0xfae80000-0xfaefffff pref]

[    5.356475] pci 0000:03:00.1: [10de:0be5] type 0 class 0x000403

[    5.356494] pci 0000:03:00.1: reg 10: [mem 0xfae7c000-0xfae7ffff]

[    5.425416] vgaarb: device added: PCI:0000:03:00.0,decodes=io+mem,owns=none,locks=none

[    5.425438] vgaarb: bridge control possible 0000:03:00.0

[    5.443721] pciback 0000:03:00.0: seizing device

[    5.443726] pciback 0000:03:00.1: seizing device

[    5.713088] pciback 0000:03:00.0: Signaling PME through PCIe PME interrupt

[    5.713090] pciback 0000:03:00.1: Signaling PME through PCIe PME interrupt

[    5.713566] pciback 0000:03:00.1: PCI INT B -> GSI 25 (level, low) -> IRQ 25

[    5.713574] pciback 0000:03:00.1: PCI INT B disabled

[    5.713610] pciback 0000:03:00.0: enabling device (0146 -> 0147)

[    5.713627] pciback 0000:03:00.0: PCI INT A -> GSI 26 (level, low) -> IRQ 26

[    5.713633] pciback 0000:03:00.0: PCI INT A disabled

xl pci-list-assignable-devices 

0000:03:00.0

0000:03:00.1

cat /etc/default/grub 

GRUB_DEFAULT=0

GRUB_TIMEOUT=5

GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`

GRUB_CMDLINE_LINUX_DEFAULT="quiet"

GRUB_CMDLINE_LINUX="nomodeset xen-pciback.passthrough=1 xen-pciback.permissive xen-pciback.hide=(03:00.0)(03:00.1)"

GRUB_DISABLE_OS_PROBER=true

GRUB_CMDLINE_XEN="dom0_mem=1G"

From all that, I assume that the VGA card, and its on-board audio device, is owned by pciback.

This is how I start the domain:

xl create xenwinxp.cfg 

Parsing config file xenwinxp.cfg

WARNING: ignoring "kernel" directive for HVM guest. Use "firmware_override" instead if you really want a non-default firmware

xc: info: VIRTUAL MEMORY ARRANGEMENT:

  Loader:        0000000000100000->00000000001819b4

  TOTAL:         0000000000000000->00000000bf800000

  ENTRY ADDRESS: 0000000000100000

xc: info: PHYSICAL MEMORY ALLOCATION:

  4KB PAGES: 0x0000000000000200

  2MB PAGES: 0x00000000000003fb

  1GB PAGES: 0x0000000000000001

libxl: error: libxl_pci.c:776:libxl__device_pci_reset: The kernel doesn't support reset from sysfs for PCI device 0000:03:00.0

Daemon running with PID 3001

Quick confirmation that it is up and running:

xl list

Name                                        ID   Mem VCPUs State Time(s)

Domain-0                                     0  1024    12     r-----     255.7

winxp                                        1  3067     4     -b----      61.2

This is my xen config file:

kernel = "/usr/lib/xen/boot/hvmloader"

builder='hvm'

memory = 3072

# shadow_memory = 1024

name = "winxp"

vcpus='4'

vif = [ 'bridge=eth0,mac=00:14:3e:00:8f:c2' ]

disk = ['file:/xen-vms/winxp/xenwinxp.img,hda,w','file:/xen-vms/isos/winxp.iso,hdc:cdrom,r']

boot="cd"

sdl=0

vnc=1

vnclisten="0.0.0.0"

vncconsole=1

vncpasswd=''

acpi=1

apic=1

xen_extended_power_mgmt=1

pae=1

arch='x86_64'

hpet = 1

hap = 1

viridian = 1

monitor=1

serial='pty'

# keymap='fr'

# soundhw = "all"

# audio="on"

ne2000=0

>

on_reboot   = 'restart'

on_crash    = 'restart'

xen_platform_pci=1

gfx_passthru=0

pci  = [ "03:00.0,msitranslate=1,power_mgmt=1" ]

acpi_s3 = 1

acpi_s4 = 1

The Windows XP boots up fine, I can see the GTX480 in the device manager, but it is not useable.

 Now, let me try gfx_passthru=1

Create the domain:

xl create xenwinxp.cfg 

Parsing config file xenwinxp.cfg

WARNING: ignoring "kernel" directive for HVM guest. Use "firmware_override" instead if you really want a non-default firmware

xc: info: VIRTUAL MEMORY ARRANGEMENT:

  Loader:        0000000000100000->00000000001819b4

  TOTAL:         0000000000000000->00000000bf800000

  ENTRY ADDRESS: 0000000000100000

xc: info: PHYSICAL MEMORY ALLOCATION:

  4KB PAGES: 0x0000000000000200

  2MB PAGES: 0x00000000000003fb

  1GB PAGES: 0x0000000000000001

libxl: error: libxl_pci.c:776:libxl__device_pci_reset: The kernel doesn't support reset from sysfs for PCI device 0000:03:00.0

Daemon running with PID 3454

When the domain starts, my dom0 screen attached to the 06:04.0 VGA controller goes blank, and after a few seconds I get a message from my monitor that 'this video format is not supported'. I can get back to my dom0 by first going Ctrl-Alt-F1 (where I will see a whole bunch of garbage in the screen) and then Ctrl-Alt-F7.

The domain is running:

xl list

Name                                        ID   Mem VCPUs State Time(s)

Domain-0                                     0  1024    12     r-----     385.3

winxp                                        3  3063     1     r-----     324.5

And if I check out what is happening through vncviewer, I see just the QEMU console (QEMU 0.10.2 monitor - type 'help' for more information)

And finally, I see that the VGA card is no longer available:

xl pci-list-assignable-devices 

0000:03:00.1

I have checked all three of the video cards outputs, no video signal anywhere.

Any ideas about what could be causing the problems I am encountering?

I would be grateful for any info.

Thanks in advance.

Sandi

On Tue, Jan 17, 2012 at 11:10 AM, Sandi Romih <romihs.forums@xxxxxxxxx> wrote:

Hello,

I hope that I am posting the correct list with this question.

I have been trying to get VGA passthrough working on my system for some time now without success.

I do not have the system with me right now, so I will not be able to provide any logs or error messages right now. I will be able to to do so in a couple of hours.

I have a VT-d enabled system (Motherboard and CPU) on which I wish to pass the secondary graphics card (EVGA GTX480 SE) through to the HVM (WinXP 32-bit).

Currently I am testing with Debian Squeeze, using the 3.1.8 kernel and Xen 4.2.unstable according to these instructions: http://www.davidgis.fr/blog/index.php?2011/12/07/860-xen-42unstable-patches-for-vga-pass-through

I can only boot the WinXP client via the VNC console (gfx_passthrough=0), and from within WinXP I see that the OS has detected the GTX480 card (as its secondary card), but I am not able to use it. I have tried the 275.33 drives, and they make no difference.

My question is related to the number of PCIe lanes I have available to the graphics card.

I only have 8 lanes available for the GTX480 (which is a PCIe x16 card), could this pose an issue for Xen?

I plan to test VGA passthrough with an ATI graphics card (also in a PCIe x8 slot) later to see if it will work or not. I will update what I find.

Regards

Sandi

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel