Re: [Xen-users] vga passthrogh on mainboards with broken ivrs tables

["Eric Houby" <ehouby@xxxxxxxxx>]

Michael,

I have a GA-990FXA-UD3 H/W ver 1.1 that will boot up Xen 4.3.1 (Fedora 20)
with IOMMU enabled. 

(XEN) AMD-Vi: IVHD Device Entry: type 0x48 id 0 flags 0xd7
(XEN) AMD-Vi: IVHD Special: 0000:00:14.0 variety 0x1 handle 0x8
(XEN) AMD-Vi: IVHD Device Entry: type 0x48 id 0 flags 0xd7
(XEN) AMD-Vi: IVHD Special: 0000:00:14.0 variety 0x2 handle 0
(XEN) AMD-Vi: IVHD Device Entry: type 0x48 id 0 flags 0
(XEN) AMD-Vi: IVHD Special: 0000:00:00.1 variety 0x1 handle 0x8
(XEN) IVHD Error: Conflicting IO-APIC 0x8 entries
(XEN) AMD-Vi: IOMMU 0 Enabled.
(XEN) AMD-Vi: Enabling global vector map
(XEN) AMD-Vi: Using global interrupt remap table is not recommended (see
XSA-36)!
(XEN) I/O virtualisation enabled
(XEN)  - Dom0 mode: Relaxed
(XEN) Interrupt remapping enabled
(XEN) ENABLING IO-APIC IRQs

My xen command line is:

dom0_mem=2048M,max:2048M dom0_max_vcpus=2 dom0_vcpus_pin loglvl=all
guest_loglvl=all iommu=debug,verbose,no-amd-iommu-perdev-intremap

PCI passthrough of USB controllers works without issues, but I have not been
able to passthrough a secondary graphics card (HD 5750) without getting a
BSoD on my Win7 HVM.  I have had the same 5750 and VM image working on a
GA-890FXA, but have not been able to get the combination working on the
GA-990FXA.  If you are able to get IOMMU enabled, I will be interested to
see if you get the graphics card to passthrough.

Good luck.

-Eric




-----Original Message-----
From: xen-users-bounces@xxxxxxxxxxxxx
[mailto:xen-users-bounces@xxxxxxxxxxxxx] On Behalf Of Michaael
Mair-Keimberger
Sent: Sunday, December 22, 2013 7:08 AM
To: xen-users@xxxxxxxxxxxxx
Subject: [Xen-users] vga passthrogh on mainboards with broken ivrs tables

Hi Lists,

I'm having troubles getting vga passthrough working again with latest
xen-4.3.1.

My system is a bulldozer amd 8core cpu with a gigabyte GA-990FXA-UD5
motherboard.
The host system uses a amd hd6850 while the guest (windows 7) uses a amd hd
7970.

So far everything worked great until xsa-36 [1]. I've haven't updated xen
since this "fix", but yesterday i'll gave it another try. I've updated to
xen-4.3.1 and tried to fix those broken ivrs tables with the newly
introduced ivrx_ioapic kernel parameter. I'm not sure if this should help
me, but at least it sounds promising.
However i didn't get it to work and i hope someone can give my some hints
how to fix that, otherwise i have to downgrade xen again.


grub2 config:

menuentry 'Gentoo Linux XEN' {
        root=hd0,1
        multiboot /boot/xen.gz dom0_mem=8192M,max:8192M iommu=1
dom0_max_vcpus=4 dom0_vcpus_pin ivrs_ioapic[8]=00:14.0
        module /boot/gentoo-3.12.6 root=/dev/md3 radeon.pcie_gen2=1
radeon.dpm=1 softlevel=xen xen-pciback.permissive
xen-pciback.hide=(01:00.0)(01:00.1)(00:12.0)(00:12.2)
}

lspci:

00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI
bridge (external gfx0 port B) (rev 02)
00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD/ATI] RD990 I/O Memory
Management Unit (IOMMU)
00:02.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI
bridge (PCI express gpp port B)
00:09.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI
bridge (PCI express gpp port H)
00:0a.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI
bridge (external gfx1 port A)
00:0d.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI
bridge (external gfx1 port B)
00:11.0 SATA controller: Advanced Micro Devices, Inc. [AMD/ATI]
SB7x0/SB8x0/SB9x0 SATA Controller [AHCI mode] (rev 40)
00:12.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI]
SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
00:12.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI]
SB7x0/SB8x0/SB9x0 USB EHCI Controller
00:13.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI]
SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
00:13.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI]
SB7x0/SB8x0/SB9x0 USB EHCI Controller
00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 SMBus Controller
(rev 42)
00:14.1 IDE interface: Advanced Micro Devices, Inc. [AMD/ATI]
SB7x0/SB8x0/SB9x0 IDE Controller (rev 40)
00:14.2 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 Azalia
(Intel HDA) (rev 40)
00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0
LPC host controller (rev 40)
00:14.4 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 PCI to PCI
Bridge (rev 40)
00:14.5 USB controller: Advanced Micro Devices, Inc. [AMD/ATI]
SB7x0/SB8x0/SB9x0 USB OHCI2 Controller
00:15.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB700/SB800/SB900
PCI to PCI bridge (PCIE port 0)
00:15.1 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB700/SB800/SB900
PCI to PCI bridge (PCIE port 1)
00:15.2 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB900 PCI to PCI
bridge (PCIE port 2)
00:15.3 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB900 PCI to PCI
bridge (PCIE port 3)
00:16.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI]
SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
00:16.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI]
SB7x0/SB8x0/SB9x0 USB EHCI Controller
00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor
Function 0
00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor
Function 1
00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor
Function 2
00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor
Function 3
00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor
Function 4
00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor
Function 5
01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI]
Tahiti XT [Radeon HD 7970]
01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT HDMI
Audio [Radeon HD 7970 Series]
02:00.0 USB controller: Etron Technology, Inc. EJ168 USB 3.0 Host Controller
(rev 01)
03:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s
Controller (rev 11)
04:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI]
Barts PRO [Radeon HD 6850]
04:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Barts HDMI
Audio [Radeon HD 6800 Series]
05:0e.0 FireWire (IEEE 1394): VIA Technologies, Inc. VT6306/7/8 [Fire II(M)]
IEEE 1394 OHCI Controller (rev c0)
06:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd.
RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06)
07:00.0 USB controller: Etron Technology, Inc. EJ168 USB 3.0 Host Controller
(rev 01)
08:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s
Controller (rev 11)

lspci -t:

-[0000:00]-+-00.0
           +-00.2
           +-02.0-[01]--+-00.0
           |            \-00.1
           +-09.0-[02]----00.0
           +-0a.0-[03]----00.0
           +-0d.0-[04]--+-00.0
           |            \-00.1
           +-11.0
           +-12.0
           +-12.2
           +-13.0
           +-13.2
           +-14.0
           +-14.1
           +-14.2
           +-14.3
           +-14.4-[05]----0e.0
           +-14.5
           +-15.0-[06]----00.0
           +-15.1-[07]----00.0
           +-15.2-[08]----00.0
           +-15.3-[09]--
           +-16.0
           +-16.2
           +-18.0
           +-18.1
           +-18.2
           +-18.3
           +-18.4
           \-18.5

xl dmesg (shorten):

(XEN) Xen version 4.3.1 (@xonet.at) (x86_64-pc-linux-gnu-gcc (Gentoo 4.8.2
p1.0, pie-0.5.8) 4.8.2) debug=n Sat Dec 21 11:55:39 CET 2013
(XEN) Latest ChangeSet: 
(XEN) Bootloader: GRUB 2.02~beta1
(XEN) Command line: dom0_mem=8192M,max:8192M iommu=1 dom0_max_vcpus=4
dom0_vcpus_pin ivrs_ioapic[8]=00:14.0 ...
(XEN) IOAPIC[0]: apic_id 8, version 33, address 0xfec00000, GSI 0-23
(XEN) Enabling APIC mode:  Flat.  Using 1 I/O APICs
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) Detected 4026.984 MHz processor.
(XEN) Initing memory sharing.
(XEN) xstate_init: using cntxt_size: 0x3c0 and states: 0x4000000000000007
(XEN) IVHD Error: Conflicting IO-APIC 0x8 entries
(XEN) AMD-Vi: Error initialization
(XEN) I/O virtualisation disabled
(XEN) ENABLING IO-APIC IRQs
(XEN)  -> Using new ACK method
(XEN) Platform timer is 14.318MHz HPET
(XEN) Allocated console ring of 16 KiB.
(XEN) HVM: ASIDs enabled.
(XEN) SVM: Supported advanced features:
(XEN)  - Nested Page Tables (NPT)
(XEN)  - Last Branch Record (LBR) Virtualisation
(XEN)  - Next-RIP Saved on #VMEXIT
(XEN)  - VMCB Clean Bits
(XEN)  - DecodeAssists
(XEN)  - Pause-Intercept Filter
(XEN)  - TSC Rate MSR
(XEN) HVM: SVM enabled
(XEN) HVM: Hardware Assisted Paging (HAP) detected
(XEN) HVM: HAP page sizes: 4kB, 2MB, 1GB
(XEN) spurious 8259A interrupt: IRQ7.


I've tried a few other configuration's (like adding the ivrs_ioapic
parameter to the module line in grub or also adding
vfio_iommu_type1.allow_unsafe_interrupts=1
to the cmdline), but so far i couldn't get it to work.

Since i'm not completely sure what i actually have to insert at the kernel
parameter i'm glad if someone could give me some insight. However, since the
output of xl dmesg is always the same, even though with or without the
parameter, i might missed some additional parameters.

If you need any more information, I'll be glad to provide them.

Thanks in advance.

Michael








[1] http://lists.xen.org/archives/html/xen-announce/2013-02/msg00002.html

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users


_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users