Xen project Mailing List

Fix: Send to Joerg Roedel's new email (no longer employed by AMD) and also Xen's AMD-Vi maintainer

---------- Forwarded message ----------
From: 小太 <nospam@xxxxxxxx>
Date: Sun, Mar 20, 2016 at 4:01 AM
Subject: Re: [Xen-users] Enabling AMD-Vi IOMMU panics Xen
To: Zir Blazer <zir_blazer@xxxxxxxxxxx>
Cc: "xen-users@xxxxxxxxxxxxx" <xen-users@xxxxxxxxxxxxx>, coreboot@xxxxxxxxxxxx, joerg.roedel@xxxxxxx

On Sat, Mar 19, 2016 at 7:14 PM, Zir Blazer <zir_blazer@xxxxxxxxxxx> wrote:

Fix: Added xen-users as CC.

________________________________
> From: nospam@xxxxxxxx
> To: xen-users@xxxxxxxxxxxxx
> Date: Sat, 19 Mar 2016 07:30:40 +0000
> Subject: [Xen-users] Enabling AMD-Vi IOMMU panics Xen
>
> Note that the board "by default" doesn't support IOMMU, but since it
> runs coreboot, I compiled in some code that initialises the IOMMU (so
> if this bug seems to be caused by the BIOS, hopefully it can be fixed
> fairly quickly).
>
> Using that command line, the IOMMU does not get enabled due to some
> problems with the IVRS table (which seems to be generated by AMD's
> closed-source AGESA), which you can see in "apic_verbosity=debug.log"
> (attached).

My answer will not really be very useful, but at least it may save you some time...

If there are issues with the ACPI IVRS Table, the issues are Firmware side. I suppose that you could try contacting Coreboot developers and asking them if they can give you a hand fixing it for a AMD Jaguar platform.

Curiously, cpuid reports the CPU as family 22 (16h) and model 48 (30h), making it a Puma (Mullins) chip.

I don't specifically recall if Xen could, but the Linux Kernel itself could workaround a broken ACPI IVRS Table since you had a Boot Loader parameter that you could set to override the broken values of it with your own provided values, check here:

https://bbs.archlinux.org/viewtopic.php?id=163102

Basically, for the Linux Kernel, you could use something like iommu=1 ivrs_ioapic[6]=00:14.0 as boot parameter or so and get a working IOMMU with a broken BIOS ACPI IVRS Table. However, this only works if you were going to use the IOMMU for QEMU-KVM-VFIO Passthrough, not Xen, but I think it could also do something similar to that. The problem is that I don't know what the Xen parameters are (If they exist), or how you are supposed to get the correct values to get it working.
If it is not possible to do something like that in Xen, your only choice is fix it Firmware-side.

To partially answer your question, Xen indeed has the ivrs_ioapic and ivrs_hpet params too: http://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=xen/drivers/passthrough/amd/iommu_acpi.c;h=79c1f8cd3e7d02e96ea9183b6a726b885500a05d;hb=HEAD#l638

That said, I have no idea how to find the device IDs for the IOAPICs.

According to https://github.com/torvalds/linux/commit/c2ff5cf5294bcbd7fa50f7d860e90a66db7e5059 (and http://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=06bbcaf48d09c18a41c482866941ddd5d2846b44 for Xen), the southboard IOAPIC is always at 00:14.0 (I'm also curious of where this information was sourced from - it doesn't seem to be in either the IOMMU spec or BKDG), though I don't know whether it's IOAPIC #4 or #5.

According to http://ubuntuforums.org/showthread.php?t=2254677, their (presumably northbridge) IOAPIC was at 00:00.1, though they don't mention how they determined that either.

00:00.1 doesn't show up under a lspci either, so I'm unsure if I have that on my board, though AMD's IOMMU spec (http://support.amd.com/TechDocs/48882_IOMMU.pdf page 260) claims they aren't normally enumerable.

The lspci output on the board is as follows:
00:00.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1566]
00:00.2 IOMMU [0806]: Advanced Micro Devices, Inc. [AMD] Device [1022:1567]
00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:156b]
00:02.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 16h Processor Functions 5:1 [1022:1439]
00:02.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 16h Processor Functions 5:1 [1022:1439]
00:02.4 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 16h Processor Functions 5:1 [1022:1439]
00:08.0 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Device [1022:1537]
00:10.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] FCH USB XHCI Controller [1022:7814] (rev 11)
00:11.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [IDE mode] [1022:7800] (rev 40)
00:13.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] FCH USB EHCI Controller [1022:7808] (rev 39)
00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:780b] (rev 42)
        Subsystem: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:780b]
        Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap- 66MHz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Kernel driver in use: piix4_smbus
        Kernel modules: i2c_piix4
00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:780e] (rev 11)
00:14.7 SD Host controller [0805]: Advanced Micro Devices, Inc. [AMD] FCH SD Flash Controller [1022:7813] (rev 01)
00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1580]
00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1581]
00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1582]
00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1583]
00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1584]
00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1585]
01:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit Network Connection [8086:157b] (rev 03)
02:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit Network Connection [8086:157b] (rev 03)
03:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit Network Connection [8086:157b] (rev 03)

If 00:14.0 is indeed one of the IOAPIC devices, I'm also curious on how it relates to SMBus.

I'm CC'ing coreboot and also Joerg Roedel, the author of the linux commit, to see if they can help out.

If we work out a process to properly identify the IOAPIC number to device ID mappings, I'll try to properly document it somewhere and make patches for Xen/coreboot/linux to use it.

[Xen-users] Fwd: Enabling AMD-Vi IOMMU panics Xen