Re: [Xen-devel] Xen-unstable boot panic due to changeset 26517 AMD, IOMMU: Clean up old entries in remapping tables when creating new one

[Sander Eikelenboom <linux@xxxxxxxxxxxxxx>]

Friday, February 8, 2013, 9:14:35 PM, you wrote:

> On 2/6/2013 6:52 AM, Jan Beulich wrote:
>>>>> On 06.02.13 at 12:24, Sander Eikelenboom <linux@xxxxxxxxxxxxxx> wrote:
>>> Hmm with the patch it does boot, but disables the I/O virtualization.
>> Good. While, as said before, I still don't understand why it didn't
>> crash earlier without that patch, I'm glad it's fixed. Will post the
>> patch for inclusion momentarily.
>>
>>> Output of xl-dmesg attached, do you still need a xen-sums of the situation
>>> without the debug patch (where it does crash) ?
>> And you can't expect much else with broken ACPI tables:
>>
>> (XEN) AMD-Vi: IVHD Device Entry: type 0x48 id 0 flags 0xd7
>> (XEN) AMD-Vi: IVHD Special: 0000:00:14.0 variety 0x2 handle 0
>>
>> This is a HPET entry.
>>
>> (XEN) AMD-Vi: IVHD Device Entry: type 0x48 id 0 flags 0
>> (XEN) AMD-Vi: IVHD Special: 0000:00:00.1 variety 0x1 handle 0x7
>>
>> And this is an entry for IO-APIC #2 (ID 7), whereas FADT says
>>
>> (XEN) ACPI: IOAPIC (id[0x06] address[0xfec00000] gsi_base[0])
>> (XEN) IOAPIC[0]: apic_id 6, version 33, address 0xfec00000, GSI 0-23
>> (XEN) ACPI: IOAPIC (id[0x07] address[0xfec20000] gsi_base[24])
>> (XEN) IOAPIC[1]: apic_id 7, version 33, address 0xfec20000, GSI 24-55
>>
>> so the IOMMU table is lacking an entry for the first IO-APIC, and
>> without that we can't set up per-device interrupt remapping (in
>> which case we choose to disable the IOMMU altogether, albeit it
>> had been questioned whether that isn't making a bad situation
>> worse in some cases).
>>
>> If you want the IOMMU back (at the price of re-opening the
>> security issue described in XSA-36), you'd have to pass
>> "iommu=amd-iommu-perdev-intremap" to the hypervisor.
>>
>> Jan

> Jan,

> It seems that all the recent issues with the AMD IOMMU regarding IOAPIC are
> mainly caused by mismatch information from IVRS and MADT. Xen sets up 
> "nr_ioapics"
> by checking the number of IOAPICs reported in MADT, while the amd/iommu_acpi.c
> code uses information from the IVHD entries of the IVRS to initialize IOMMU.

> Most of the issues we are seeing are often triggered when platform BIOS 
> decides
> to disable one of the two IOAPICs in the RD890s configuration. I am trying to
> summarize the cases here:


> CASE1: BIOS disable the IOAPIC in the southbridge (SB8X0 chipset)
> This is the case we are seeing here with the AMD 890-FX motherboard.
> Here, the MADT is reporting 2 IOAPICs as shown by the message:

> (XEN) ACPI: IOAPIC (id[0x06] address[0xfec00000] gsi_base[0])
> (XEN) IOAPIC[0]: apic_id 6, version 33, address 0xfec00000, GSI 0-23
> (XEN) ACPI: IOAPIC (id[0x07] address[0xfec20000] gsi_base[24])
> (XEN) IOAPIC[1]: apic_id 7, version 33, address 0xfec20000, GSI 24-55

> However, when Xen tries to setup the IOMMU interrupt remapping table using 
> IVHD
> entries, there is only one IOAPIC (IOAPIC 1 with apic_id 7).

> (XEN) AMD-Vi: IVHD Device Entry: type 0x48 id 0 flags 0
> (XEN) AMD-Vi: IVHD Special: 0000:00:00.1 variety 0x1 handle 0x7
> (XEN) IVHD Error: no information for IO-APIC 0x6
> (XEN) AMD-Vi: Error initialization

> In this case, if we should be able to look at the IVHD to correlate IOAPIC ID 
> (0 or 1)
> from the "handle" field and map it back to the BDF to setup the remapping 
> table.

> CASE2: BIOS disable the IOAPIC in the I/O bridge (RD890s chipset)
> This happens in the case when we were testing the per-device interrupt 
> remapping
> table patch. (I think this is the issue you might be seeing in one of the Xen 
> test system.)
> In this case, the MADT reports 1 IOAPIC while the IVRS contains two IVHD 
> entries with both
> entries have the "hahandle" set to "0".  Unfortunately, in this case, there 
> is no obvious
> workaround, and the current solution is to disable IOMMU.

> I am working with some of the BIOS engineers and vendors to try to issue 
> root-cause
> and provide BIOS update.

> Jan, Sander:
> Could you please provide the system information:
> 1. Motherboard vendor
> 2. BIOS version

Suravee,

1. My motherboard is a "890FXA-GD70" from MSI 
(http://www.msi.com/product/mb/890FXA-GD70.html)
2. As for the bios version:
        - I'm currently running 1.8 beta1, it's a beta bios. It boots and 
works, but has the problem you described above.

        - I have tried all newer bioses up to the latest bios (1.15), but with 
these bioses the system halts during boot when the iommu option in the bios is 
enabled.
          With xen it halts right after the "(XEN) IVHD Error: no information 
for IO-APIC 0x6", so it could be another problem initializing the iommu i'm 
afraid.
          It also halts while trying to boot a baremetal linux kernel.
          When the iommu is disabled it boots fine.

I hope a more direct approach of the bios engineers has some result, customer 
support most of the time have no clue and react like a firewall, bouncing and 
dropping all the packets :-(

Thanks for trying and picking it up !
--

Sander

> Thank you,

> Suravee


>>
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@xxxxxxxxxxxxx
>> http://lists.xen.org/xen-devel
>>





_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel