Re: [Xen-devel] XSA-36 / howto fix broken IVRS ACPI table

[Eric Shelton <eshelton@xxxxxxxxx>]

> Hello,
>
> I got a patched BIOS (version F8c) from Gigabyte which removes the 2nd IOAPIC
> entry (for device 0000:00:00.1) from the IVRS table.
>
> This causes Xen to enable per-device vector maps:
> (XEN) AMD-Vi: IOMMU 0 Enabled.
> (XEN) AMD-Vi: Enabling per-device vector maps
> (XEN) I/O virtualisation enabled
> (XEN)  - Dom0 mode: Relaxed
> (XEN) Interrupt remapping enabled
>
> However the problem seems not really to be fixed:
> Interrupts generated within one domain can still harm other domains which at
> least causes the kernel within these other domains to disable interrupts.
>
> Before going to investigate/debug this problem, I want to know if one IOAPIC
> is sufficient as the AMD 970 chipset seems to have one IOAPIC related to the
> northbridge and one for the southbridge.
> The one currently enabled is the southbridge one (0000:00:14.0). Does it also
> support devices connected to the northbridge or needs the northbridge one to
> be enabled, too?
> Are there other limitations/problems to expect regarding the disabled
> northbridge IOAPIC?
>
> Thanks & best regards
> Hans

I have looked into it.  Your best references are the 970 BIOS and
Register Programming guides, to see how it _should_ be configured.
The SB8xx guides are useful if you want to see what is going on in the
SB (SB950 has some minor differences (x4 vs x2 slot), but no formal
documentation is released).  If you _really_ want to dig into it, AMD
released BIOS code via the coreboot project which implements the
configuration shown in the guides.

The problem: Because the NB IOAPIC is not configured, any interrupts
from the x16 and x1 PCIE slots are on 1 of only 4 signal lines sent to
the SB IOAPIC: INTA, INTB, INTC, and INTD (it appears a swizzle is
applied to the slots, which may tend to spread the interrupts among
the 4 pins - although slot choice will affect that).

INTA (SB pin 16) is shared with the built-in HD audio and the rear USB
3 controller.
INTB (SB pin 17) is shared with the any interrupts generated on the x4
slot, the built-in RTL ethernet, EHCI on 3 USB 2 controllers, and the
IDE device (if ports 4+5 are so configured).
INTC (SB pin 18) is shared with OHCI on the same 3 USB 2 controllers,
a 4th USB 2 controller, and the front USB 3 controller (also, if your
x16 video card interrupts via PINA, it will go here).
INTD (SB pin 19) is shared with SATA.
I think pins 21-23 are for bridge interrupts.

>From your dmesg, it looks like your Intel ethernet is on pin 19.

Thus, most of the devices are sharing interrupt vectors, with INTC
being particularly crowded (I recall you mentioning USB previously).
With a combined total of 56 available GSIs (32 NB, 24 SB), almost
_none_ of these need to be shared (SB may be a little tight).  On top
of that, although I'm not sure if the BIOS should configure/enable it,
it is likely that none of your devices are using MSIs.

The solution:  My understanding is that AMD provides vendors with CIMX
code for providing the same standard NB interrupt configuration shown
in the BIOS guide, which does a far, far better job of spreading out
the interrupts.  Apparently they are not calling that code.  They
should.  Plus, it would be helpful to give the GPP (x4 slot)
interrupts 1 or 2 dedicated SB pins, seeing as someone might put a
busy device there.

Funny thing is, if they enable the NB IOAPIC, the original 2nd IOAPIC
entry they just removed will be correct (suggesting they called some
of the CIMX code).

I would like to hear if they will make this or a similar correction.

- Eric

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel