Re: [RFC PATCH] xen/amd-iommu: Add interrupt remapping quirk for ath11k

[Andrew Cooper <andrew.cooper3@xxxxxxxxxx>]

On 13/03/2025 3:47 pm, Roger Pau Monné wrote:
> On Thu, Mar 13, 2025 at 11:30:28AM -0400, Jason Andryuk wrote:
>> On 2025-02-27 05:23, Roger Pau Monné wrote:
>>> On Wed, Feb 26, 2025 at 04:11:25PM -0500, Jason Andryuk wrote:
>>>> The ath11k device supports and tries to enable 32 MSIs.  Linux in PVH
>>>> dom0 and HVM domU fails enabling 32 and falls back to just 1, so that is
>>>> all that has been tested.
>>> DYK why it fails to enable 32?
>> In Linux msi_capability_init()
>>
>>         /* Reject multi-MSI early on irq domain enabled architectures */
>>         if (nvec > 1 && !pci_msi_domain_supports(dev,
>> MSI_FLAG_MULTI_PCI_MSI, ALLOW_LEGACY))
>>                 return 1;
>>
>> MSI_FLAG_MULTI_PCI_MSI is only set for AMD and Intel interrupt remapping,
>> and Xen PVH and HVM don't have either of those.  They are using "VECTOR", so
>> this check fails.
> Oh, interesting.  So classic PV MSI domain supports
> MSI_FLAG_MULTI_PCI_MSI, even when no IOMMU is exposed there either.
>
> Thanks, so it's nothing specific to Xen, just how Linux works.

This is something which TGLX and I have discussed in the past.  It is a
mistake for any x86 system to do MSI multi-message without an IOMMU.

MSI multi-message gets you a power-of-2, aligned, block of vectors, up
to a maximum of 32, which must always target the same CPU.

The LAPIC prioritisation is on groups of 16, aligned, vectors.

If MSI has 16 or fewer vectors, then any interrupt causes all others to
be blocked owing to LAPIC behaviour.

With 32 vectors, you can get two vectors (one from the first 16, one
from the second 16) where the higher vector can interrupt the lower
one.  And you pay 32 vectors for this.

With the IOMMU, every message gets a controllable CPU and controllable
priority, because they come from the IRTE, not the device.

Removing Multi-MSI support makes vector allocation much easier because
you you never need to allocate/move blocks.

~Andrew