Re: [RFC PATCH] xen/amd-iommu: Add interrupt remapping quirk for ath11k

[Jason Andryuk <jason.andryuk@xxxxxxx>]

On 2025-02-27 03:54, Jan Beulich wrote:
> On 26.02.2025 22:11, Jason Andryuk wrote:
> > Sometimes we have to quirk the PCI IRTE to use a non-zero remap_index
> > corresponding to the guest's view of the MSI data register.  The MSI
> > data guest vector equals interrupt remapping table index.
> >
> > The ath11k wifi device does unusual things with MSIs.  The driver lets
> > Linux program the MSI capability.  Linux internally caches the MSI data
> > it thinks it programmed.  It sets its affinity to CPU0.  The ath11k
> > driver then reads the MSI address from the PCI configuration space.  The
> > MSI address and cached data are then passed to other components on the
> > same card to generate MSI interrupts.
> I'm curious whether it's known how e.g. KVM deals with this.
There were some vfio patches that did not get merged, FWICT.  A Linux 
patch added a quirk to allow the guest to read the hardware MSI values. 
QEMU intercepted access to a memory region of a BAR and swapped guest 
MSI values for hardware MSI values.
https://lore.kernel.org/ath11k/20240812170014.1583783-1-alex.williamson@xxxxxxxxxx/

I tried something similar, but abandoned it.  The ath11k driver uses 
Linux's cached value of the guest MSI data and passes that to the 
device.   It doesn't re-read the hardware value out of the configuration 
space.  That made me think using the guest data as an index would be a 
better workaround.

> > Signed-off-by: Xenia Ragiadakou <xenia.ragiadakou@xxxxxxx>
> > Signed-off-by: Jason Andryuk <jason.andryuk@xxxxxxx>
> Just to clarify: Who's the original patch author? The common expectation
> is that the first S-o-b: matches From:.
I took Xenia's changes to xen/drivers/passthrough/pci.c and 
xen/include/xen/pci.h from an earlier patch and re-used them.  I wrote 
the rest, so I put myself in the Form: line.
> > ---
> > Is something like this feasible for inclusion upstream?  I'm asking
> > before I look into what it would take to support Intel.
> Well, I wouldn't outright say "no". It needs to be pretty clear that this
> doesn't put at risk the "normal" cases. Which is going to be somewhat
> difficult considering how convoluted the involved code (sadly) is. See
> also the commentary related remark at the very bottom.
Ok

> > e.g. Replace amd_iommu_perdev_intremap with something generic.
> >
> > The ath11k device supports and tries to enable 32 MSIs.  Linux in PVH
> > dom0 and HVM domU fails enabling 32 and falls back to just 1, so that is
> > all that has been tested.
> >
> > Using msi_desc->gvec should be okay since with posted interrupts - the
> > gvec is expected to match.
> >
> > hvm_pi_update_irte() changes the IRTE but not the MSI data in the PCI
> > capability, so that isn't suitable by itself.
> These last two paragraphs look to again be related to the VT-d aspect.
> Yet there's the middle one which apparently doesn't, hence I'm uncertain
> I read all of this as it's intended.
Sorry, I was just putting down thoughts.  Yes, the last two were 
thinking about VT-d integration.
In terms of the number of MSI, I wanted to highlight that I only tested 
with 1 MSI since I always worry about code I haven't tested.
> > --- a/xen/drivers/passthrough/amd/iommu_intr.c
> > +++ b/xen/drivers/passthrough/amd/iommu_intr.c
> > @@ -543,6 +543,31 @@ int cf_check amd_iommu_msi_msg_update_ire(
> >       if ( !msg )
> >           return 0;
> >   
> > +    if ( pdev->gvec_as_irte_idx && amd_iommu_perdev_intremap )
> > +    {
> > +        int new_remap_index = 0;
> > +        if ( msi_desc->gvec )
> > +        {
> > +            printk("%pp: gvec remap_index %#x -> %#x\n", &pdev->sbdf,
> > +                   msi_desc->remap_index, msi_desc->gvec);
> > +            new_remap_index = msi_desc->gvec;
> > +        }
> > +
> > +        if ( new_remap_index && new_remap_index != msi_desc->remap_index &&
> > +             msi_desc->remap_index != -1 )
> > +        {
> > +            /* Clear any existing entries */
> > +            update_intremap_entry_from_msi_msg(iommu, bdf, nr,
> > +                                               &msi_desc->remap_index,
> > +                                               NULL, NULL);
> > +
> > +            for ( i = 0; i < nr; ++i )
> > +                msi_desc[i].remap_index = -1;
> > +
> > +            msi_desc->remap_index = new_remap_index;
> You zap nr entries, and then set only 1? Doesn't the zapping loop need to
> instead be a setting one? Perhaps with a check up front that the last value
> used will still fit in 8 bits? Or else make applying the quirk conditional
> upon nr == 1?
The code below here sets all `nr` entries on success:

    rc = update_intremap_entry_from_msi_msg(iommu, bdf, nr,
                                            &msi_desc->remap_index,
                                            msg, &data);
    if ( !rc )
    {
        for ( i = 1; i < nr; ++i )
            msi_desc[i].remap_index = msi_desc->remap_index + i;
        msg->data = data;
    }

    return rc;

The single passed in &msi_desc->remap_index is used as the start value 
(when < INTREMAP_MAX_ENTRIES) or is assigned a value.  Checking 
remap_index + nr fits is a good idea.
Maybe all the remap_index settting should be moved into 
update_intremap_entry_from_msi_msg()?
> > --- a/xen/drivers/passthrough/pci.c
> > +++ b/xen/drivers/passthrough/pci.c
> > @@ -306,6 +306,17 @@ static void apply_quirks(struct pci_dev *pdev)
> > +#define QCA6390_DEVICE_ID              0x1101
> > +#define QCN9074_DEVICE_ID              0x1104
> > +#define WCN6855_DEVICE_ID              0x1103
> > +        { PCI_VENDOR_ID_QCOM, QCA6390_DEVICE_ID },
> > +        { PCI_VENDOR_ID_QCOM, QCN9074_DEVICE_ID },
> > +        { PCI_VENDOR_ID_QCOM, WCN6855_DEVICE_ID },
> > +    };
> May I ask what's the source of information on which specific devices are
> affected by this anomalous behavior? Just the Linux driver?
These are just taken from the Linux driver.  Tested with WCN6855 0x1103.

> I'm also uncertain #define-s are very useful in such a case. Raw hex numbers
> in the table with a comment indicating the device name ought to be as fine.
Ok.

> > --- a/xen/include/xen/pci.h
> > +++ b/xen/include/xen/pci.h
> > @@ -127,6 +127,8 @@ struct pci_dev {
> >       /* Device with errata, ignore the BARs. */
> >       bool ignore_bars;
> >   
> > +    bool gvec_as_irte_idx;
> > +
> >       /* Device misbehaving, prevent assigning it to guests. */
> >       bool broken;
> >   
> Overall more commentary would be needed throughout the patch. This field is
> just one example where some minimal explanation is missing.
Ok.

Thanks for taking a look.

Regards,
Jason