[RFC PATCH] xen/amd-iommu: Add interrupt remapping quirk for ath11k

[Jason Andryuk <jason.andryuk@xxxxxxx>]

Sometimes we have to quirk the PCI IRTE to use a non-zero remap_index
corresponding to the guest's view of the MSI data register.  The MSI
data guest vector equals interrupt remapping table index.

The ath11k wifi device does unusual things with MSIs.  The driver lets
Linux program the MSI capability.  Linux internally caches the MSI data
it thinks it programmed.  It sets its affinity to CPU0.  The ath11k
driver then reads the MSI address from the PCI configuration space.  The
MSI address and cached data are then passed to other components on the
same card to generate MSI interrupts.

With Xen, vPCI and QEMU PCI passthrough have a guest idea of the MSI
address and data.  But Xen programs the actual hardware with its own
address and data.  With per-device IRT, xen uses index 0.  When the
ath11k driver passes the guest address and data to the hardware, it
generates faults when there is no IRTE for the guest data (~0x25).

To work around this, we can, for per-device IRTs, program the hardware
to use the guest data & associated IRTE.  The address doesn't matter
since the IRTE handles that, and the Xen address & vector can be used as
expected.

For vPCI, the guest MSI data is available at the time of initial MSI
setup, but that is not the case for HVM.  With HVM, the initial MSI
setup is done when PHYSDEVOP_map_pirq is called, but the guest vector is
only available later when XEN_DOMCTL_bind_pt_irq is called.  In that
case, we need to tear down and create a new IRTE.  This later location
can also handle vPCI.

Add pirq_guest_bind_gvec to plumb down the gvec without modifying all
call sites.  Use msi_desc->gvec to pass through the desired value.

Only tested with AMD-Vi.  Requires per-device IRT.  With AMD-Vi, the
number of MSIs is passed in, but a minimum of a page is allocated for
the table.  The vector is 8 bits giving indices 0-255.  Even with 128bit
IRTEs, 16 bytes, 1 page 4096 / 16 = 256 entries, so we don't have to
worry about overflow.  N MSIs can only have the last one at 255, so the
guest can't expect to have N vectors starting above 255 - N.

Signed-off-by: Xenia Ragiadakou <xenia.ragiadakou@xxxxxxx>
Signed-off-by: Jason Andryuk <jason.andryuk@xxxxxxx>
---
Is something like this feasible for inclusion upstream?  I'm asking
before I look into what it would take to support Intel.

e.g. Replace amd_iommu_perdev_intremap with something generic.

The ath11k device supports and tries to enable 32 MSIs.  Linux in PVH
dom0 and HVM domU fails enabling 32 and falls back to just 1, so that is
all that has been tested.

Using msi_desc->gvec should be okay since with posted interrupts - the
gvec is expected to match.

hvm_pi_update_irte() changes the IRTE but not the MSI data in the PCI
capability, so that isn't suitable by itself.
---
 xen/arch/x86/include/asm/msi.h           |  3 ++-
 xen/arch/x86/irq.c                       | 13 +++++++++++-
 xen/arch/x86/msi.c                       |  1 +
 xen/drivers/passthrough/amd/iommu_intr.c | 25 ++++++++++++++++++++++++
 xen/drivers/passthrough/pci.c            | 24 +++++++++++++++++++++++
 xen/drivers/passthrough/x86/hvm.c        |  3 ++-
 xen/include/xen/irq.h                    |  2 ++
 xen/include/xen/pci.h                    |  2 ++
 8 files changed, 70 insertions(+), 3 deletions(-)

diff --git a/xen/arch/x86/include/asm/msi.h b/xen/arch/x86/include/asm/msi.h
index 378b85ee94..ea1004af14 100644
--- a/xen/arch/x86/include/asm/msi.h
+++ b/xen/arch/x86/include/asm/msi.h
@@ -107,7 +107,8 @@ struct msi_desc {
     } msi_attrib;
 
     bool irte_initialized;
-    uint8_t gvec;            /* guest vector. valid when pi_desc isn't NULL */
+    uint8_t gvec;            /* guest vector. valid when pi_desc isn't NULL or
+                                when pci_dev gvec_as_irte_idx is true */
     const struct pi_desc *pi_desc; /* pointer to posted descriptor */
 
     struct list_head list;
diff --git a/xen/arch/x86/irq.c b/xen/arch/x86/irq.c
index ff3ac832f4..3fc73feaea 100644
--- a/xen/arch/x86/irq.c
+++ b/xen/arch/x86/irq.c
@@ -1600,7 +1600,8 @@ int pirq_shared(struct domain *d, int pirq)
     return shared;
 }
 
-int pirq_guest_bind(struct vcpu *v, struct pirq *pirq, int will_share)
+int pirq_guest_bind_gvec(struct vcpu *v, struct pirq *pirq, int will_share,
+                         uint8_t gvec)
 {
     struct irq_desc         *desc;
     irq_guest_action_t *action, *newaction = NULL;
@@ -1674,7 +1675,12 @@ int pirq_guest_bind(struct vcpu *v, struct pirq *pirq, 
int will_share)
                                           &cpu_online_map) )
                 affinity = desc->affinity;
             if ( affinity )
+            {
+                if ( gvec && desc->msi_desc )
+                    desc->msi_desc->gvec = gvec;
+
                 desc->handler->set_affinity(desc, affinity);
+            }
         }
 
         desc->status &= ~IRQ_DISABLED;
@@ -1730,6 +1736,11 @@ int pirq_guest_bind(struct vcpu *v, struct pirq *pirq, 
int will_share)
     return rc;
 }
 
+int pirq_guest_bind(struct vcpu *v, struct pirq *pirq, int will_share)
+{
+    return pirq_guest_bind_gvec(v, pirq, will_share, 0);
+}
+
 static irq_guest_action_t *__pirq_guest_unbind(
     struct domain *d, struct pirq *pirq, struct irq_desc *desc)
 {
diff --git a/xen/arch/x86/msi.c b/xen/arch/x86/msi.c
index bf5b71822e..cef2987038 100644
--- a/xen/arch/x86/msi.c
+++ b/xen/arch/x86/msi.c
@@ -487,6 +487,7 @@ static struct msi_desc *alloc_msi_entry(unsigned int nr)
         entry[nr].remap_index = -1;
         entry[nr].pi_desc = NULL;
         entry[nr].irte_initialized = false;
+        entry[nr].gvec = 0;
     }
 
     return entry;
diff --git a/xen/drivers/passthrough/amd/iommu_intr.c 
b/xen/drivers/passthrough/amd/iommu_intr.c
index c0273059cb..2e228d2c21 100644
--- a/xen/drivers/passthrough/amd/iommu_intr.c
+++ b/xen/drivers/passthrough/amd/iommu_intr.c
@@ -543,6 +543,31 @@ int cf_check amd_iommu_msi_msg_update_ire(
     if ( !msg )
         return 0;
 
+    if ( pdev->gvec_as_irte_idx && amd_iommu_perdev_intremap )
+    {
+        int new_remap_index = 0;
+        if ( msi_desc->gvec )
+        {
+            printk("%pp: gvec remap_index %#x -> %#x\n", &pdev->sbdf,
+                   msi_desc->remap_index, msi_desc->gvec);
+            new_remap_index = msi_desc->gvec;
+        }
+
+        if ( new_remap_index && new_remap_index != msi_desc->remap_index &&
+             msi_desc->remap_index != -1 )
+        {
+            /* Clear any existing entries */
+            update_intremap_entry_from_msi_msg(iommu, bdf, nr,
+                                               &msi_desc->remap_index,
+                                               NULL, NULL);
+
+            for ( i = 0; i < nr; ++i )
+                msi_desc[i].remap_index = -1;
+
+            msi_desc->remap_index = new_remap_index;
+        }
+    }
+
     rc = update_intremap_entry_from_msi_msg(iommu, bdf, nr,
                                             &msi_desc->remap_index,
                                             msg, &data);
diff --git a/xen/drivers/passthrough/pci.c b/xen/drivers/passthrough/pci.c
index e1a09344df..7031aedb94 100644
--- a/xen/drivers/passthrough/pci.c
+++ b/xen/drivers/passthrough/pci.c
@@ -306,6 +306,17 @@ static void apply_quirks(struct pci_dev *pdev)
         { PCI_VENDOR_ID_INTEL, 0x6fa0 },
         { PCI_VENDOR_ID_INTEL, 0x6fc0 },
     };
+    static const struct {
+        uint16_t vendor, device;
+    } hide_irt[] = {
+#define PCI_VENDOR_ID_QCOM             0x17cb
+#define QCA6390_DEVICE_ID              0x1101
+#define QCN9074_DEVICE_ID              0x1104
+#define WCN6855_DEVICE_ID              0x1103
+        { PCI_VENDOR_ID_QCOM, QCA6390_DEVICE_ID },
+        { PCI_VENDOR_ID_QCOM, QCN9074_DEVICE_ID },
+        { PCI_VENDOR_ID_QCOM, WCN6855_DEVICE_ID },
+    };
     unsigned int i;
 
     for ( i = 0; i < ARRAY_SIZE(ignore_bars); i++)
@@ -316,6 +327,19 @@ static void apply_quirks(struct pci_dev *pdev)
              * from trying to size the BARs or add handlers to trap accesses.
              */
             pdev->ignore_bars = true;
+
+    for ( i = 0; i < ARRAY_SIZE(hide_irt); i++)
+    {
+        if ( vendor == hide_irt[i].vendor &&
+             device == hide_irt[i].device )
+        {
+            pdev->gvec_as_irte_idx = true;
+            printk("%pp %04x:%04x quirk gvec as intr remap index\n",
+                   &pdev->sbdf, hide_irt[i].vendor, hide_irt[i].device);
+            if ( !amd_iommu_perdev_intremap )
+                printk("gvec quirk requires per-device intr remap!\n");
+        }
+    }
 }
 
 static struct pci_dev *alloc_pdev(struct pci_seg *pseg, u8 bus, u8 devfn)
diff --git a/xen/drivers/passthrough/x86/hvm.c 
b/xen/drivers/passthrough/x86/hvm.c
index f5faff7a49..5d17f93b06 100644
--- a/xen/drivers/passthrough/x86/hvm.c
+++ b/xen/drivers/passthrough/x86/hvm.c
@@ -307,7 +307,8 @@ int pt_irq_create_bind(
              */
             pirq_dpci->dom = d;
             /* bind after hvm_irq_dpci is setup to avoid race with irq 
handler*/
-            rc = pirq_guest_bind(d->vcpu[0], info, 0);
+            rc = pirq_guest_bind_gvec(d->vcpu[0], info, 0,
+                                      pirq_dpci->gmsi.gvec);
             if ( rc == 0 && pt_irq_bind->u.msi.gtable )
             {
                 rc = msixtbl_pt_register(d, info, pt_irq_bind->u.msi.gtable);
diff --git a/xen/include/xen/irq.h b/xen/include/xen/irq.h
index 95034c0d6b..96109d6ebe 100644
--- a/xen/include/xen/irq.h
+++ b/xen/include/xen/irq.h
@@ -192,6 +192,8 @@ extern void pirq_guest_eoi(struct pirq *pirq);
 extern void desc_guest_eoi(struct irq_desc *desc, struct pirq *pirq);
 extern int pirq_guest_unmask(struct domain *d);
 extern int pirq_guest_bind(struct vcpu *v, struct pirq *pirq, int will_share);
+extern int pirq_guest_bind_gvec(struct vcpu *v, struct pirq *pirq,
+                                int will_share, uint8_t gvec);
 extern void pirq_guest_unbind(struct domain *d, struct pirq *pirq);
 extern void pirq_set_affinity(struct domain *d, int pirq,
                               const cpumask_t *mask);
diff --git a/xen/include/xen/pci.h b/xen/include/xen/pci.h
index 4f12bcf089..14afd78f75 100644
--- a/xen/include/xen/pci.h
+++ b/xen/include/xen/pci.h
@@ -127,6 +127,8 @@ struct pci_dev {
     /* Device with errata, ignore the BARs. */
     bool ignore_bars;
 
+    bool gvec_as_irte_idx;
+
     /* Device misbehaving, prevent assigning it to guests. */
     bool broken;
 
-- 
2.34.1