[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[PATCH 2/3] x86/vmsi: add support for extended destination ID in address field


  • To: <xen-devel@xxxxxxxxxxxxxxxxxxxx>
  • From: Roger Pau Monne <roger.pau@xxxxxxxxxx>
  • Date: Thu, 20 Jan 2022 16:23:18 +0100
  • Arc-authentication-results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none
  • Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=HoF6XYV9k/waGwdbkp6UIajlOGJSYD9wsXW9TkdA094=; b=Y7oKeoS8KEkC7zk9P6bz4TntTAi0jIfEfCL53HKyfgFd2Eb2byVuO9XOvMhg4oRnOMMiYxYqeEm2pKN/9mdr4U30zdKkAeSK8IJslcGp9DcNki/7GWptWSHWnRmf/MU0zLdQ1LscRCL7w0xMNsznjCs4FyMToj2z3bj/T7954nyxe//urPncgr6uSM1HNpuxm6cbqWhfRU0MDjhfOyUjLFPmGn5AkmS/JJ7UTy1G8zkrkNTX+dQReFWEYV6NRqGzPe6prb8K4aOFRtDauM6Sg8t86Y0o8UuhqM7u4+WQsH4isU0KMJXwO5uWTuo2KVsYRIWRUfbLgGaE3EGAwyMMpQ==
  • Arc-seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=fkvYEg2hSMrh/6MtfyT5eY8SZB5WXFJRosoiM3JUs0r3yvXQg1XQeUdyh3LaUAn1e0fnSxqDJQgp4Q0d/+pWDya8NqmtkxxprOC7gEcV4+xI30+lOTHKfNI2Gyg8XTbp6k8EyWDb1y46yRJcOnR4rW2Lzo5AH/5D7UyqMibMT2DBRd+6iXJYNUh7Q09R2myB8XvCcBsI1FYr+veyegZf02UwrdosfWcstUUn0xWEfmOAPtZb8qGikb3gvePhUfl+u4V+VAx2YjfzbgI3RyauSx9A1IyuyiRTgGEDZu5Fg9Dl7kTlfraaT6Aj3C+Bf11g28WC+FOEpMEpciiYWve1dA==
  • Authentication-results: esa1.hc3370-68.iphmx.com; dkim=pass (signature verified) header.i=@citrix.onmicrosoft.com
  • Cc: Roger Pau Monne <roger.pau@xxxxxxxxxx>, Jan Beulich <jbeulich@xxxxxxxx>, Andrew Cooper <andrew.cooper3@xxxxxxxxxx>, Wei Liu <wl@xxxxxxx>, "George Dunlap" <george.dunlap@xxxxxxxxxx>, Julien Grall <julien@xxxxxxx>, "Stefano Stabellini" <sstabellini@xxxxxxxxxx>, Paul Durrant <paul@xxxxxxx>
  • Delivery-date: Thu, 20 Jan 2022 15:24:40 +0000
  • Ironport-data: A9a23:D/CauqLVxQv7plCqFE+RTpIlxSXFcZb7ZxGr2PjKsXjdYENS3z1Sy zEXXWnSafyJZGPzcogkPo62phgC7MDWytBkSgdlqX01Q3x08seUXt7xwmUcns+xwm8vaGo9s q3yv/GZdJhcokcxIn5BC5C5xZVG/fjgqoHUVaiUakideSc+EH170Us5w7Zj6mJVqYPR7z2l6 IuaT/L3YDdJ6xYsWo7Dw/vewP/HlK2aVAIw5jTSV9gS1LPtvyB94KYkDbOwNxPFrrx8RYZWc QphIIaRpQs19z91Yj+sfy2SnkciGtY+NiDW4pZatjTLbrGvaUXe345iXMfwZ3u7hB2pmsxD5 Pp/6aCfQD8YMb3Rw/4GDCVhRnQW0a1uoNcrIFC6uM2XiUbHb2Ht07NlC0Re0Y8wo7gtRzsUr LpBdW5LPkvra+GemdpXTsF2gcsuNo/zNZ43sXB81zDJS/0hRPgvRo2Uv48BjW9s2aiiG97ne pFCVwZlbi7rIDRvHAcZKYwmsd6R0yyXnzpw9wvO+PtfD3Lo5A5s1LngNvLFd9rMQt9a9m6Dv X7P9Wn9BhAcNfScxCCD/3bqgfXA9QvjUZ8WPK218LhtmlL77nweDlgaWEW2pdG9i1WiQJRPJ koM4C0soKMuskuxQbHVXRe1vXqFtR40QMdLHqsx7wTl4rXQyxaUAC4DVDEpVTA9nJZoH3pwj AbPxo63Q2w02FGIdZ6D3peRsQGeFhpJEUALXR4LXRZe+cK8nLhm23ojUe1fOKKyi9T0HxT5z DaLsDUyit0vsCIb60mo1QuZ2mzx//AlWiZwv1yKBTz9smuVcab8P9TwgWU3+8qsO2pworOpm HEf0/aT4+kVZX1mvHzcGb5ddF1FChvsDdE9vbKNN8Vwn9hO0yT6FWy13N2YDB01WirjUWW4C HI/QSsLuPdu0IKCNMebmb6ZBcUw1rTHHt/4TP3SZdcmSsEvKFXZpX4+ORDMhjCFfK0QfUcXY 8vznSGEVi5yNEia5GDuG7d1PUEDm0jSOl8/tbiklk/6gNJylVaeSKsfMUvmUwzKxPjsnekhy P4Gb5Hi40wGCIXWO3CLmaZOcwxiBSVlVPje9pwGHsbec1EOMDxwVJfsLUYJJtYNc1J9zLmYp xlQmyZwlTLCuJEwAVzaMy84MOK2A88XQLBSFXVEAGtEEkMLOO6HxKwea4E2bf8g8ulixuRzV P4LZ4OLBfEnd9gN02p1gUDVoNMweRK1qxiJOib5MjEzc4Q5H17C+8P+fxup/y4LV3Llucw7q rym9wXaXZtcGFgyUJeIMKqinwGroHwQuONuRE+UcNNdT1rhrdpxICvrg/5pf8xVcUffxiGX3 hq9CAsDobWfuJc89dTE3PjWr4qgH+ZkMFBdGm3XseS/OSXApzLxyo5cSueYOzvaUTqsqqmlY OxUydD6MeEGwwkW49YtTe4zwPtntdX1prJcwgB1J1nxbgymWuF6P32L/chTrakRlLVXjhS7B xCU8d5ANLTXZM68SAwNJBAoZ/io3O0PnmWA9uw8JUj36XMl/LeDVkkObRCAhDYEcelwOYIhh +wgpNQX+0q0jR9zaoSKiSVd9mKtKH0cUvp46sFGUdGz0gd7mEtfZZH8CzPt5MDdYtpBBUAmP zuIifeQnL9b3EfDLyI+GHWlMTCxXnjSVMSmFGM/Gmk=
  • Ironport-hdrordr: A9a23:AAMgsawrqdq7ekTpSHQoKrPxtuskLtp133Aq2lEZdPULSKKlfp GV88jziyWZtN9wYhEdcdDpAtjmfZq6z+8L3WBxB8bfYOCCggqVxe5ZnPLfKlHbakjDH6tmpN 1dmstFea3N5DpB/L7HCWCDer5KqrT3k9HLuQ6d9QYRcegDUdAQ0+4TMHf8LqQZfngjOXJvf6 DsmPav6gDQM0g/X4CePD0oTuLDr9rEmNbPZgMHPQcu7E2rgSmz4LD3PhCE1lNGOgk/jYsKwC zgqUjU96+ju/a0xlv10HLS1Y1fnJ/ExsFYDMKBp8AJInHHixquZq5mR7qe1QpF7d2H2RIPqp 3hsh0gN8N85zf4eXy0mwLk303a3DMn+xbZuBalqEqmhfa8aCMxCsJHi44cWADe8VAcsNZ117 8O936FtrJMZCmw3RjV1pztbVVHh0C0qX0tnao4lHpES7YTb7dXsMg24F5VKpEdByj3gbpXXt WGNPusp8q+TGnqLUww5gJUsZmRtzUIb1i7q3E5y4yoO2M8pgE886MarPZv6EvouqhNDKWs3N 60QJiAoos+OfP+XZgNd9vpfvHHf1AlOSi8S156AW6XXJ3vaEi94KIe3t0OlZWXkdozvd0PpK g=
  • Ironport-sdr: GsDhXFmZr69eVszDg7x+v1nEvaUn6bhScPyfOd/mkKJy5u543Hgp/I4Bd7PWoerUnUTMj2lMRU XqPg6zGjwUYFuC141pbtskOEa9SqPit+DFO0vrmPnP3jbQR7vsDEE4JV4ywcAwuUVNisPo3U5x NnTwDZ99Akbm6iMMrCJPMMLVyZM78LGFl+wdKmNV/6+cSaIngUsRCNA2D4hDdQhSlKfWPZeBKX kyjhvLBOrwynYyNpuPfdf19yt5/f6sEb9DbSg/VK3LG0+/R89MXNxc54p72ObSmXZnqtFlg7bB bt4HLCwkme0wO5Mqrycwr+aL
  • List-id: Xen developer discussion <xen-devel.lists.xenproject.org>

Both QEMU/KVM and HyperV support using bits 11:5 from the MSI address
field in order to store the high part of the target APIC ID. This
allows expanding the maximum APID ID usable without interrupt
remapping support from 255 to 32768.

Note the interface used by QEMU for emulated devices (via the
XEN_DMOP_inject_msi hypercall) already passes both the address and
data fields into Xen for processing, so there's no need for any change
to QEMU there.

However for PCI passthrough devices QEMU uses the
XEN_DOMCTL_bind_pt_irq hypercall which does need an addition to the
gflags field in order to pass the high bits of the APIC destination
ID.

Introduce a new CPUID flag to signal the support for the feature. The
introduced flag covers both the support for extended ID for the
IO-APIC RTE and the MSI address registers. Such flag is currently only
exposed when the domain is using vPCI (ie: a PVH dom0).

Signed-off-by: Roger Pau Monné <roger.pau@xxxxxxxxxx>
---
 xen/arch/x86/hvm/irq.c              |  3 ++-
 xen/arch/x86/hvm/vmsi.c             | 13 ++++++++++---
 xen/arch/x86/include/asm/hvm/hvm.h  |  5 +++--
 xen/arch/x86/include/asm/msi.h      |  1 +
 xen/arch/x86/traps.c                |  3 +++
 xen/drivers/passthrough/x86/hvm.c   | 10 +++++++---
 xen/include/public/arch-x86/cpuid.h |  6 ++++++
 xen/include/public/domctl.h         |  1 +
 8 files changed, 33 insertions(+), 9 deletions(-)

diff --git a/xen/arch/x86/hvm/irq.c b/xen/arch/x86/hvm/irq.c
index 52aae4565f..b9b5182369 100644
--- a/xen/arch/x86/hvm/irq.c
+++ b/xen/arch/x86/hvm/irq.c
@@ -383,7 +383,8 @@ int hvm_set_pci_link_route(struct domain *d, u8 link, u8 
isa_irq)
 int hvm_inject_msi(struct domain *d, uint64_t addr, uint32_t data)
 {
     uint32_t tmp = (uint32_t) addr;
-    uint8_t  dest = (tmp & MSI_ADDR_DEST_ID_MASK) >> MSI_ADDR_DEST_ID_SHIFT;
+    unsigned int dest = (MASK_EXTR(tmp, MSI_ADDR_EXT_DEST_ID_MASK) << 8) |
+                        MASK_EXTR(tmp, MSI_ADDR_DEST_ID_MASK);
     uint8_t  dest_mode = !!(tmp & MSI_ADDR_DESTMODE_MASK);
     uint8_t  delivery_mode = (data & MSI_DATA_DELIVERY_MODE_MASK)
         >> MSI_DATA_DELIVERY_MODE_SHIFT;
diff --git a/xen/arch/x86/hvm/vmsi.c b/xen/arch/x86/hvm/vmsi.c
index 13e2a190b4..ec0f3bc13f 100644
--- a/xen/arch/x86/hvm/vmsi.c
+++ b/xen/arch/x86/hvm/vmsi.c
@@ -66,7 +66,7 @@ static void vmsi_inj_irq(
 
 int vmsi_deliver(
     struct domain *d, int vector,
-    uint8_t dest, uint8_t dest_mode,
+    unsigned int dest, unsigned int dest_mode,
     uint8_t delivery_mode, uint8_t trig_mode)
 {
     struct vlapic *target;
@@ -107,7 +107,9 @@ void vmsi_deliver_pirq(struct domain *d, const struct 
hvm_pirq_dpci *pirq_dpci)
 {
     uint32_t flags = pirq_dpci->gmsi.gflags;
     int vector = pirq_dpci->gmsi.gvec;
-    uint8_t dest = (uint8_t)flags;
+    unsigned int dest = MASK_EXTR(flags, XEN_DOMCTL_VMSI_X86_DEST_ID_MASK) |
+                        (MASK_EXTR(flags,
+                                   XEN_DOMCTL_VMSI_X86_EXT_DEST_ID_MASK) << 8);
     bool dest_mode = flags & XEN_DOMCTL_VMSI_X86_DM_MASK;
     uint8_t delivery_mode = MASK_EXTR(flags, XEN_DOMCTL_VMSI_X86_DELIV_MASK);
     bool trig_mode = flags & XEN_DOMCTL_VMSI_X86_TRIG_MASK;
@@ -123,7 +125,8 @@ void vmsi_deliver_pirq(struct domain *d, const struct 
hvm_pirq_dpci *pirq_dpci)
 }
 
 /* Return value, -1 : multi-dests, non-negative value: dest_vcpu_id */
-int hvm_girq_dest_2_vcpu_id(struct domain *d, uint8_t dest, uint8_t dest_mode)
+int hvm_girq_dest_2_vcpu_id(struct domain *d, unsigned int dest,
+                            unsigned int dest_mode)
 {
     int dest_vcpu_id = -1, w = 0;
     struct vcpu *v;
@@ -645,6 +648,8 @@ static unsigned int msi_gflags(uint16_t data, uint64_t 
addr, bool masked)
      */
     return MASK_INSR(MASK_EXTR(addr, MSI_ADDR_DEST_ID_MASK),
                      XEN_DOMCTL_VMSI_X86_DEST_ID_MASK) |
+           MASK_INSR(MASK_EXTR(addr, MSI_ADDR_EXT_DEST_ID_MASK),
+                     XEN_DOMCTL_VMSI_X86_EXT_DEST_ID_MASK) |
            MASK_INSR(MASK_EXTR(addr, MSI_ADDR_REDIRECTION_MASK),
                      XEN_DOMCTL_VMSI_X86_RH_MASK) |
            MASK_INSR(MASK_EXTR(addr, MSI_ADDR_DESTMODE_MASK),
@@ -835,6 +840,7 @@ void vpci_msi_arch_print(const struct vpci_msi *msi)
            msi->data & MSI_DATA_LEVEL_ASSERT ? "" : "de",
            msi->address & MSI_ADDR_DESTMODE_LOGIC ? "log" : "phys",
            msi->address & MSI_ADDR_REDIRECTION_LOWPRI ? "lowest" : "fixed",
+           (MASK_EXTR(msi->address, MSI_ADDR_EXT_DEST_ID_MASK) << 8) |
            MASK_EXTR(msi->address, MSI_ADDR_DEST_ID_MASK),
            msi->arch.pirq);
 }
@@ -904,6 +910,7 @@ int vpci_msix_arch_print(const struct vpci_msix *msix)
                entry->data & MSI_DATA_LEVEL_ASSERT ? "" : "de",
                entry->addr & MSI_ADDR_DESTMODE_LOGIC ? "log" : "phys",
                entry->addr & MSI_ADDR_REDIRECTION_LOWPRI ? "lowest" : "fixed",
+               (MASK_EXTR(entry->addr, MSI_ADDR_EXT_DEST_ID_MASK) << 8) |
                MASK_EXTR(entry->addr, MSI_ADDR_DEST_ID_MASK),
                entry->masked, entry->arch.pirq);
         if ( i && !(i % 64) )
diff --git a/xen/arch/x86/include/asm/hvm/hvm.h 
b/xen/arch/x86/include/asm/hvm/hvm.h
index b26302d9e7..f001b43a21 100644
--- a/xen/arch/x86/include/asm/hvm/hvm.h
+++ b/xen/arch/x86/include/asm/hvm/hvm.h
@@ -270,11 +270,12 @@ uint64_t hvm_get_guest_time_fixed(const struct vcpu *v, 
uint64_t at_tsc);
 
 int vmsi_deliver(
     struct domain *d, int vector,
-    uint8_t dest, uint8_t dest_mode,
+    unsigned int dest, unsigned int dest_mode,
     uint8_t delivery_mode, uint8_t trig_mode);
 struct hvm_pirq_dpci;
 void vmsi_deliver_pirq(struct domain *d, const struct hvm_pirq_dpci *);
-int hvm_girq_dest_2_vcpu_id(struct domain *d, uint8_t dest, uint8_t dest_mode);
+int hvm_girq_dest_2_vcpu_id(struct domain *d, unsigned int dest,
+                            unsigned int dest_mode);
 
 enum hvm_intblk
 hvm_interrupt_blocked(struct vcpu *v, struct hvm_intack intack);
diff --git a/xen/arch/x86/include/asm/msi.h b/xen/arch/x86/include/asm/msi.h
index e228b0f3f3..531b860e42 100644
--- a/xen/arch/x86/include/asm/msi.h
+++ b/xen/arch/x86/include/asm/msi.h
@@ -54,6 +54,7 @@
 #define MSI_ADDR_DEST_ID_SHIFT         12
 #define         MSI_ADDR_DEST_ID_MASK          0x00ff000
 #define  MSI_ADDR_DEST_ID(dest)                (((dest) << 
MSI_ADDR_DEST_ID_SHIFT) & MSI_ADDR_DEST_ID_MASK)
+#define         MSI_ADDR_EXT_DEST_ID_MASK      0x0000fe0
 
 /* MAX fixed pages reserved for mapping MSIX tables. */
 #define FIX_MSIX_MAX_PAGES              512
diff --git a/xen/arch/x86/traps.c b/xen/arch/x86/traps.c
index 485bd66971..3d2d75978c 100644
--- a/xen/arch/x86/traps.c
+++ b/xen/arch/x86/traps.c
@@ -1150,6 +1150,9 @@ void cpuid_hypervisor_leaves(const struct vcpu *v, 
uint32_t leaf,
         res->a |= XEN_HVM_CPUID_DOMID_PRESENT;
         res->c = d->domain_id;
 
+        if ( has_vpci(d) )
+            res->a |= XEN_HVM_CPUID_EXT_DEST_ID;
+
         break;
 
     case 5: /* PV-specific parameters */
diff --git a/xen/drivers/passthrough/x86/hvm.c 
b/xen/drivers/passthrough/x86/hvm.c
index 351daafdc9..666c4b7757 100644
--- a/xen/drivers/passthrough/x86/hvm.c
+++ b/xen/drivers/passthrough/x86/hvm.c
@@ -269,7 +269,7 @@ int pt_irq_create_bind(
     {
     case PT_IRQ_TYPE_MSI:
     {
-        uint8_t dest, delivery_mode;
+        unsigned int dest, delivery_mode;
         bool dest_mode;
         int dest_vcpu_id;
         const struct vcpu *vcpu;
@@ -345,7 +345,9 @@ int pt_irq_create_bind(
         }
         /* Calculate dest_vcpu_id for MSI-type pirq migration. */
         dest = MASK_EXTR(pirq_dpci->gmsi.gflags,
-                         XEN_DOMCTL_VMSI_X86_DEST_ID_MASK);
+                         XEN_DOMCTL_VMSI_X86_DEST_ID_MASK) |
+               (MASK_EXTR(pirq_dpci->gmsi.gflags,
+                         XEN_DOMCTL_VMSI_X86_EXT_DEST_ID_MASK) << 8);
         dest_mode = pirq_dpci->gmsi.gflags & XEN_DOMCTL_VMSI_X86_DM_MASK;
         delivery_mode = MASK_EXTR(pirq_dpci->gmsi.gflags,
                                   XEN_DOMCTL_VMSI_X86_DELIV_MASK);
@@ -782,7 +784,9 @@ static int _hvm_dpci_msi_eoi(struct domain *d,
          (pirq_dpci->gmsi.gvec == vector) )
     {
         unsigned int dest = MASK_EXTR(pirq_dpci->gmsi.gflags,
-                                      XEN_DOMCTL_VMSI_X86_DEST_ID_MASK);
+                                      XEN_DOMCTL_VMSI_X86_DEST_ID_MASK) |
+                            (MASK_EXTR(pirq_dpci->gmsi.gflags,
+                                XEN_DOMCTL_VMSI_X86_EXT_DEST_ID_MASK) << 8);
         bool dest_mode = pirq_dpci->gmsi.gflags & XEN_DOMCTL_VMSI_X86_DM_MASK;
 
         if ( vlapic_match_dest(vcpu_vlapic(current), NULL, 0, dest,
diff --git a/xen/include/public/arch-x86/cpuid.h 
b/xen/include/public/arch-x86/cpuid.h
index ce46305bee..49bcc93b6b 100644
--- a/xen/include/public/arch-x86/cpuid.h
+++ b/xen/include/public/arch-x86/cpuid.h
@@ -102,6 +102,12 @@
 #define XEN_HVM_CPUID_IOMMU_MAPPINGS   (1u << 2)
 #define XEN_HVM_CPUID_VCPU_ID_PRESENT  (1u << 3) /* vcpu id is present in EBX 
*/
 #define XEN_HVM_CPUID_DOMID_PRESENT    (1u << 4) /* domid is present in ECX */
+/*
+ * Bits 55:49 from the IO-APIC RTE and bits 11:5 from the MSI address can be
+ * used to store high bits for the Destination ID. This expands the Destination
+ * ID field from 8 to 15 bits, allowing to target APIC IDs up 32768.
+ */
+#define XEN_HVM_CPUID_EXT_DEST_ID      (1u << 5)
 
 /*
  * Leaf 6 (0x40000x05)
diff --git a/xen/include/public/domctl.h b/xen/include/public/domctl.h
index b85e6170b0..17ac7ef82b 100644
--- a/xen/include/public/domctl.h
+++ b/xen/include/public/domctl.h
@@ -588,6 +588,7 @@ struct xen_domctl_bind_pt_irq {
 #define XEN_DOMCTL_VMSI_X86_DELIV_MASK   0x007000
 #define XEN_DOMCTL_VMSI_X86_TRIG_MASK    0x008000
 #define XEN_DOMCTL_VMSI_X86_UNMASKED     0x010000
+#define XEN_DOMCTL_VMSI_X86_EXT_DEST_ID_MASK 0xfe0000
 
             uint64_aligned_t gtable;
         } msi;
-- 
2.34.1




 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.