[PATCH v3 32/39] xen/riscv: save and clear the old IMSIC VS-file on vCPU migration

[Oleksii Kurochko <oleksii.kurochko@xxxxxxxxx>]

When a vCPU migrates to another pCPU, its guest interrupt file has to
move as well: the eidelivery, eithreshold and eip/eie state of the old
IMSIC VS-file has to be carried over to the newly acquired one.

Once all interrupt producers have been retargeted to the new VS-file,
so that nothing can reach the old one anymore, read the state of the
old one and clear it at the same time. This has to be done on the pCPU
owning the old VS-file, as a guest interrupt file is only accessible
through vsiselect/vsireg with hstatus.VGEIN selecting it. After that
the old VS-file is no longer in use and its guest external interrupt
number can be released.

The state is saved into struct imsic_mrif. Its eix[] array follows the
layout of a memory-resident interrupt file (MRIF) as defined by the AIA
specification: pairs of 64-bit interrupt-pending and interrupt-enable
doublewords, each pair covering 64 interrupt identities. eithreshold
and eidelivery aren't part of an MRIF and are stored next to it as they
belong to the interrupt file state too. Using the MRIF layout allows
the same structure to hold the interrupt file of a vCPU which doesn't
own a guest interrupt file, which is how the AIA specification
describes MRIFs being used by a hypervisor.

Here the structure is only a temporary copy of a guest interrupt file:
it is filled and consumed by Xen alone and its address is never given
to an IOMMU, so no MSI is ever recorded into it. Hence plain
(non-atomic) accesses are sufficient and it can live on the stack.

The saved state still has to be restored into the new VS-file, so keep
the BUG_ON("unimplemented") placeholder at the end of
imsic_migrate_vcpu() to avoid silently losing the guest's interrupt
state.

Signed-off-by: Oleksii Kurochko <oleksii.kurochko@xxxxxxxxx>
---
Changes in v3:
 - Reword the commit message: explain why the MRIF layout is used and
   why the copy isn't an MSI target, drop references to later patches.
 - Compare vsfile_cpu against CPU_NONE instead of NR_CPUS.
 - Introduce imsic_switchcase_ret() here, together with its first user,
   drop its unused variadic form and leave the trailing semicolon to
   its users.
 - Introduce the mrif member of struct imsic_vsfile_data here, together
   with its first user.
 - Initialise vsfile_data.mrif in imsic_migrate_vcpu() with a compound
   literal instead of adding a separate tmrif local variable, so the MRIF
   the old interrupt file is dumped into is visibly the one later
   restored from.
 - Drop the redundant parentheses around the arguments of
   imsic_vs_csr_swap() and its unnecessary local variable.
 - Use plain 64 instead of BITS_PER_TYPE(uint64_t) for the number of
   interrupt identities covered by an EIx group, and explain in the
   comment ahead of IMSIC_MAX_EIX where it comes from (the MRIF layout
   from the AIA spec).
 - Keep each EIx group of the MRIF as a uint64_t and introduce
   imsic_eix_swap64() to swap a whole group, which lets the
   #ifdef CONFIG_RISCV_32 in imsic_vsfile_local_read_clear() go.
 - Add missing break after ASSERT_UNREACHABLE() in the default case of
   imsic_eix_swap().
 - Use MASK_INSR() to set HSTATUS.VGEIN in
   imsic_vsfile_local_read_clear().
 - Explain in imsic_vsfile_local_read_clear() why hstatus and vsiselect
   are saved and restored.
 - Reword the comment on why no atomic accessors are needed to store
   the values into the MRIF.
 - Take the number of EIx groups from vsfile_data.nr_eix, as the
   nr_hw_eix local variable is gone from imsic_migrate_vcpu().
 - Move the vgein_release() stub to "xen/riscv: remap interrupts to new
   IMSIC VS-file", where it gains its first user.
---
Changes in v2:
 - New patch.
---
---
 xen/arch/riscv/imsic.c | 124 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 124 insertions(+)

diff --git a/xen/arch/riscv/imsic.c b/xen/arch/riscv/imsic.c
index 47854264edf3..2c4838b6b228 100644
--- a/xen/arch/riscv/imsic.c
+++ b/xen/arch/riscv/imsic.c
@@ -57,6 +57,28 @@ static unsigned int __ro_after_init guest_num_msis;
  */
 #define GUEST_IMSIC_MAX_MSIS 255U
 
+/*
+ * The interrupt identities an IMSIC interrupt file provides are 0 (which is
+ * never valid, but still occupies a bit) up to IMSIC_MAX_ID inclusive, so
+ * IMSIC_MAX_ID + 1 bits have to be covered.
+ *
+ * Each EIx group covers 64 identities, matching the memory-resident interrupt
+ * file (MRIF) layout from the AIA spec, where pending and enable bits are
+ * stored as pairs of 64-bit doublewords.
+ */
+#define IMSIC_MAX_EIX DIV_ROUND_UP(IMSIC_MAX_ID + 1, 64)
+
+struct imsic_mrif_eix {
+    uint64_t eip;
+    uint64_t eie;
+};
+
+struct imsic_mrif {
+    struct imsic_mrif_eix eix[IMSIC_MAX_EIX];
+    unsigned long eithreshold;
+    unsigned long eidelivery;
+};
+
 #define IMSIC_DISABLE_EIDELIVERY    0
 #define IMSIC_ENABLE_EIDELIVERY     1
 #define IMSIC_DISABLE_EITHRESHOLD   1
@@ -86,6 +108,12 @@ do {                            \
     csr_clear(CSR_SIREG, v);    \
 } while (0)
 
+#define imsic_vs_csr_swap(c, v)     \
+({                                  \
+    csr_write(CSR_VSISELECT, c);    \
+    csr_swap(CSR_VSIREG, v);        \
+})
+
 #define imsic_vs_csr_write(c, v)    \
 do {                                \
     csr_write(CSR_VSISELECT, c);    \
@@ -95,12 +123,17 @@ do {                                \
 /*
  * Leaf macros for the switchcase expansion pyramid below:
  * imsic_switchcase_break(ireg, op, v) - emit "case ireg: op(ireg, v); break"
+ * imsic_switchcase_ret(ireg, op, v) - emit "case ireg: return op(ireg, v)"
  */
 #define imsic_switchcase_break(ireg, op, v) \
     case ireg:                              \
         op(ireg, v);                        \
         break
 
+#define imsic_switchcase_ret(ireg, op, v)   \
+    case ireg:                              \
+        return op(ireg, v)
+
 /*
  * Generic switchcase expansion pyramid.
  * F is the per-operation leaf macro, ireg is the base register index.
@@ -126,6 +159,22 @@ do {                                \
     imsic_switchcase_32(F, (ireg) + 0, __VA_ARGS__); \
     imsic_switchcase_32(F, (ireg) + 32, __VA_ARGS__)
 
+static unsigned long imsic_eix_swap(unsigned int ireg, unsigned long val)
+{
+    switch ( ireg )
+    {
+    imsic_switchcase_64(imsic_switchcase_ret, IMSIC_EIP0,
+                        imsic_vs_csr_swap, val);
+    imsic_switchcase_64(imsic_switchcase_ret, IMSIC_EIE0,
+                        imsic_vs_csr_swap, val);
+    default:
+        ASSERT_UNREACHABLE();
+        break;
+    }
+
+    return 0;
+}
+
 static void imsic_eix_write(unsigned int ireg, unsigned long val)
 {
     switch ( ireg )
@@ -152,6 +201,17 @@ static void imsic_eix_write64(unsigned int ireg, uint64_t 
val)
         imsic_eix_write(ireg + 1, val >> 32);
 }
 
+/* Swap a whole 64-bit EIx group, laid out as described above. */
+static uint64_t imsic_eix_swap64(unsigned int ireg, uint64_t val)
+{
+    uint64_t old = imsic_eix_swap(ireg, val);
+
+    if ( IS_ENABLED(CONFIG_RISCV_32) )
+        old |= (uint64_t)imsic_eix_swap(ireg + 1, val >> 32) << 32;
+
+    return old;
+}
+
 unsigned int vcpu_guest_file_id(const struct vcpu *v)
 {
     return ACCESS_ONCE(v->arch.vimsic_state->guest_file_id);
@@ -514,6 +574,8 @@ struct imsic_vsfile_data {
     unsigned int hgei;
     /* Number of 64-bit EIx groups to cover, see imsic_cfg.nr_eix. */
     unsigned int nr_eix;
+    /* Where the interrupt file state is saved to or restored from. */
+    struct imsic_mrif *mrif;
 };
 
 /*
@@ -598,6 +660,60 @@ static void cf_check imsic_vsfile_local_clear(void *data)
     csr_write(CSR_VSISELECT, old_vsiselect);
 }
 
+static void cf_check imsic_vsfile_local_read_clear(void *data)
+{
+    unsigned int i;
+    const struct imsic_vsfile_data *idata = data;
+    struct imsic_mrif *mrif = idata->mrif;
+    unsigned long new_hstatus, old_hstatus, old_vsiselect;
+
+    /*
+     * The file is reached by retargeting hstatus.VGEIN, and
+     * imsic_vs_csr_swap() alters vsiselect. Both belong to the vCPU running
+     * on this pCPU, so restore them before returning.
+     */
+    old_vsiselect = csr_read(CSR_VSISELECT);
+    old_hstatus = csr_read(CSR_HSTATUS);
+    new_hstatus = old_hstatus & ~HSTATUS_VGEIN;
+    new_hstatus |= MASK_INSR(idata->hgei, HSTATUS_VGEIN);
+    csr_write(CSR_HSTATUS, new_hstatus);
+
+    /*
+     * No atomic accessors are needed to store the values into the MRIF here,
+     * as imsic_vsfile_read_clear() is always called with a pointer to a
+     * temporary MRIF on the stack.
+     */
+
+    mrif->eidelivery = imsic_vs_csr_swap(IMSIC_EIDELIVERY, 0);
+    mrif->eithreshold = imsic_vs_csr_swap(IMSIC_EITHRESHOLD, 0);
+    for ( i = 0; i < idata->nr_eix; i++ )
+    {
+        mrif->eix[i].eip = imsic_eix_swap64(IMSIC_EIP0 + i * 2, 0);
+        mrif->eix[i].eie = imsic_eix_swap64(IMSIC_EIE0 + i * 2, 0);
+    }
+
+    csr_write(CSR_HSTATUS, old_hstatus);
+    csr_write(CSR_VSISELECT, old_vsiselect);
+}
+
+static void imsic_vsfile_read_clear(unsigned int vsfile_id,
+                                    unsigned int vsfile_cpu,
+                                    unsigned int nr_eix,
+                                    struct imsic_mrif *mrif)
+{
+    struct imsic_vsfile_data idata = {
+        .hgei = vsfile_id,
+        .nr_eix = nr_eix,
+        .mrif = mrif,
+    };
+
+    /* We can only read clear if we have an IMSIC VS-file */
+    if ( vsfile_cpu == CPU_NONE || !vsfile_id )
+        return;
+
+    imsic_call_on_cpu(vsfile_cpu, imsic_vsfile_local_read_clear, &idata);
+}
+
 void cf_check vcpu_imsic_deinit(struct vcpu *v)
 {
     XVFREE(v->arch.vimsic_state);
@@ -960,6 +1076,7 @@ void imsic_migrate_vcpu(struct vcpu *v)
 {
     struct imsic_vsfile_data vsfile_data = {
         .nr_eix = imsic_cfg.nr_eix,
+        .mrif = &(struct imsic_mrif){ },
     };
     struct vimsic_state *imsic_state = v->arch.vimsic_state;
     unsigned long flags;
@@ -1028,5 +1145,12 @@ void imsic_migrate_vcpu(struct vcpu *v)
      * to the new IMSIC VS-file.
      */
 
+    /* Read and clear register state from old IMSIC VS-file */
+    imsic_vsfile_read_clear(old_vsfile_id, old_vsfile_cpu, vsfile_data.nr_eix,
+                            vsfile_data.mrif);
+
+    /* Free-up old IMSIC VS-file */
+    vgein_release(v, old_vsfile_id, old_vsfile_cpu);
+
     BUG_ON("unimplemented");
 }
-- 
2.55.0