[PATCH v4 33/42] xen/riscv: save and clear the old IMSIC VS-file on vCPU migration

[Oleksii Kurochko <oleksii.kurochko@xxxxxxxxx>]

When a vCPU migrates to another pCPU, its guest interrupt file has to
move as well: the eidelivery, eithreshold and eip/eie state of the old
IMSIC VS-file has to be carried over to the newly acquired one.

Once all interrupt producers have been retargeted to the new VS-file,
so that nothing can reach the old one anymore, read the state of the
old one and clear it at the same time. This has to be done on the pCPU
owning the old VS-file, as a guest interrupt file is only accessible
through vsiselect/vsireg with hstatus.VGEIN selecting it. After that
the old VS-file is no longer in use and its guest external interrupt
number can be released.

The synchronization with the APLIC has to be done by that very pCPU
too, right before, so do both from the new imsic_vsfile_local_drain(),
in a single IPI: the move is done by the vCPU on its way back to the
guest, which each IPI delays.

imsic_ctxt_switch_from() armed HGEIE for the old VS-file when the vCPU
was switched out, but once the file is left there is nothing for Xen to
observe in it. Hence clear its bit with the new
imsic_local_hgeie_clear(), from imsic_vsfile_local_read_clear(), which
already runs on the pCPU owning the file.

The state is saved into struct imsic_mrif. Its eix[] array follows the
layout of a memory-resident interrupt file (MRIF) as defined by the AIA
specification: pairs of 64-bit interrupt-pending and interrupt-enable
doublewords, each pair covering 64 interrupt identities. eithreshold and
eidelivery aren't part of an MRIF and are stored next to it as they
belong to the interrupt file state too. Using the MRIF layout allows the
same structure to hold the interrupt file of a vCPU which doesn't own a
hardware guest interrupt file, which is how the AIA specification
describes MRIFs being used by a hypervisor.

Here the structure is only a temporary copy of a guest interrupt file:
it is filled and consumed by Xen alone and its address is never given
to an IOMMU, so no MSI is ever recorded into it. Hence plain
(non-atomic) accesses are sufficient and it can live on the stack.

The saved state still has to be restored into the new VS-file, so keep
the BUG_ON("unimplemented") placeholder at the end of
imsic_vsfile_move() to avoid silently losing the guest's interrupt
state.

Signed-off-by: Oleksii Kurochko <oleksii.kurochko@xxxxxxxxx>
---
Changes in v4:
 - Drop imsic_switchcase_ret() and imsic_eix_swap(), following the
   removal of the switchcase expansion pyramid: imsic_eix_swap64() uses
   imsic_vs_csr_swap() directly.
 - Name imsic_eix_check() in the comment above imsic_eix_swap64() rather
   than saying "above".
 - Use imsic_vsfile_select() and imsic_vsfile_deselect().
 - Say "hardware guest interrupt file" in the commit message, to make
   clear which vCPUs an MRIF can serve.
 - Clear the HGEIE bit of the old interrupt file in
   imsic_vsfile_local_read_clear(), through the new
   imsic_local_hgeie_clear(), moved here from "xen/riscv: wake up a
   descheduled vCPU on a guest external interrupt".
 - Save and clear the old file in imsic_vsfile_move(), which now does
   the move for imsic_migrate_vcpu().
 - Synchronize with the APLIC and save and clear the old file in a
   single IPI, from the new imsic_vsfile_local_drain(), which replaces
   imsic_vsfile_read_clear(). imsic_vsfile_local_read_clear(), no longer
   an IPI callback itself, takes a typed pointer and drops cf_check.
---
Changes in v3:
 - Reword the commit message: explain why the MRIF layout is used and
   why the copy isn't an MSI target, drop references to later patches.
 - Compare vsfile_cpu against CPU_NONE instead of NR_CPUS.
 - Introduce imsic_switchcase_ret() here, together with its first user,
   drop its unused variadic form and leave the trailing semicolon to
   its users.
 - Introduce the mrif member of struct imsic_vsfile_data here, together
   with its first user.
 - Initialise vsfile_data.mrif in imsic_migrate_vcpu() with a compound
   literal instead of adding a separate tmrif local variable, so the MRIF
   the old interrupt file is dumped into is visibly the one later
   restored from.
 - Drop the redundant parentheses around the arguments of
   imsic_vs_csr_swap() and its unnecessary local variable.
 - Use plain 64 instead of BITS_PER_TYPE(uint64_t) for the number of
   interrupt identities covered by an EIx group, and explain in the
   comment ahead of IMSIC_MAX_EIX where it comes from (the MRIF layout
   from the AIA spec).
 - Keep each EIx group of the MRIF as a uint64_t and introduce
   imsic_eix_swap64() to swap a whole group, which lets the
   #ifdef CONFIG_RISCV_32 in imsic_vsfile_local_read_clear() go.
 - Add missing break after ASSERT_UNREACHABLE() in the default case of
   imsic_eix_swap().
 - Use MASK_INSR() to set HSTATUS.VGEIN in
   imsic_vsfile_local_read_clear().
 - Explain in imsic_vsfile_local_read_clear() why hstatus and vsiselect
   are saved and restored.
 - Reword the comment on why no atomic accessors are needed to store
   the values into the MRIF.
 - Take the number of EIx groups from vsfile_data.nr_eix, as the
   nr_hw_eix local variable is gone from imsic_migrate_vcpu().
 - Move the vgein_release() stub to "xen/riscv: remap interrupts to new
   IMSIC VS-file", where it gains its first user.
---
Changes in v2:
 - New patch.
---
---
 xen/arch/riscv/imsic.c | 118 +++++++++++++++++++++++++++++++++++++----
 1 file changed, 107 insertions(+), 11 deletions(-)

diff --git a/xen/arch/riscv/imsic.c b/xen/arch/riscv/imsic.c
index 690a83fd6866..c705b287c7ff 100644
--- a/xen/arch/riscv/imsic.c
+++ b/xen/arch/riscv/imsic.c
@@ -58,6 +58,28 @@ static unsigned int __ro_after_init guest_num_msis;
  */
 #define GUEST_IMSIC_MAX_MSIS 255U
 
+/*
+ * The interrupt identities an IMSIC interrupt file provides are 0 (which is
+ * never valid, but still occupies a bit) up to IMSIC_MAX_ID inclusive, so
+ * IMSIC_MAX_ID + 1 bits have to be covered.
+ *
+ * Each EIx group covers 64 identities, matching the memory-resident interrupt
+ * file (MRIF) layout from the AIA spec, where pending and enable bits are
+ * stored as pairs of 64-bit doublewords.
+ */
+#define IMSIC_MAX_EIX DIV_ROUND_UP(IMSIC_MAX_ID + 1, 64)
+
+struct imsic_mrif_eix {
+    uint64_t eip;
+    uint64_t eie;
+};
+
+struct imsic_mrif {
+    struct imsic_mrif_eix eix[IMSIC_MAX_EIX];
+    unsigned long eithreshold;
+    unsigned long eidelivery;
+};
+
 #define IMSIC_DISABLE_EIDELIVERY    0
 #define IMSIC_ENABLE_EIDELIVERY     1
 #define IMSIC_DISABLE_EITHRESHOLD   1
@@ -87,6 +109,12 @@ do {                            \
     csr_clear(CSR_SIREG, v);    \
 } while (0)
 
+#define imsic_vs_csr_swap(c, v)     \
+({                                  \
+    csr_write(CSR_VSISELECT, c);    \
+    csr_swap(CSR_VSIREG, v);        \
+})
+
 #define imsic_vs_csr_write(c, v)    \
 do {                                \
     csr_write(CSR_VSISELECT, c);    \
@@ -115,6 +143,19 @@ static void imsic_eix_write64(unsigned int ireg, uint64_t 
val)
         imsic_vs_csr_write(ireg + 1, val >> 32);
 }
 
+/* Swap a whole 64-bit EIx group (see imsic_eix_check()). */
+static uint64_t imsic_eix_swap64(unsigned int ireg, uint64_t val)
+{
+    uint64_t old;
+
+    imsic_eix_check(ireg);
+    old = imsic_vs_csr_swap(ireg, val);
+    if ( IS_ENABLED(CONFIG_RISCV_32) )
+        old |= (uint64_t)imsic_vs_csr_swap(ireg + 1, val >> 32) << 32;
+
+    return old;
+}
+
 unsigned int vcpu_guest_file_id(const struct vcpu *v)
 {
     return ACCESS_ONCE(v->arch.vimsic_state->guest_file_id);
@@ -574,6 +615,8 @@ struct imsic_vsfile_data {
     unsigned int hgei;
     /* Number of 64-bit EIx groups to cover, see imsic_cfg.nr_eix. */
     unsigned int nr_eix;
+    /* Where the interrupt file state is saved to or restored from. */
+    struct imsic_mrif *mrif;
 };
 
 /* The state of this hart imsic_vsfile_select() changes. */
@@ -692,6 +735,56 @@ static void imsic_vsfile_local_clear(const struct 
imsic_vsfile_data *idata)
     local_irq_restore(flags);
 }
 
+/*
+ * Stop observing an interrupt file which a vCPU leaves when it moves to
+ * another pCPU, by clearing the bit imsic_ctxt_switch_from() set.
+ */
+static void imsic_local_hgeie_clear(unsigned int hgei)
+{
+    csr_clear(CSR_HGEIE, BIT(hgei, UL));
+}
+
+static void
+imsic_vsfile_local_read_clear(const struct imsic_vsfile_data *idata)
+{
+    unsigned int i;
+    struct imsic_mrif *mrif = idata->mrif;
+    struct imsic_vsfile_ctx ctx;
+
+    imsic_local_hgeie_clear(idata->hgei);
+
+    imsic_vsfile_select(idata->hgei, &ctx);
+
+    /*
+     * No atomic accessors are needed to store the values into the MRIF here,
+     * as imsic_vsfile_move() always passes it a pointer to a temporary MRIF
+     * on the stack.
+     */
+
+    mrif->eidelivery = imsic_vs_csr_swap(IMSIC_EIDELIVERY, 0);
+    mrif->eithreshold = imsic_vs_csr_swap(IMSIC_EITHRESHOLD, 0);
+    for ( i = 0; i < idata->nr_eix; i++ )
+    {
+        mrif->eix[i].eip = imsic_eix_swap64(IMSIC_EIP0 + i * 2, 0);
+        mrif->eix[i].eie = imsic_eix_swap64(IMSIC_EIE0 + i * 2, 0);
+    }
+
+    imsic_vsfile_deselect(&ctx);
+}
+
+/*
+ * Make sure that the MSIs already sent to the interrupt file @data names have
+ * arrived there, then save and clear the file. Both have to be done by the
+ * pCPU owning the file, and doing them together takes a single IPI.
+ */
+static void cf_check imsic_vsfile_local_drain(void *data)
+{
+    const struct imsic_vsfile_data *idata = data;
+
+    imsic_aplic_sync(data);
+    imsic_vsfile_local_read_clear(idata);
+}
+
 void cf_check vcpu_imsic_deinit(struct vcpu *v)
 {
     XVFREE(v->arch.vimsic_state);
@@ -1129,6 +1222,7 @@ static void imsic_vsfile_move(struct vcpu *v)
 {
     struct imsic_vsfile_data vsfile_data = {
         .nr_eix = imsic_cfg.nr_eix,
+        .mrif = &(struct imsic_mrif){ },
     };
     struct vimsic_state *imsic_state = v->arch.vimsic_state;
     unsigned long flags;
@@ -1161,18 +1255,20 @@ static void imsic_vsfile_move(struct vcpu *v)
     vaplic_reconfigure_target(v);
 
     /*
-     * Synchronizing interactions between a hart and the APLIC.
-     *
-     * The MSIs to be flushed are the ones still on their way to the old
-     * interrupt file, so the barrier has to be done by the pCPU which owns
-     * that file.
-     */
-    imsic_call_on_cpu(old_vsfile_cpu, imsic_aplic_sync, NULL);
-
-    /*
-     * At this point, all interrupt producers have been moved
-     * to the new IMSIC VS-file.
+     * Synchronize with the APLIC, so that no straggler MSI reaches the old
+     * interrupt file anymore, then read and clear the old file's register
+     * state into the MRIF. Both have to be done by the pCPU owning the old
+     * file.
      */
+    imsic_call_on_cpu(old_vsfile_cpu, imsic_vsfile_local_drain,
+                      &(struct imsic_vsfile_data){
+                          .hgei = old_vsfile_id,
+                          .nr_eix = vsfile_data.nr_eix,
+                          .mrif = vsfile_data.mrif,
+                      });
+
+    /* Free-up old IMSIC VS-file */
+    vgein_release(v, old_vsfile_id, old_vsfile_cpu);
 
     BUG_ON("unimplemented");
 }
-- 
2.56.1