|
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [XEN][RESEND][PATCH 1/3] xen/arm/p2m: perform IPA-based TLBI when IPA is known
From: Haseeb Ashraf <haseeb.ashraf@xxxxxxxxxxx>
Running Xen as an L1 hypervisor on KVM exposes the high cost of
emulating VMALLS12E1IS, especially when it is issued repeatedly.
Two p2m-related paths are particularly affected:
(a) vCPU migration on a pCPU, where the flush is intended to avoid
stage-1 translations associated with one vCPU leaking into
another, even though the effective stage-2 translation regime is
shared at the domain p2m level.
(b) p2m updates where the affected IPA range is known, for example
when XENMEM_remove_from_physmap removes mappings one page at a
time.
Address both cases by narrowing the invalidation where possible.
For the vCPU migration case, replace the combined guest flush with a
stage-1-only invalidation. This preserves the required separation
between per-vCPU stage-1 translations while avoiding unnecessary
stage-2 invalidation.
For p2m updates where the affected IPA range is known, invalidate by
IPA range instead of always flushing the whole guest TLB. Since the
Arm64 TLBI-by-IPA sequence loops per page, cap the range at an
empirically chosen threshold and fall back to the full guest flush
above that point.
On Graviton4 running Xen-on-KVM, the cost of a full guest TLBI was
measured to be roughly comparable to tens of thousands of single-page
IPA invalidations, so a 256MB threshold is used as a practical cutoff.
On Arm32 this patch keeps the conservative full-flush behaviour for the
IPA-range helper. Follow-up patches will refine that path further.
Suggested-by: Julien Grall <julien@xxxxxxx>
Signed-off-by: Haseeb Ashraf <haseeb.ashraf@xxxxxxxxxxx>
Changes in v4:
- Reworked the Arm64 IPA-based TLBI sequence to make completion of the
broadcast TLBI explicit instead of relying on a stage-1 helper for
the final barriers/workaround.
- Reused __tlb_repeat_sync() for the repeat-TLBI workaround handling.
- Added a small robustness check in p2m_force_tlb_flush_range_sync().
- Refined comments and commit message wording.
Changes in v3:
- Updated IPA-based TLBI sequence to apply ARM64 repeat TLBI
workaround to only final TLBI and DSB of the sequence.
- Removed TLB_HELPER_IPA and instead directly used the TLBI
instruction where needed as that was the only instance where it is
being used.
- Removed flush_guest_tlb_range_ipa_local() as it was not being used.
- Updated comments as per feedback in v2 about holding lock before
p2m_load_vttbr.
- Updated references of ARM ARM to use newer version DDI 0487L.b
instead of older version DDI 0487A.e.
Changes in v2:
- This commit implements the baseline implementation to address the
problem at hand. Removed the FEAT_nTLBPA implementation from this
commit which will be implemented in following commit using CPU
capability.
- Moved ARM32 and ARM64 specific implementations of TLBIs to
architecture specific flushtlb.h.
- Added references of ARM ARM in code comments.
- Evaluated and added a threshold to select between IPA-based TLB
invalidation vs fallback to full stage TLB invalidation above
the threshold.
---
xen/arch/arm/include/asm/arm32/flushtlb.h | 53 +++++++++++
xen/arch/arm/include/asm/arm64/flushtlb.h | 62 +++++++++++++
xen/arch/arm/include/asm/mmu/p2m.h | 2 +
xen/arch/arm/mmu/p2m.c | 105 ++++++++++++++++------
4 files changed, 195 insertions(+), 27 deletions(-)
diff --git a/xen/arch/arm/include/asm/arm32/flushtlb.h
b/xen/arch/arm/include/asm/arm32/flushtlb.h
index 5483be08fb..1497f16ba5 100644
--- a/xen/arch/arm/include/asm/arm32/flushtlb.h
+++ b/xen/arch/arm/include/asm/arm32/flushtlb.h
@@ -45,6 +45,43 @@ TLB_HELPER(flush_xen_tlb_local, TLBIALLH, nsh)
#undef TLB_HELPER
+/*
+ * Flush TLB of local processor. Use when flush for only stage-1 is intended.
+ *
+ * The following function should be used where intention is to clear only
+ * stage-1 TLBs. This would be helpful in future in identifying which stage-1
+ * TLB flushes can be skipped such as in present of FEAT_nTLBPA.
+ */
+static inline void flush_guest_tlb_s1_local(void)
+{
+ /*
+ * Same instruction can invalidate both stage-1 and stage-2 TLBs depending
+ * upon the execution context.
+ *
+ * See ARMv8 (DDI 0487L.b): G5-11698 Table G5-23.
+ */
+ return flush_guest_tlb_local();
+}
+
+/*
+ * Flush TLB of inner-shareable processor domain. Use when flush for only
+ * stage-1 is intended.
+ *
+ * The following function should be used where intention is to clear only
+ * stage-1 TLBs. This would be helpful in future in identifying which stage-1
+ * TLB flushes can be skipped such as in present of FEAT_nTLBPA.
+ */
+static inline void flush_guest_tlb_s1(void)
+{
+ /*
+ * Same instruction can invalidate both stage-1 and stage-2 TLBs depending
+ * upon the execution context.
+ *
+ * See ARMv8 (DDI 0487L.b): G5-11698 Table G5-23.
+ */
+ return flush_guest_tlb();
+}
+
/* Flush TLB of local processor for address va. */
static inline void __flush_xen_tlb_one_local(vaddr_t va)
{
@@ -60,6 +97,22 @@ static inline void __flush_xen_tlb_one(vaddr_t va)
/* Only for ARM64_WORKAROUND_REPEAT_TLBI */
static inline void __tlb_repeat_sync(void) {}
+/*
+ * Flush a range of IPA's mappings from the TLB of all processors in the
+ * inner-shareable domain.
+ */
+static inline void flush_guest_tlb_range_ipa(paddr_t ipa,
+ unsigned long size)
+{
+ /*
+ * Following can invalidate both stage-1 and stage-2 TLBs depending upon
+ * the execution mode.
+ *
+ * See ARMv8 (DDI 0487L.b): G5-11698 Table G5-23.
+ */
+ flush_guest_tlb();
+}
+
#endif /* __ASM_ARM_ARM32_FLUSHTLB_H__ */
/*
* Local variables:
diff --git a/xen/arch/arm/include/asm/arm64/flushtlb.h
b/xen/arch/arm/include/asm/arm64/flushtlb.h
index 1606b26bf2..cbe2a0da57 100644
--- a/xen/arch/arm/include/asm/arm64/flushtlb.h
+++ b/xen/arch/arm/include/asm/arm64/flushtlb.h
@@ -1,6 +1,8 @@
#ifndef __ASM_ARM_ARM64_FLUSHTLB_H__
#define __ASM_ARM_ARM64_FLUSHTLB_H__
+#include <xen/sizes.h> /* For SZ_* macros. */
+
/*
* Every invalidation operation use the following patterns:
*
@@ -67,6 +69,12 @@ TLB_HELPER_LOCAL(flush_guest_tlb_local, vmalls12e1)
/* Flush innershareable TLBs, current VMID only */
TLB_HELPER(flush_guest_tlb, vmalls12e1is)
+/* Flush local TLBs, current VMID, stage-1 only */
+TLB_HELPER(flush_guest_tlb_s1_local, vmalle1, nsh)
+
+/* Flush innershareable TLBs, current VMID, stage-1 only */
+TLB_HELPER(flush_guest_tlb_s1, vmalle1is, ish)
+
/* Flush local TLBs, all VMIDs, non-hypervisor mode */
TLB_HELPER_LOCAL(flush_all_guests_tlb_local, alle1)
@@ -116,6 +124,60 @@ static inline void __tlb_repeat_sync(void)
: : : "memory");
}
+/*
+ * Complete a broadcast TLBI sequence.
+ *
+ * For all relevant errata it is only necessary to execute a single
+ * additional TLBI;DSB sequence after the invalidation sequence has completed.
+ */
+static inline void tlbi_complete_broadcast(void)
+{
+ __tlb_repeat_sync();
+ dsb(ish);
+ isb();
+}
+
+/*
+ * Flush a range of IPA's mappings from the TLB of all processors in the
+ * inner-shareable domain.
+ */
+static inline void flush_guest_tlb_range_ipa(paddr_t ipa, unsigned long size)
+{
+ paddr_t end;
+
+ /*
+ * If IPA range is too big (empirically found to be 256M), then fallback to
+ * full TLB flush.
+ */
+ if ( size > SZ_256M )
+ return flush_guest_tlb();
+
+ end = ipa + size;
+
+ /*
+ * See ARM ARM DDI 0487L.b D8.17.6.1 (Invalidating TLB entries from stage 2
+ * translations) for details of TLBI sequence.
+ */
+ dsb(ishst); /* Ensure prior page-table updates have completed */
+ while ( ipa < end )
+ {
+ /* Flush stage-2 TLBs for IPA address */
+ asm_inline volatile(
+ "tlbi ipas2e1is, %0"
+ : : "r" (ipa >> PAGE_SHIFT) : "memory");
+ ipa += PAGE_SIZE;
+ }
+
+ /*
+ * Invalidate stage-1 entries for the current VMID as well. Keep this
+ * separate from completion of the broadcast TLBI sequence so that the
+ * final barriers/workaround are explicit.
+ */
+ asm_inline volatile("tlbi vmalle1is" : : : "memory");
+
+ tlbi_complete_broadcast();
+}
+
#endif /* __ASM_ARM_ARM64_FLUSHTLB_H__ */
/*
* Local variables:
diff --git a/xen/arch/arm/include/asm/mmu/p2m.h
b/xen/arch/arm/include/asm/mmu/p2m.h
index 58496c0b09..8a16722b82 100644
--- a/xen/arch/arm/include/asm/mmu/p2m.h
+++ b/xen/arch/arm/include/asm/mmu/p2m.h
@@ -10,6 +10,8 @@ extern unsigned int p2m_root_level;
struct p2m_domain;
void p2m_force_tlb_flush_sync(struct p2m_domain *p2m);
+void p2m_force_tlb_flush_range_sync(struct p2m_domain *p2m, uint64_t start_ipa,
+ uint64_t page_count);
void p2m_tlb_flush_sync(struct p2m_domain *p2m);
void p2m_clear_root_pages(struct p2m_domain *p2m);
diff --git a/xen/arch/arm/mmu/p2m.c b/xen/arch/arm/mmu/p2m.c
index 51abf3504f..525afcc210 100644
--- a/xen/arch/arm/mmu/p2m.c
+++ b/xen/arch/arm/mmu/p2m.c
@@ -231,37 +231,33 @@ void p2m_restore_state(struct vcpu *n)
isb();
/*
- * Flush local TLB for the domain to prevent wrong TLB translation
- * when running multiple vCPU of the same domain on a single pCPU.
+ * This only needs to invalidate stage-1 TLB state. All vCPUs of the
+ * domain share the same p2m, and therefore the same effective stage-2
+ * translation regime.
*/
if ( *last_vcpu_ran != INVALID_VCPU_ID && *last_vcpu_ran != n->vcpu_id )
- flush_guest_tlb_local();
+ flush_guest_tlb_s1_local();
*last_vcpu_ran = n->vcpu_id;
}
/*
- * Force a synchronous P2M TLB flush.
+ * Loads VTTBR from given P2M.
*
* Must be called with the p2m lock held.
+ *
+ * This returns switched out VTTBR.
*/
-void p2m_force_tlb_flush_sync(struct p2m_domain *p2m)
+static uint64_t p2m_load_vttbr(struct p2m_domain *p2m, unsigned long *flags)
{
- unsigned long flags = 0;
uint64_t ovttbr;
- ASSERT(p2m_is_write_locked(p2m));
-
- /*
- * ARM only provides an instruction to flush TLBs for the current
- * VMID. So switch to the VTTBR of a given P2M if different.
- */
ovttbr = READ_SYSREG64(VTTBR_EL2);
if ( ovttbr != p2m->vttbr )
{
uint64_t vttbr;
- local_irq_save(flags);
+ local_irq_save(*flags);
/*
* ARM64_WORKAROUND_AT_SPECULATE: We need to stop AT to allocate
@@ -280,8 +276,14 @@ void p2m_force_tlb_flush_sync(struct p2m_domain *p2m)
isb();
}
- flush_guest_tlb();
+ return ovttbr;
+}
+/*
+ * Restores VTTBR which was switched out as a result of p2m_load_vttbr().
+ */
+static void p2m_restore_vttbr(uint64_t ovttbr, unsigned long flags)
+{
if ( ovttbr != READ_SYSREG64(VTTBR_EL2) )
{
WRITE_SYSREG64(ovttbr, VTTBR_EL2);
@@ -289,10 +291,66 @@ void p2m_force_tlb_flush_sync(struct p2m_domain *p2m)
isb();
local_irq_restore(flags);
}
+}
+
+/*
+ * Force a synchronous P2M TLB flush.
+ *
+ * Must be called with the p2m lock held.
+ */
+void p2m_force_tlb_flush_sync(struct p2m_domain *p2m)
+{
+ unsigned long flags = 0;
+ uint64_t ovttbr;
+
+ ASSERT(p2m_is_write_locked(p2m));
+
+ /*
+ * ARM only provides an instruction to flush TLBs for the current
+ * VMID. So switch to the VTTBR of a given P2M if different.
+ */
+ ovttbr = p2m_load_vttbr(p2m, &flags);
+
+ flush_guest_tlb();
+
+ p2m_restore_vttbr(ovttbr, flags);
p2m->need_flush = false;
}
+/*
+ * Force a synchronous P2M TLB flush on a range of addresses.
+ *
+ * Must be called with the p2m lock held.
+ */
+void p2m_force_tlb_flush_range_sync(struct p2m_domain *p2m, uint64_t start_ipa,
+ uint64_t page_count)
+{
+ unsigned long flags = 0;
+ uint64_t ovttbr;
+
+ ASSERT(p2m_is_write_locked(p2m));
+
+ if ( !page_count )
+ return;
+
+ /*
+ * ARM only provides an instruction to flush TLBs for the current
+ * VMID. So switch to the VTTBR of a given P2M if different.
+ */
+ ovttbr = p2m_load_vttbr(p2m, &flags);
+
+ /* Invalidate TLB entries by IPA range */
+ if ( unlikely(page_count > ULONG_MAX / PAGE_SIZE) )
+ flush_guest_tlb();
+ else
+ {
+ flush_guest_tlb_range_ipa(start_ipa, page_count * PAGE_SIZE);
+ }
+
+ p2m_restore_vttbr(ovttbr, flags);
+}
+
void p2m_tlb_flush_sync(struct p2m_domain *p2m)
{
if ( p2m->need_flush )
@@ -1034,7 +1092,8 @@ static int __p2m_set_entry(struct p2m_domain *p2m,
* For more details see (D4.7.1 in ARM DDI 0487A.j).
*/
p2m_remove_pte(entry, p2m->clean_pte);
- p2m_force_tlb_flush_sync(p2m);
+ p2m_force_tlb_flush_range_sync(p2m, gfn_x(sgfn) << PAGE_SHIFT,
+ 1UL << page_order);
p2m_write_pte(entry, split_pte, p2m->clean_pte);
@@ -1090,8 +1149,8 @@ static int __p2m_set_entry(struct p2m_domain *p2m,
p2m_remove_pte(entry, p2m->clean_pte);
if ( removing_mapping )
- /* Flush can be deferred if the entry is removed */
- p2m->need_flush |= !!lpae_is_valid(orig_pte);
+ p2m_force_tlb_flush_range_sync(p2m, gfn_x(sgfn) << PAGE_SHIFT,
+ 1UL << page_order);
else
{
lpae_t pte = mfn_to_p2m_entry(smfn, t, a);
@@ -1102,18 +1161,10 @@ static int __p2m_set_entry(struct p2m_domain *p2m,
/*
* It is necessary to flush the TLB before writing the new entry
* to keep coherency when the previous entry was valid.
- *
- * Although, it could be defered when only the permissions are
- * changed (e.g in case of memaccess).
*/
if ( lpae_is_valid(orig_pte) )
- {
- if ( likely(!p2m->mem_access_enabled) ||
- P2M_CLEAR_PERM(pte) != P2M_CLEAR_PERM(orig_pte) )
- p2m_force_tlb_flush_sync(p2m);
- else
- p2m->need_flush = true;
- }
+ p2m_force_tlb_flush_range_sync(p2m, gfn_x(sgfn) << PAGE_SHIFT,
+ 1UL << page_order);
else if ( !p2m_is_valid(orig_pte) ) /* new mapping */
p2m->stats.mappings[level]++;
--
2.43.0
|
 |
Lists.xenproject.org is hosted with RackSpace, monitoring our |