|
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [PATCH v7 30/50] x86/hvm: introduce per-vCPU mapcache for vCPU-PT domains
From: Roger Pau Monné <roger.pau@xxxxxxxxxx>
Add a struct mapcache to struct mapcache_vcpu for the per-vCPU
accounting, with MAPCACHE_PERVCPU_ENTRIES (64) slots. Its inuse and
garbage bitmaps are then an unsigned long each: declare them inline
in struct mapcache_vcpu, rather than mapping them in the per-domain
area as the domain-wide bitmaps are.
A per-vCPU mapcache is only ever used by its own vCPU, and so by one
pCPU at a time. We can therefore skip both the lock and the TLB-epoch
tracking the domain-wide mapcache needs; a wrap still flushes the
local TLB.
struct mapcache_vcpu grows from 136 to 176 bytes, into padding ahead of
a 64-byte-aligned member of struct arch_vcpu: struct vcpu stays at 2880
bytes (x86_64, debug build).
Unlike a domain-wide mapcache, whose one pool all the domain's vCPUs
draw on, a per-vCPU one cannot, by design, use more than its own
slots: MAPCACHE_PERVCPU_ENTRIES (64) is the most single-page
mappings such a vCPU can hold at once, interrupt handlers' included,
and one more hits the BUG_ON() in map_domain_page(). The PoD
zero-page sweep held up to 16 on top of its callers' until
41ddcfe808ec ("x86/PoD: map one page at a time in
p2m_pod_zero_check()"). A domain-wide
mapcache is sized at MAPCACHE_VCPU_ENTRIES (16) slots per vCPU, in
one shared pool; a per-vCPU one gets 64, for a wider margin. The
extra slots take no memory: the area is a whole L1 page whatever the
number of slots, and each bitmap is still one unsigned long. Wraps,
each of which flushes the local TLB, also come less often.
mapcache_current_vcpu()'s lazy-context-switch handling
(sync_local_execstate() plus a pgtable_vcpu re-read) now applies to
any vCPU with a mapcache: when the idle vCPU runs lazily on an HVM
vCPU's monitor table, the full switch happens before the idle context
uses a mapcache.
In release builds map_domain_page() still tries the directmap first,
for any vCPU; debug builds exercise the new mapcache.
For a vCPU-PT domain, the slots of map_domain_pages() move too: a
vCPU's MAPCACHE_MULTIPAGE_ENTRIES slots are just above the single-page
ones of its own mapcache, and its block of the domain-wide multipage
area is no longer set up.
Signed-off-by: Roger Pau Monné <roger.pau@xxxxxxxxxx>
Assisted-by: Claude Code:claude-fable-5, Claude Code:claude-opus-4-8, Claude
Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xxxxxxxxxxxxxx>
---
Changes in v7:
- Rebased over map_domain_pages(), which now comes earlier in the
series: a vCPU-PT domain's multipage slots are above its per-vCPU
mapcache, as that patch had them. struct vcpu's size measured
again.
Changes in v6:
- Message: "x86/PoD: map one page at a time in p2m_pod_zero_check()" is
in staging now, so cite it by its hash.
Changes in v5:
- Message: a domain-wide mapcache is one pool that all the domain's
vCPUs draw on, not per-vCPU slots another vCPU could borrow.
Changes in v4:
- d->arch.vcpu_pt is now d->arch.asi.vcpu_pt.
- mapcache_lock() / mapcache_unlock() are now mapcache_enter() /
mapcache_exit(). has_mapcache()'s comment no longer changes, and the
early return in mapcache_enter() has a one-line comment instead of a
sentence added to the block (Roger).
- mapcache_vcpu_init(): ents and nr become assignments here, ents now
depending on the kind of mapcache (Jan).
- A per-vCPU mapcache gets 64 slots (MAPCACHE_PERVCPU_ENTRIES) in
place of MAPCACHE_VCPU_ENTRIES (16), which still sizes the
domain-wide mapcache; its bitmaps, now full, are described as an
unsigned long each (Roger).
- Move the comment on why a per-vCPU mapcache needs only a local flush
from map_domain_page()'s TLB flush to mapcache_new_epoch()'s early
return, as one line (Roger).
Changes in v3:
- HVM only, and per-vCPU from the start: the HVM enablement of "x86:
Initialize the mapcache for PV, HVM, and idle domains" (from the
directmap removal series) is folded in, without the idle domain and
without a domain-wide HVM mapcache.
- Retitle (was: "x86/mm: introduce a per-vCPU mapcache when using ASI").
- Say which second unmaps the not-present check in unmap_domain_page()
catches: those of an entry the first unmap zapped.
- State the budget of single-page mappings a vCPU can hold at once.
- The mapcache accounting refactor, the move of struct mapcache_vcpu
to struct arch_vcpu and the not-present check in unmap_domain_page()
are split out, for review, into the three patches before this one.
---
xen/arch/x86/domain_page.c | 74 +++++++++++++++++++++++--------
xen/arch/x86/include/asm/config.h | 7 ++-
xen/arch/x86/include/asm/domain.h | 6 +++
xen/arch/x86/mm.c | 8 ++--
4 files changed, 72 insertions(+), 23 deletions(-)
diff --git a/xen/arch/x86/domain_page.c b/xen/arch/x86/domain_page.c
index 1cce58a466..7bd6399ca6 100644
--- a/xen/arch/x86/domain_page.c
+++ b/xen/arch/x86/domain_page.c
@@ -22,7 +22,7 @@
/* Whether the vCPU supports mapcache usage, might not be in use. */
static bool has_mapcache(const struct vcpu *v)
{
- return is_pv_vcpu(v);
+ return is_pv_vcpu(v) || v->domain->arch.asi.vcpu_pt;
}
/*
@@ -71,14 +71,21 @@ static inline struct vcpu *mapcache_current_vcpu(void)
}
/*
- * The accounting for v's mapcache: the domain's, with *dcache set so the
- * caller takes the domain-wide lock and TLB-epoch machinery into account.
+ * The accounting for v's mapcache: its own for a vCPU-PT domain, the domain's
+ * otherwise, in which case *dcache is set so the caller takes the domain-wide
+ * lock and TLB-epoch machinery into account.
*/
static struct mapcache *vcpu_mapcache(struct vcpu *v,
struct mapcache_domain **dcache)
{
struct domain *d = v->domain;
+ if ( d->arch.asi.vcpu_pt )
+ {
+ *dcache = NULL;
+ return &v->arch.mapcache.cache;
+ }
+
*dcache = &d->arch.pv.mapcache;
return &(*dcache)->cache;
}
@@ -86,6 +93,10 @@ static struct mapcache *vcpu_mapcache(struct vcpu *v,
static void mapcache_enter(struct mapcache_domain *dcache,
struct mapcache_vcpu *vcache)
{
+ /* Only its own vCPU uses a per-vCPU mapcache: no lock or epoch needed. */
+ if ( !dcache )
+ return;
+
/* The domain's vCPUs share the mapcache's slots. */
spin_lock(&dcache->lock);
@@ -105,6 +116,10 @@ static void mapcache_enter(struct mapcache_domain *dcache,
static void mapcache_new_epoch(struct mapcache_domain *dcache,
struct mapcache_vcpu *vcache)
{
+ /* Per-vCPU: no epoch, as page-table loads flush its non-global entries. */
+ if ( !dcache )
+ return;
+
ASSERT(spin_is_locked(&dcache->lock));
vcache->shadow_epoch = ++dcache->epoch;
@@ -113,7 +128,8 @@ static void mapcache_new_epoch(struct mapcache_domain
*dcache,
static void mapcache_exit(struct mapcache_domain *dcache)
{
- spin_unlock(&dcache->lock);
+ if ( dcache )
+ spin_unlock(&dcache->lock);
}
#define mapcache_l2_entry(e) ((e) >> PAGETABLE_ORDER)
@@ -285,11 +301,15 @@ void unmap_domain_page(const void *ptr)
}
/*
- * The mapcache index of @v's first map_domain_pages() slot: at its stride of
- * the multipage area.
+ * The mapcache index of @v's first map_domain_pages() slot: above the
+ * single-page slots of its own mapcache with per-vCPU page tables, otherwise
+ * at its stride of the domain-wide multipage area.
*/
static unsigned int multipage_first_slot(const struct vcpu *v)
{
+ if ( v->domain->arch.asi.vcpu_pt )
+ return MAPCACHE_PERVCPU_ENTRIES;
+
return PFN_DOWN(MAPCACHE_MULTIPAGE_START(v) - MAPCACHE_VIRT_START);
}
@@ -426,13 +446,12 @@ int mapcache_vcpu_init(struct vcpu *v)
struct mapcache_domain *dcache;
struct mapcache *cache;
unsigned long i;
- unsigned int ents = d->max_vcpus * MAPCACHE_VCPU_ENTRIES;
- unsigned int nr = PFN_UP(BITS_TO_LONGS(ents) * sizeof(long));
+ unsigned int ents, nr;
BUILD_BUG_ON(MAPCACHE_MULTIPAGE_ENTRIES > BITS_PER_LONG);
- /* The vCPU's stride of the multipage area. */
- if ( !is_idle_domain(d) )
+ /* Without per-vCPU page tables, the vCPU's stride of the multipage area.
*/
+ if ( !is_idle_domain(d) && !d->arch.asi.vcpu_pt )
{
int rc = create_perdomain_mapping(v, MAPCACHE_MULTIPAGE_START(v),
MAPCACHE_MULTIPAGE_ENTRIES,
@@ -446,22 +465,41 @@ int mapcache_vcpu_init(struct vcpu *v)
return 0;
cache = vcpu_mapcache(v, &dcache);
- if ( !cache->inuse )
+ if ( !dcache )
+ {
+ BUILD_BUG_ON(MAPCACHE_PERVCPU_ENTRIES + MAPCACHE_MULTIPAGE_ENTRIES >
+ MAPCACHE_ENTRIES);
+
+ /* Per-vCPU bitmaps are an unsigned long each, in the vCPU itself. */
+ cache->inuse = v->arch.mapcache.inuse;
+ cache->garbage = v->arch.mapcache.garbage;
+ ents = MAPCACHE_PERVCPU_ENTRIES;
+ }
+ else if ( !cache->inuse )
return 0;
+ else
+ ents = d->max_vcpus * MAPCACHE_VCPU_ENTRIES;
+
+ nr = PFN_UP(BITS_TO_LONGS(ents) * sizeof(long));
if ( ents > cache->entries )
{
- /* Populate page tables. */
- int rc = create_perdomain_mapping(v, MAPCACHE_VIRT_START, ents,
+ /* Populate page tables, a per-vCPU cache's multipage slots included.
*/
+ int rc = create_perdomain_mapping(v, MAPCACHE_VIRT_START,
+ ents + (dcache ? 0
+ :
MAPCACHE_MULTIPAGE_ENTRIES),
NIL(l1_pgentry_t *), NULL);
- /* Populate bit maps. */
- if ( !rc )
+ /* Populate the domain-wide bit maps (kept in the per-domain area). */
+ if ( !rc && dcache )
+ {
rc = create_perdomain_mapping(v, (unsigned long)cache->inuse,
nr, NULL, NIL(struct page_info *));
- if ( !rc )
- rc = create_perdomain_mapping(v, (unsigned long)cache->garbage,
- nr, NULL, NIL(struct page_info *));
+ if ( !rc )
+ rc = create_perdomain_mapping(v, (unsigned long)cache->garbage,
+ nr, NULL,
+ NIL(struct page_info *));
+ }
if ( rc )
return rc;
diff --git a/xen/arch/x86/include/asm/config.h
b/xen/arch/x86/include/asm/config.h
index cea4cd4662..5768b6efe3 100644
--- a/xen/arch/x86/include/asm/config.h
+++ b/xen/arch/x86/include/asm/config.h
@@ -279,7 +279,12 @@ extern unsigned long xen_phys_start;
/* map_domain_page() map cache. The second per-domain-mapping sub-area. */
#define MAPCACHE_VCPU_ENTRIES (CONFIG_PAGING_LEVELS * CONFIG_PAGING_LEVELS)
-/* Multipage slots of each vCPU, at its stride of the multipage area. */
+/* Single-page slots of a per-vCPU mapcache (vCPU-PT domains). */
+#define MAPCACHE_PERVCPU_ENTRIES 64
+/*
+ * Multipage slots of each vCPU: above the single-page ones of a per-vCPU
+ * mapcache, otherwise at the vCPU's stride of the multipage area.
+ */
#define MAPCACHE_MULTIPAGE_ORDER 3
#define MAPCACHE_MULTIPAGE_ENTRIES (1U << MAPCACHE_MULTIPAGE_ORDER)
#define MAPCACHE_ENTRIES (MAX_VIRT_CPUS * MAPCACHE_VCPU_ENTRIES)
diff --git a/xen/arch/x86/include/asm/domain.h
b/xen/arch/x86/include/asm/domain.h
index d2515cd01a..5084848730 100644
--- a/xen/arch/x86/include/asm/domain.h
+++ b/xen/arch/x86/include/asm/domain.h
@@ -77,6 +77,12 @@ struct mapcache_vcpu {
uint32_t idx;
uint32_t refcnt;
} hash[MAPHASH_ENTRIES];
+
+ /* Accounting of a per-vCPU mapcache (vCPU-PT domains). */
+ struct mapcache cache;
+ /* Backing for cache.{inuse,garbage}. */
+ DECLARE_BITMAP(inuse, MAPCACHE_PERVCPU_ENTRIES);
+ DECLARE_BITMAP(garbage, MAPCACHE_PERVCPU_ENTRIES);
};
struct mapcache_domain {
diff --git a/xen/arch/x86/mm.c b/xen/arch/x86/mm.c
index d1fc57cf82..8900cde2d0 100644
--- a/xen/arch/x86/mm.c
+++ b/xen/arch/x86/mm.c
@@ -6538,10 +6538,10 @@ void unmap_vcpu_state(const struct vcpu *v, unsigned
int idx, unsigned int nr)
* Drop a transient mapping of a page behind a vCPU's state window, and this
* CPU's translations of it: at its address, and at its alias in the
* per-domain mirror, which every domain but a 64-bit PV one has (flushing
- * that address there is harmless). The mapcache does not drop them with
- * the slot's last mapping: it defers the flush, and CPUs running other vCPUs
- * of its domain may have cached the slot too. A directmap address, where
- * the caller has no mapcache, needs nothing.
+ * that address there is harmless). Neither kind of mapcache drops them with
+ * the slot's last mapping: each defers the flush, and CPUs running other
+ * vCPUs of the domain may have cached a domain-wide one's slot too. A
+ * directmap address, where the caller has no mapcache, needs nothing.
*/
void unmap_vcpu_state_page(const void *va)
{
--
2.56.0
|
![]() |
Lists.xenproject.org is hosted with RackSpace, monitoring our |