|
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [PATCH v7 12/50] x86/domain_page: introduce map_domain_pages()
map_domain_page() maps a single page, so callers needing a virtually
contiguous mapping of several (not necessarily physically contiguous)
frames have had to fall back to vmap(), which maps whatever the frames
hold in every context for the lifetime of the mapping, and whose
vunmap() flushes every CPU's TLB, once per page. Provide a multipage
counterpart which maps them in the per-domain area of the calling
vCPU's domain instead, which only that domain's page tables carry:
map_domain_pages(mfn, nr) / unmap_domain_pages(ptr, nr)
Each vCPU has MAPCACHE_MULTIPAGE_ENTRIES slots of its own for such
mappings, a fixed block at the end of the mapcache sub-area. No
single-page mapcache is needed: the multipage slots are in the same VA
range whether or not the domain has one. The mappings are
order-aligned and tracked in a per-vCPU bitmap, and bypass the maphash
and the deferred garbage/epoch machinery.
At unmap a mapping's PTEs are cleared, and a local flush done. Since
each vCPU has its own slots, the local flush suffices for correctness,
although every vCPU's slots are in the domain's one L3: even if a stray
TLB entry is speculatively loaded on another processor, no other vCPU
will use that mapping, and it will be flushed before the vCPU which owns
the slots runs on that CPU.
With regard to exposure: the domain's other vCPUs may be able to reach
the mapping speculatively while it exists, and afterwards through a
stale TLB entry until their CPU next flushes. This doesn't expose
anything new, as the page is already mapped in the directmap anyway
(unless it's a PV domain and the page is past the short directmap), and
it is less than vmap() exposes: every context carries the vmap area.
An unmap will BUG() when given a pointer or a @nr which indexes
outside the vCPU's slots, unless that pointer is in the vmap area (see
below).
These slots are not available before the first page-table switch in
boot, nor with the idle vCPU's page tables loaded. Currently no calls
are made during boot, and the only known way to reach
map_domain_pages() with the idle page tables loaded is on an error path
that will already ultimately crash the host. Nonetheless, for
robustness, if called from a context where no mapping is available, it
will fall back to using vmap(); and unmap recognizes such an address
and vunmap()'s it. This briefly exposes contents to every context,
and requires a system-wide TLB shootdown on unmap, but allows
execution to continue. The alternative would be to BUG() in such a
context. The first fallback is logged.
The handling of a lazy context switch in mapcache_current_vcpu() moves
into pgtable_current_vcpu(), which map_domain_pages() uses whatever the
domain: an IPI can switch away page tables held lazily at any time.
Eight slots hold the largest mapping a known caller needs, a
three-page XSAVE area (four slots, order 2), with room for a two-page
mapping beside it. The multipage area takes 256MiB of address space
for MAX_VIRT_CPUS, and one L1 table per 64 vCPUs. The slot bitmap is a
field of struct arch_vcpu, as struct mapcache_vcpu exists for PV vCPUs
only; it sits in the struct's tail padding, which leaves its size
unchanged.
Later patches convert the two known in-tree consumers. Moving the
XSAVE areas off the xenheap needs the primitive too: the XSAVE area
exceeds one page on AMX-class hardware, and XSAVE/XRSTOR require it to
be virtually contiguous.
Assisted-by: Claude Code:claude-fable-5, Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xxxxxxxxxxxxxx>
---
Changes in v7:
- Standalone on staging, without the per-vCPU mapcache or the asi=
option: the multipage slots are each vCPU's block of the domain-wide
multipage area, the slot bitmap is a field of struct arch_vcpu (in its
tail padding), and there is no vmap-strict sub-option; the vmap()
fallback and its message stay.
Changes in v5:
- New in this version.
---
xen/arch/x86/domain_page.c | 166 +++++++++++++++++++++++++++---
xen/arch/x86/include/asm/config.h | 11 ++
xen/arch/x86/include/asm/domain.h | 3 +
xen/include/xen/domain_page.h | 21 ++++
4 files changed, 188 insertions(+), 13 deletions(-)
diff --git a/xen/arch/x86/domain_page.c b/xen/arch/x86/domain_page.c
index f02be1675b..ea759155da 100644
--- a/xen/arch/x86/domain_page.c
+++ b/xen/arch/x86/domain_page.c
@@ -6,6 +6,7 @@
* Copyright (c) 2003-2006, Keir Fraser <keir@xxxxxxxxxxxxx>
*/
+#include <xen/bitmap.h>
#include <xen/domain_page.h>
#include <xen/efi.h>
#include <xen/mm.h>
@@ -18,24 +19,22 @@
#include <asm/hardirq.h>
#include <asm/setup.h>
-static inline struct vcpu *mapcache_current_vcpu(void)
+/*
+ * The vCPU whose page tables are loaded here, and stay loaded until the
+ * caller is done with what it maps in them.
+ */
+static struct vcpu *pgtable_current_vcpu(void)
{
- const struct vcpu *v = this_cpu(pgtable_vcpu);
const struct vcpu *curr = current;
- /*
- * During early boot pgtable_vcpu is not set, callers must handle NULL.
- * Non-PV domains don't have a mapcache, the directmap covers all physical
- * address space.
- */
- if ( !v || !is_pv_vcpu(v) )
+ /* During early boot pgtable_vcpu is not set, callers must handle NULL. */
+ if ( !this_cpu(pgtable_vcpu) )
return NULL;
/*
- * If we are in a lazy context-switch state from a PV vCPU do a full switch
- * to the idle vCPU now, otherwise an incoming FLUSH_VCPU_STATE IPI would
- * change the page tables under our feet an invalidate any in-use mapcache
- * entries.
+ * If we are in a lazy context-switch state do a full switch to the idle
+ * vCPU now, otherwise an incoming FLUSH_VCPU_STATE IPI would change the
+ * page tables under our feet and invalidate any in-use mappings in them.
*/
if ( unlikely(this_cpu(curr_vcpu) != curr) )
{
@@ -54,6 +53,21 @@ static inline struct vcpu *mapcache_current_vcpu(void)
return ACCESS_ONCE(this_cpu(pgtable_vcpu));
}
+static inline struct vcpu *mapcache_current_vcpu(void)
+{
+ const struct vcpu *v = this_cpu(pgtable_vcpu);
+
+ /*
+ * During early boot pgtable_vcpu is not set, callers must handle NULL.
+ * Non-PV domains don't have a mapcache, the directmap covers all physical
+ * address space.
+ */
+ if ( !v || !is_pv_vcpu(v) )
+ return NULL;
+
+ return pgtable_current_vcpu();
+}
+
#define mapcache_l2_entry(e) ((e) >> PAGETABLE_ORDER)
#define MAPCACHE_L2_ENTRIES (mapcache_l2_entry(MAPCACHE_ENTRIES - 1) + 1)
#define MAPCACHE_L1ENT(idx) \
@@ -223,6 +237,119 @@ void unmap_domain_page(const void *ptr)
local_irq_restore(flags);
}
+/*
+ * The mapcache index of @v's first map_domain_pages() slot: at its stride of
+ * the multipage area.
+ */
+static unsigned int multipage_first_slot(const struct vcpu *v)
+{
+ return PFN_DOWN(MAPCACHE_MULTIPAGE_START(v) - MAPCACHE_VIRT_START);
+}
+
+/*
+ * Multipage mappings come from slots of the vCPU whose page tables are
+ * loaded, which no other vCPU uses, and bypass the maphash and the
+ * garbage/epoch machinery: at unmap a mapping's PTEs are zapped, and their
+ * translations dropped from this CPU's TLB, at their addresses and at their
+ * aliases in the per-domain mirror.
+ */
+void *map_domain_pages(const mfn_t *mfn, unsigned int nr)
+{
+ unsigned long flags;
+ unsigned int i, first;
+ struct vcpu *v;
+ int idx;
+
+ if ( !nr || nr > MAPCACHE_MULTIPAGE_ENTRIES )
+ {
+ ASSERT_UNREACHABLE();
+ return NULL;
+ }
+
+ if ( nr == 1 )
+ return map_domain_page(mfn[0]);
+
+ /* Early boot, or the idle vCPU's page tables: no multipage slots. */
+ v = pgtable_current_vcpu();
+ if ( !v || is_idle_vcpu(v) )
+ {
+ printk_once(XENLOG_WARNING
+ "%pS: map_domain_pages() falling back to vmap()\n",
+ __builtin_return_address(0));
+ return vmap(mfn, nr);
+ }
+
+ first = multipage_first_slot(v);
+
+ local_irq_save(flags);
+
+ idx = bitmap_find_free_region(&v->arch.multipage_inuse,
+ MAPCACHE_MULTIPAGE_ENTRIES,
+ get_order_from_pages(nr));
+ if ( idx >= 0 )
+ for ( i = 0; i < nr; i++ )
+ l1e_write(&MAPCACHE_L1ENT(first + idx + i),
+ l1e_from_mfn(mfn[i], __PAGE_HYPERVISOR_RW));
+
+ local_irq_restore(flags);
+
+ if ( idx < 0 )
+ return NULL;
+
+ return (void *)MAPCACHE_VIRT_START + pfn_to_paddr(first + idx);
+}
+
+void unmap_domain_pages(const void *ptr, unsigned int nr)
+{
+ unsigned long va = (unsigned long)ptr & PAGE_MASK;
+ unsigned long flags, slot;
+ unsigned int idx, first, i;
+ struct vcpu *v;
+
+ BUG_ON(!nr);
+
+ if ( nr == 1 )
+ {
+ unmap_domain_page(ptr);
+ return;
+ }
+
+ if ( va >= VMAP_VIRT_START && va < VMAP_VIRT_END )
+ {
+ vunmap((void *)va);
+ return;
+ }
+
+ v = pgtable_current_vcpu();
+ BUG_ON(!v || is_idle_vcpu(v) || va < MAPCACHE_VIRT_START);
+
+ first = multipage_first_slot(v);
+ slot = PFN_DOWN(va - MAPCACHE_VIRT_START);
+ BUG_ON(slot < first || slot - first >= MAPCACHE_MULTIPAGE_ENTRIES);
+
+ idx = slot - first;
+ BUG_ON(nr > MAPCACHE_MULTIPAGE_ENTRIES - idx);
+
+ local_irq_save(flags);
+
+ for ( i = 0; i < nr; i++ )
+ {
+ unsigned long addr = va + i * PAGE_SIZE;
+
+ ASSERT(l1e_get_flags(MAPCACHE_L1ENT(slot + i)) & _PAGE_PRESENT);
+ l1e_write(&MAPCACHE_L1ENT(slot + i), l1e_empty());
+ flush_tlb_one_local(addr);
+ if ( !is_pv_64bit_vcpu(v) )
+ flush_tlb_one_local(addr - PERDOMAIN_VIRT_START +
+ PERDOMAIN_ALT_VIRT_START);
+ }
+
+ bitmap_release_region(&v->arch.multipage_inuse, idx,
+ get_order_from_pages(nr));
+
+ local_irq_restore(flags);
+}
+
void mapcache_domain_init(struct domain *d)
{
struct mapcache_domain *dcache = &d->arch.pv.mapcache;
@@ -237,7 +364,7 @@ void mapcache_domain_init(struct domain *d)
BUILD_BUG_ON(MAPCACHE_VIRT_END + PAGE_SIZE * (3 +
2 * PFN_UP(BITS_TO_LONGS(MAPCACHE_ENTRIES) * sizeof(long))) >
- MAPCACHE_VIRT_START + (PERDOMAIN_SLOT_MBYTES << 20));
+ MAPCACHE_MULTIPAGE_VIRT_START);
bitmap_pages = PFN_UP(BITS_TO_LONGS(MAPCACHE_ENTRIES) * sizeof(long));
dcache->inuse = (void *)MAPCACHE_VIRT_END + PAGE_SIZE;
dcache->garbage = dcache->inuse +
@@ -254,6 +381,19 @@ int mapcache_vcpu_init(struct vcpu *v)
unsigned int ents = d->max_vcpus * MAPCACHE_VCPU_ENTRIES;
unsigned int nr = PFN_UP(BITS_TO_LONGS(ents) * sizeof(long));
+ BUILD_BUG_ON(MAPCACHE_MULTIPAGE_ENTRIES > BITS_PER_LONG);
+
+ /* The vCPU's stride of the multipage area. */
+ if ( !is_idle_domain(d) )
+ {
+ int rc = create_perdomain_mapping(d, MAPCACHE_MULTIPAGE_START(v),
+ MAPCACHE_MULTIPAGE_ENTRIES,
+ NIL(l1_pgentry_t *), NULL);
+
+ if ( rc )
+ return rc;
+ }
+
if ( !is_pv_vcpu(v) || !dcache->inuse )
return 0;
diff --git a/xen/arch/x86/include/asm/config.h
b/xen/arch/x86/include/asm/config.h
index b3c68308ad..8e9b856a6c 100644
--- a/xen/arch/x86/include/asm/config.h
+++ b/xen/arch/x86/include/asm/config.h
@@ -279,10 +279,21 @@ extern unsigned long xen_phys_start;
/* map_domain_page() map cache. The second per-domain-mapping sub-area. */
#define MAPCACHE_VCPU_ENTRIES (CONFIG_PAGING_LEVELS * CONFIG_PAGING_LEVELS)
+/* Multipage slots of each vCPU, at its stride of the multipage area. */
+#define MAPCACHE_MULTIPAGE_ORDER 3
+#define MAPCACHE_MULTIPAGE_ENTRIES (1U << MAPCACHE_MULTIPAGE_ORDER)
#define MAPCACHE_ENTRIES (MAX_VIRT_CPUS * MAPCACHE_VCPU_ENTRIES)
#define MAPCACHE_VIRT_START PERDOMAIN_VIRT_SLOT(1)
#define MAPCACHE_VIRT_END (MAPCACHE_VIRT_START + \
MAPCACHE_ENTRIES * PAGE_SIZE)
+/* The multipage area, at the end of the sub-area. */
+#define MAPCACHE_MULTIPAGE_VA_SHIFT (MAPCACHE_MULTIPAGE_ORDER + PAGE_SHIFT)
+#define MAPCACHE_MULTIPAGE_VIRT_START \
+ (MAPCACHE_VIRT_START + (PERDOMAIN_SLOT_MBYTES << 20) - \
+ (MAX_VIRT_CPUS << MAPCACHE_MULTIPAGE_VA_SHIFT))
+#define MAPCACHE_MULTIPAGE_START(v) \
+ (MAPCACHE_MULTIPAGE_VIRT_START + \
+ ((v)->vcpu_id << MAPCACHE_MULTIPAGE_VA_SHIFT))
/* Argument translation area. The third per-domain-mapping sub-area. */
#define ARG_XLAT_VIRT_START PERDOMAIN_VIRT_SLOT(2)
diff --git a/xen/arch/x86/include/asm/domain.h
b/xen/arch/x86/include/asm/domain.h
index 6ba59deeb5..3980d6228a 100644
--- a/xen/arch/x86/include/asm/domain.h
+++ b/xen/arch/x86/include/asm/domain.h
@@ -685,6 +685,9 @@ struct arch_vcpu
struct vcpu_msrs *msrs;
+ /* Which of the vCPU's map_domain_pages() slots are in use. */
+ unsigned long multipage_inuse;
+
struct {
bool next_interrupt_enabled;
} monitor;
diff --git a/xen/include/xen/domain_page.h b/xen/include/xen/domain_page.h
index c89b149e54..bf45a4cf55 100644
--- a/xen/include/xen/domain_page.h
+++ b/xen/include/xen/domain_page.h
@@ -31,6 +31,27 @@ void *map_domain_page(mfn_t mfn);
*/
void unmap_domain_page(const void *ptr);
+/*
+ * Map @nr page frames, not necessarily physically contiguous, at
+ * consecutive virtual addresses. Returns the virtual address of the
+ * first frame, or NULL when no space is available. As with
+ * map_domain_page(), the mapping is only valid within the current VCPU,
+ * and must be unmapped by it; unmap_domain_pages() takes the same @nr,
+ * and accepts a pointer carrying an in-page offset. Implemented on x86
+ * only at present.
+ *
+ * @nr has an implementation-defined upper bound (x86:
+ * MAPCACHE_MULTIPAGE_ENTRIES). Exceeding it is a caller bug rather than a
+ * transient failure, and fails the mapping; callers needing more pages
+ * than a per-VCPU window holds want vmap() directly.
+ *
+ * unmap_domain_pages() is fatal given a pointer or an @nr which indexes
+ * outside the current VCPU's multipage slots. A wrong @nr within them
+ * is not detected: callers must pass the @nr they mapped with.
+ */
+void *map_domain_pages(const mfn_t *mfn, unsigned int nr);
+void unmap_domain_pages(const void *ptr, unsigned int nr);
+
/*
* Given a VA from map_domain_page(), return its underlying MFN.
*/
--
2.56.0
|
![]() |
Lists.xenproject.org is hosted with RackSpace, monitoring our |