|
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [PATCH v4 16/30] x86/mm: build and maintain a sparse view of the directmap
The directmap maps all of RAM, and every root page-table carries it, so
while any guest context runs, all guest memory is reachable from Xen's
address space. For ASI, guest contexts are to run on a view of the
directmap which maps only what Xen must reach through it at all times,
leaving guest memory to transient mappings (map_domain_page()).
Rather than remove the directmap for everyone, keep the full directmap
as it is, for the idle domain and for guest contexts not using ASI, and
build a second, sparse view: page-tables for the same virtual addresses
(the part of the directmap PV guests can see, where everything Xen
reaches through the directmap lives), mapping a subset of what the full
directmap maps, each entry a copy of the full directmap's. Boot and the
idle domain keep the full directmap, so nothing in them needs
converting.
The sparse view maps the xenheap, and memory never handed to the heap
allocator (Xen's image, boot-time allocations, firmware tables). It is
built once the boot allocator has handed its memory to the heap, from
the full directmap minus the ranges the heap was given, plus the xenheap
allocations made meanwhile. The page-tables then come from the heap,
and no other CPU is up. The sparse view has an L3 for each of its L4
slots from the start, so its L4 entries never change and root
page-tables can take copies of them. It is then kept in step:
- xenheap allocations are added, and taken out again before the pages
are freed. If that fails the pages stay mapped, and are leaked with
a warning rather than reused;
- memory handed to the heap later (boot modules, .init) is taken out,
and is not handed over if that fails;
- a PV dom0's initrd, assigned to dom0 in place, is taken out;
- page-tables from the boot allocator, which the sparse view maps with
every other boot-time allocation, are taken out when idle_pg_table's
hierarchy frees them, and are not freed if that fails, so that the
sparse view never keeps a page the heap may hand out;
- changes to the full directmap through map_pages_to_xen() and
modify_xen_mappings() are mirrored. After boot, map_pages_to_xen()
maps into the directmap only memory Xen keeps (stack guard and shadow
stack pages, firmware tables); memory hot-add maps the new range into
the full directmap only, so that the sparse view never needs
page-table pages for it.
modify_xen_mappings_lite() needs no change: it only changes the mappings
of Xen's image, in L4 slot 261, which every root page-table shares with
idle_pg_table, outside the range the sparse view replaces.
Common code gains four hooks for this, arch_heap_pages_init(),
arch_xenheap_needs_scrub() and arch_xenheap_pages_{alloc,free}(), no-ops
unless CONFIG_SPARSE_DIRECTMAP is selected. A failed
arch_xenheap_pages_alloc() may have mapped part of the pages:
alloc_xenheap_pages() then takes them out through
arch_xenheap_pages_free(), as a free would, before freeing them, and
leaks them if that fails too. xen_mapping_lookup(), built only with
CONFIG_SPARSE_DIRECTMAP, reads an entry of either hierarchy under the
L3 table lock.
alloc_xenheap_pages() accepts pages still awaiting scrubbing
(MEMF_no_scrub). That is harmless while the directmap maps all memory
anyway, but the sparse view would make such a page's contents, a dying
domain's data for instance, reachable from every context on it until
overwritten. With a sparse view configured, arch_xenheap_needs_scrub()
makes xenheap allocations scrub those pages first, as domain heap
allocations do; clean pages need nothing extra.
The allocator may still have to scrub a page before returning it; this
will be mapped with map_domain_page(). This is fine for HVM domains
on the sparse view, as the mapcache is per-vCPU, so no other vCPU can
read unscrubbed data. PV domains' mapcache is still per-domain, so
for PV domains on the sparse view, this opens up a window where other
vCPUs could potentially speculatively access the unscrubbed data.
This would be fixed by having per-vCPU mapcaches for PV domains, or
synchronously scrubbing pages on free. Neither are in the scope of the
current work.
arch_mfns_in_directmap() says whether an MFN range is reachable through
the directmap; it now answers for every context, so false once a sparse
view is configured. Its one caller, init_node_heap(), then keeps a NUMA
node's heap metadata in the xenheap instead of carving it out of the
node's memory through the directmap: carved out of a range handed to the
heap, it would be left out of the sparse view, which every heap
allocation needs it in.
The sparse view is built only when some domain type is to use it, which
nothing can request yet: this patch changes nothing on its own.
Assisted-by: Claude Code:claude-opus-5-5
Signed-off-by: George Dunlap <gwd@xxxxxxxxxxxxxx>
---
Changes in v4:
- Message: "domain type", not "guest type".
- Wrap a line that reached 80 columns.
- Memory hot-add maps the new range into the full directmap only.
- Define opt_sparse_dmap_hwdom here, beside the PV and HVM knobs
(Roger).
- add_to_sparse_view(): assert that the lookup finds the MFN expected
(Roger).
- cmp_mfn_range(): explicit return values, as in cmp_node() (Roger).
- Comments and variable name changes in sparse_dmap_init()'s loop over
the heap's ranges (Roger).
- Message: drop "(NUMA node heap metadata)" (Roger).
- Message: drop the reference to the directmap removal series (Roger).
- Message: modify_xen_mappings_lite() needs no change (Roger).
- Message: note the limit of scrubbing at allocation for PV domains
(Roger).
- Message: only boot and the idle domain keep the full directmap.
- Set the NX bit in this patch, rather than adding it in a later patch.
Changes in v3:
- New in this version.
---
xen/arch/x86/Kconfig | 1 +
xen/arch/x86/Makefile | 1 +
xen/arch/x86/include/asm/mm.h | 16 +-
xen/arch/x86/include/asm/sparse-directmap.h | 67 +++
xen/arch/x86/mm.c | 125 +++++-
xen/arch/x86/pv/dom0_build.c | 4 +
xen/arch/x86/setup.c | 4 +
xen/arch/x86/sparse-directmap.c | 431 ++++++++++++++++++++
xen/arch/x86/x86_64/mm.c | 10 +-
xen/common/Kconfig | 18 +
xen/common/page_alloc.c | 41 +-
xen/include/xen/mm.h | 35 ++
12 files changed, 737 insertions(+), 16 deletions(-)
create mode 100644 xen/arch/x86/include/asm/sparse-directmap.h
create mode 100644 xen/arch/x86/sparse-directmap.c
diff --git a/xen/arch/x86/Kconfig b/xen/arch/x86/Kconfig
index e5535ac484..cd72102ccc 100644
--- a/xen/arch/x86/Kconfig
+++ b/xen/arch/x86/Kconfig
@@ -31,6 +31,7 @@ config X86
select HAS_SCHED_GRANULARITY
select HAS_SHARED_INFO
imply HAS_SOFT_RESET
+ select HAS_SPARSE_DIRECTMAP
select HAS_UBSAN
select HAS_VMAP
select HAS_VPCI if HVM
diff --git a/xen/arch/x86/Makefile b/xen/arch/x86/Makefile
index 2be90ea098..86adf042e9 100644
--- a/xen/arch/x86/Makefile
+++ b/xen/arch/x86/Makefile
@@ -63,6 +63,7 @@ obj-y += shutdown.o
obj-y += smp.o
obj-y += smpboot.o
obj-y += spec_ctrl.o
+obj-$(CONFIG_SPARSE_DIRECTMAP) += sparse-directmap.o
obj-y += srat.o
obj-y += string.o
obj-$(CONFIG_SYSCTL) += sysctl.o
diff --git a/xen/arch/x86/include/asm/mm.h b/xen/arch/x86/include/asm/mm.h
index 12d1c06c12..7612cf66ef 100644
--- a/xen/arch/x86/include/asm/mm.h
+++ b/xen/arch/x86/include/asm/mm.h
@@ -7,6 +7,7 @@
#include <xen/rwlock.h>
#include <asm/io.h>
#include <asm/page.h>
+#include <asm/sparse-directmap.h>
#include <asm/uaccess.h>
/*
@@ -597,6 +598,15 @@ mfn_t alloc_xen_pagetable(void);
void free_xen_pagetable(mfn_t mfn);
void *alloc_mapped_pagetable(mfn_t *pmfn);
+int map_pages_in(l4_pgentry_t *root, unsigned long virt, mfn_t mfn,
+ unsigned long nr_mfns, pte_attr_t flags);
+int modify_mappings_in(l4_pgentry_t *root, unsigned long s, unsigned long e,
+ pte_attr_t nf);
+#ifdef CONFIG_SPARSE_DIRECTMAP
+mfn_t xen_mapping_lookup(l4_pgentry_t *root, unsigned long va,
+ pte_attr_t *flags, unsigned long *nr);
+#endif
+
int __sync_local_execstate(void);
/* Arch-specific portion of memory_op hypercall. */
@@ -657,12 +667,16 @@ void write_32bit_pse_identmap(uint32_t *l2);
/*
* x86 maps part of physical memory via the directmap region.
- * Return whether the range of MFN falls in the directmap region.
+ * Return whether the range of MFN falls in the directmap region in every
+ * context: never, with a sparse directmap view configured.
*/
static inline bool arch_mfns_in_directmap(unsigned long mfn, unsigned long nr)
{
unsigned long eva = min(DIRECTMAP_VIRT_END, HYPERVISOR_VIRT_END);
+ if ( sparse_dmap_configured() )
+ return false;
+
return (mfn + nr) <= (virt_to_mfn(eva - 1) + 1);
}
diff --git a/xen/arch/x86/include/asm/sparse-directmap.h
b/xen/arch/x86/include/asm/sparse-directmap.h
new file mode 100644
index 0000000000..ea14d44334
--- /dev/null
+++ b/xen/arch/x86/include/asm/sparse-directmap.h
@@ -0,0 +1,67 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+#ifndef X86_SPARSE_DIRECTMAP_H
+#define X86_SPARSE_DIRECTMAP_H
+
+#include <xen/mm-frame.h>
+#include <xen/mm-types.h>
+#include <asm/page.h>
+
+#ifdef CONFIG_SPARSE_DIRECTMAP
+
+/*
+ * The part of the directmap the sparse view covers: the part PV guests can
+ * see, which is where everything Xen reaches through the directmap lives.
+ * SPARSE_DMAP_END_MFN is the first MFN past it.
+ */
+#define SPARSE_DMAP_START DIRECTMAP_VIRT_START
+#define SPARSE_DMAP_END HYPERVISOR_VIRT_END
+#define SPARSE_DMAP_END_MFN (virt_to_mfn(SPARSE_DMAP_END - 1) + 1)
+
+/*
+ * Which domains use the sparse view: PV domains and HVM domains other than
+ * the hardware domain, and the hardware domain, PV or PVH.
+ */
+extern bool opt_sparse_dmap_pv, opt_sparse_dmap_hvm, opt_sparse_dmap_hwdom;
+
+/* Root of the sparse view, once built (NULL before, or if not configured). */
+extern l4_pgentry_t *sparse_dmap_root;
+
+static inline bool sparse_dmap_configured(void)
+{
+ return opt_sparse_dmap_pv || opt_sparse_dmap_hvm || opt_sparse_dmap_hwdom;
+}
+
+void sparse_dmap_init(void);
+int sparse_dmap_mirror_map(unsigned long virt, mfn_t mfn, unsigned long nr,
+ pte_attr_t flags);
+int sparse_dmap_mirror_modify(unsigned long s, unsigned long e,
+ pte_attr_t nf);
+int sparse_dmap_remove(mfn_t mfn, unsigned long nr);
+int sparse_dmap_drop_pagetable(mfn_t mfn);
+
+#else /* !CONFIG_SPARSE_DIRECTMAP */
+
+static inline bool sparse_dmap_configured(void) { return false; }
+static inline void sparse_dmap_init(void) {}
+static inline int sparse_dmap_mirror_map(unsigned long virt, mfn_t mfn,
+ unsigned long nr, pte_attr_t flags)
+{
+ return 0;
+}
+static inline int sparse_dmap_mirror_modify(unsigned long s, unsigned long e,
+ pte_attr_t nf)
+{
+ return 0;
+}
+static inline int sparse_dmap_remove(mfn_t mfn, unsigned long nr)
+{
+ return 0;
+}
+static inline int sparse_dmap_drop_pagetable(mfn_t mfn)
+{
+ return 0;
+}
+
+#endif /* CONFIG_SPARSE_DIRECTMAP */
+
+#endif /* X86_SPARSE_DIRECTMAP_H */
diff --git a/xen/arch/x86/mm.c b/xen/arch/x86/mm.c
index 514e121c8b..7e3009251c 100644
--- a/xen/arch/x86/mm.c
+++ b/xen/arch/x86/mm.c
@@ -127,6 +127,7 @@
#include <asm/setup.h>
#include <asm/shadow.h>
#include <asm/shared.h>
+#include <asm/sparse-directmap.h>
#include <asm/trampoline.h>
#include <asm/traps.h>
#include <asm/x86_emulate.h>
@@ -5422,11 +5423,101 @@ mfn_t xen_map_to_mfn(unsigned long va)
return ret;
}
+#ifdef CONFIG_SPARSE_DIRECTMAP
+/*
+ * Look @va up in the hierarchy rooted at @root, without allocating anything.
+ * Returns the MFN mapped at @va, or INVALID_MFN, with the mapping's flags in
+ * L1 form in *flags, and in *nr the number of pages from @va to the end of
+ * the entry that maps @va (or that finds it unmapped).
+ */
+mfn_t xen_mapping_lookup(l4_pgentry_t *root, unsigned long va,
+ pte_attr_t *flags, unsigned long *nr)
+{
+ bool locking = system_state > SYS_STATE_boot;
+ l4_pgentry_t l4e = root[l4_table_offset(va)];
+ const l3_pgentry_t *pl3e = NULL;
+ const l2_pgentry_t *pl2e = NULL;
+ const l1_pgentry_t *pl1e = NULL;
+ struct page_info *l3page;
+ unsigned int shift = L4_PAGETABLE_SHIFT;
+ mfn_t ret = INVALID_MFN;
+
+ L3T_INIT(l3page);
+ *flags = 0;
+
+ if ( !(l4e_get_flags(l4e) & _PAGE_PRESENT) )
+ goto out;
+
+ pl3e = map_l3t_from_l4e(l4e) + l3_table_offset(va);
+ l3page = l4e_get_page(l4e);
+ L3T_LOCK(l3page);
+
+ shift = L3_PAGETABLE_SHIFT;
+ if ( !(l3e_get_flags(*pl3e) & _PAGE_PRESENT) )
+ goto out;
+ if ( l3e_get_flags(*pl3e) & _PAGE_PSE )
+ {
+ /* The superpage PAT bit sits among the address bits. */
+ *flags = lNf_to_l1f(l3e_get_flags(*pl3e)) |
+ (l3e_get_intpte(*pl3e) & _PAGE_PSE_PAT ? _PAGE_PAT : 0);
+ ret = _mfn((l3e_get_pfn(*pl3e) &
+ ~((1UL << (2 * PAGETABLE_ORDER)) - 1)) +
+ (PFN_DOWN(va) & ((1UL << (2 * PAGETABLE_ORDER)) - 1)));
+ goto out;
+ }
+
+ pl2e = map_l2t_from_l3e(*pl3e) + l2_table_offset(va);
+ shift = L2_PAGETABLE_SHIFT;
+ if ( !(l2e_get_flags(*pl2e) & _PAGE_PRESENT) )
+ goto out;
+ if ( l2e_get_flags(*pl2e) & _PAGE_PSE )
+ {
+ *flags = lNf_to_l1f(l2e_get_flags(*pl2e)) |
+ (l2e_get_intpte(*pl2e) & _PAGE_PSE_PAT ? _PAGE_PAT : 0);
+ ret = _mfn((l2e_get_pfn(*pl2e) & ~((1UL << PAGETABLE_ORDER) - 1)) +
+ l1_table_offset(va));
+ goto out;
+ }
+
+ pl1e = map_l1t_from_l2e(*pl2e) + l1_table_offset(va);
+ shift = PAGE_SHIFT;
+ if ( l1e_get_flags(*pl1e) & _PAGE_PRESENT )
+ {
+ *flags = l1e_get_flags(*pl1e);
+ ret = l1e_get_mfn(*pl1e);
+ }
+
+ out:
+ L3T_UNLOCK(l3page);
+ unmap_domain_page(pl1e);
+ unmap_domain_page(pl2e);
+ unmap_domain_page(pl3e);
+ *nr = ((((va >> shift) + 1) << shift) - va) >> PAGE_SHIFT;
+ return ret;
+}
+#endif /* CONFIG_SPARSE_DIRECTMAP */
+
+/*
+ * Free a page-table page that the hierarchy rooted at @root has stopped
+ * using. One of idle_pg_table's may be in the sparse directmap view: it is
+ * taken out first, under the sparse view's L3 lock inside an L3 lock of
+ * idle_pg_table's hierarchy (the only order in which the two nest), and
+ * kept if that fails. The sparse view's own page-tables are not in it, and
+ * are freed directly: the caller may hold the sparse view's L3 lock.
+ */
+static void free_pagetable_in(const l4_pgentry_t *root, mfn_t mfn)
+{
+ if ( root == idle_pg_table && sparse_dmap_drop_pagetable(mfn) )
+ return;
+
+ free_xen_pagetable(mfn);
+}
+
/*
* map_pages_to_xen() in the hierarchy rooted at @root: idle_pg_table's, or
* another hierarchy sharing its virtual addresses.
*/
-static int map_pages_in(
+int map_pages_in(
l4_pgentry_t *root,
unsigned long virt,
mfn_t mfn,
@@ -5531,10 +5622,10 @@ static int map_pages_in(
ol2e = l2t[i];
if ( (l2e_get_flags(ol2e) & _PAGE_PRESENT) &&
!(l2e_get_flags(ol2e) & _PAGE_PSE) )
- free_xen_pagetable(l2e_get_mfn(ol2e));
+ free_pagetable_in(root, l2e_get_mfn(ol2e));
}
unmap_domain_page(l2t);
- free_xen_pagetable(l3e_get_mfn(ol3e));
+ free_pagetable_in(root, l3e_get_mfn(ol3e));
}
}
@@ -5632,7 +5723,7 @@ static int map_pages_in(
flush_flags(l1e_get_flags(l1t[i]));
flush_area(virt, flush_flags);
unmap_domain_page(l1t);
- free_xen_pagetable(l2e_get_mfn(ol2e));
+ free_pagetable_in(root, l2e_get_mfn(ol2e));
}
}
@@ -5764,7 +5855,7 @@ static int map_pages_in(
flush_area(virt - PAGE_SIZE,
FLUSH_TLB_GLOBAL |
FLUSH_ORDER(PAGETABLE_ORDER));
- free_xen_pagetable(l2e_get_mfn(ol2e));
+ free_pagetable_in(root, l2e_get_mfn(ol2e));
}
else if ( locking )
spin_unlock(&map_pgdir_lock);
@@ -5811,7 +5902,7 @@ static int map_pages_in(
flush_area(virt - PAGE_SIZE,
FLUSH_TLB_GLOBAL |
FLUSH_ORDER(2*PAGETABLE_ORDER));
- free_xen_pagetable(l3e_get_mfn(ol3e));
+ free_pagetable_in(root, l3e_get_mfn(ol3e));
}
else if ( locking )
spin_unlock(&map_pgdir_lock);
@@ -5838,7 +5929,12 @@ int map_pages_to_xen(
unsigned long nr_mfns,
pte_attr_t flags)
{
- return map_pages_in(idle_pg_table, virt, mfn, nr_mfns, flags);
+ int rc = map_pages_in(idle_pg_table, virt, mfn, nr_mfns, flags);
+
+ if ( !rc )
+ rc = sparse_dmap_mirror_map(virt, mfn, nr_mfns, flags);
+
+ return rc;
}
int __init populate_pt_range(unsigned long virt, unsigned long nr_mfns)
@@ -5861,8 +5957,8 @@ int __init populate_pt_range(unsigned long virt, unsigned
long nr_mfns)
* modify_mappings_in() does so in the hierarchy rooted at @root, as
* map_pages_in() does.
*/
-static int modify_mappings_in(l4_pgentry_t *root, unsigned long s,
- unsigned long e, pte_attr_t nf)
+int modify_mappings_in(l4_pgentry_t *root, unsigned long s,
+ unsigned long e, pte_attr_t nf)
{
bool locking = system_state > SYS_STATE_boot;
l3_pgentry_t *pl3e = NULL;
@@ -6072,7 +6168,7 @@ static int modify_mappings_in(l4_pgentry_t *root,
unsigned long s,
if ( locking )
spin_unlock(&map_pgdir_lock);
flush_area(NULL, FLUSH_TLB_GLOBAL); /* flush before free */
- free_xen_pagetable(l1mfn);
+ free_pagetable_in(root, l1mfn);
}
else if ( locking )
spin_unlock(&map_pgdir_lock);
@@ -6116,7 +6212,7 @@ static int modify_mappings_in(l4_pgentry_t *root,
unsigned long s,
if ( locking )
spin_unlock(&map_pgdir_lock);
flush_area(NULL, FLUSH_TLB_GLOBAL); /* flush before free */
- free_xen_pagetable(l2mfn);
+ free_pagetable_in(root, l2mfn);
}
else if ( locking )
spin_unlock(&map_pgdir_lock);
@@ -6142,7 +6238,12 @@ static int modify_mappings_in(l4_pgentry_t *root,
unsigned long s,
int modify_xen_mappings(unsigned long s, unsigned long e, pte_attr_t nf)
{
- return modify_mappings_in(idle_pg_table, s, e, nf);
+ int rc = modify_mappings_in(idle_pg_table, s, e, nf);
+
+ if ( !rc )
+ rc = sparse_dmap_mirror_modify(s, e, nf);
+
+ return rc;
}
int destroy_xen_mappings(unsigned long s, unsigned long e)
diff --git a/xen/arch/x86/pv/dom0_build.c b/xen/arch/x86/pv/dom0_build.c
index dcb1d407ec..f18959f3b4 100644
--- a/xen/arch/x86/pv/dom0_build.c
+++ b/xen/arch/x86/pv/dom0_build.c
@@ -22,6 +22,7 @@
#include <asm/pv/mm.h>
#include <asm/setup.h>
#include <asm/shadow.h>
+#include <asm/sparse-directmap.h>
/* Allow ring-3 access in long mode as guest cannot use ring 1 ... */
#define BASE_PROT (_PAGE_PRESENT|_PAGE_RW|_PAGE_ACCESSED|_PAGE_USER)
@@ -650,6 +651,9 @@ static int __init dom0_construct(const struct boot_domain
*bd)
{
if ( assign_pages(mfn_to_page(_mfn(initrd_mfn)), count, d, 0) )
BUG();
+ /* The initrd is dom0's now: out of the sparse directmap. */
+ if ( sparse_dmap_remove(_mfn(initrd_mfn), PFN_UP(initrd_len)) )
+ panic("Cannot unmap dom0's initrd\n");
/*
* We have mapped the initrd directly into dom0, and assigned the
* pages. Tell the boot_module handling that we've freed it, so the
diff --git a/xen/arch/x86/setup.c b/xen/arch/x86/setup.c
index fef3c2dd44..f95aa36e05 100644
--- a/xen/arch/x86/setup.c
+++ b/xen/arch/x86/setup.c
@@ -54,6 +54,7 @@
#include <asm/spec_ctrl.h>
#include <asm/stubs.h>
#include <asm/tboot.h>
+#include <asm/sparse-directmap.h>
#include <asm/trampoline.h>
#include <asm/traps.h>
@@ -1944,6 +1945,9 @@ void asmlinkage __init noreturn __start_xen(void)
system_state = SYS_STATE_boot;
+ /* The heap now has all memory: page-tables come from it from here on. */
+ sparse_dmap_init();
+
bsp_stack = cpu_alloc_stack(0); /* Needs to know IDT vs FRED */
if ( !bsp_stack )
panic("No memory for BSP stack\n");
diff --git a/xen/arch/x86/sparse-directmap.c b/xen/arch/x86/sparse-directmap.c
new file mode 100644
index 0000000000..d35a8d6979
--- /dev/null
+++ b/xen/arch/x86/sparse-directmap.c
@@ -0,0 +1,431 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * A sparse view of the directmap.
+ *
+ * Xen maps all of RAM at DIRECTMAP_VIRT_START, and every root page-table
+ * carries those mappings. The sparse view is a second set of page-tables
+ * for the same virtual addresses (the part of the directmap PV guests can
+ * see), mapping only what Xen needs to reach through the directmap at all
+ * times:
+ *
+ * - Memory never handed to the heap: Xen's image, boot-time allocations,
+ * firmware tables. Not a PV dom0's initrd, used in place, nor the frame
+ * table and M2P pages carved from hot-added memory.
+ * - Memory handed to the heap, then allocated to the xenheap.
+ *
+ * The rest of the heap's memory, where guest memory comes from, is not
+ * mapped. Every mapping in the sparse view is either identical to the
+ * full directmap's at the same address, or empty.
+ *
+ * Guest contexts selected on the command line run on root page-tables
+ * carrying the sparse view's L3s in the directmap slots.
+ */
+
+#include <xen/domain_page.h>
+#include <xen/init.h>
+#include <xen/lib.h>
+#include <xen/mm.h>
+#include <xen/numa.h>
+#include <xen/sort.h>
+
+#include <asm/page.h>
+#include <asm/sparse-directmap.h>
+
+bool __ro_after_init opt_sparse_dmap_pv;
+bool __ro_after_init opt_sparse_dmap_hvm;
+bool __ro_after_init opt_sparse_dmap_hwdom;
+
+/* Root of the sparse view: only ever walked and copied from, never loaded. */
+static l4_pgentry_t __aligned(PAGE_SIZE) sparse_l4[L4_PAGETABLE_ENTRIES];
+l4_pgentry_t *__ro_after_init sparse_dmap_root;
+
+/*
+ * Before the sparse view is built: the MFN ranges handed to the heap, which
+ * it leaves out, and the xenheap allocations made since, which it puts back.
+ */
+struct mfn_range {
+ unsigned long s, e;
+};
+static struct mfn_range early_heap[256];
+static unsigned int nr_early_heap;
+static struct mfn_range early_xenheap[2 * MAX_NUMNODES];
+static unsigned int nr_early_xenheap;
+static bool early_overflow;
+
+/*
+ * Clip [*mfn, *mfn + *nr) to what the sparse view covers; false if nothing is
+ * left.
+ */
+static bool clip_to_sparse_view(unsigned long *mfn, unsigned long *nr)
+{
+ unsigned long end = SPARSE_DMAP_END_MFN;
+
+ if ( *mfn >= end )
+ return false;
+ if ( *nr > end - *mfn )
+ *nr = end - *mfn;
+
+ return *nr;
+}
+
+static void early_range_add(struct mfn_range *r, unsigned int *nr,
+ unsigned int max, unsigned long s, unsigned long e)
+{
+ if ( *nr && r[*nr - 1].e == s )
+ r[*nr - 1].e = e;
+ else if ( *nr < max )
+ r[(*nr)++] = (struct mfn_range){ s, e };
+ else
+ early_overflow = true;
+}
+
+static void early_range_del(struct mfn_range *r, unsigned int *nr,
+ unsigned long s, unsigned long e)
+{
+ unsigned int i;
+
+ for ( i = 0; i < *nr; i++ )
+ if ( r[i].s == s && r[i].e == e )
+ {
+ r[i] = r[--*nr];
+ return;
+ }
+}
+
+/*
+ * Copy the full directmap's mappings of MFNs [mfn, mfn + nr) into the sparse
+ * view.
+ */
+static int add_to_sparse_view(l4_pgentry_t *root, unsigned long mfn,
+ unsigned long nr)
+{
+ unsigned long start = (unsigned long)mfn_to_virt(mfn);
+ unsigned long va = start, end = start + (nr << PAGE_SHIFT);
+
+ while ( va < end )
+ {
+ pte_attr_t flags;
+ unsigned long n;
+ mfn_t va_mfn = xen_mapping_lookup(idle_pg_table, va, &flags, &n);
+ int rc;
+
+ ASSERT(mfn_eq(va_mfn, INVALID_MFN) ||
+ mfn_x(va_mfn) == mfn + ((va - start) >> PAGE_SHIFT));
+ n = min(n, (end - va) >> PAGE_SHIFT);
+ if ( !mfn_eq(va_mfn, INVALID_MFN) &&
+ (rc = map_pages_in(root, va, va_mfn, n, flags)) )
+ return rc;
+ va += n << PAGE_SHIFT;
+ }
+
+ return 0;
+}
+
+static int remove_from_sparse_view(unsigned long mfn, unsigned long nr)
+{
+ unsigned long va = (unsigned long)mfn_to_virt(mfn);
+
+ return modify_mappings_in(sparse_dmap_root, va, va + (nr << PAGE_SHIFT),
+ _PAGE_NONE);
+}
+
+/* Memory is about to be handed to the heap allocator. */
+bool arch_heap_pages_init(mfn_t mfn, unsigned long nr)
+{
+ unsigned long s = mfn_x(mfn);
+
+ if ( !sparse_dmap_configured() || !clip_to_sparse_view(&s, &nr) )
+ return true;
+
+ if ( !sparse_dmap_root )
+ {
+ early_range_add(early_heap, &nr_early_heap, ARRAY_SIZE(early_heap),
+ s, s + nr);
+ return true;
+ }
+
+ if ( remove_from_sparse_view(s, nr) )
+ {
+ printk(XENLOG_ERR
+ "sparse directmap: cannot unmap MFNs %#lx-%#lx, not freeing\n",
+ s, s + nr - 1);
+ return false;
+ }
+
+ return true;
+}
+
+/*
+ * Xenheap pages enter the sparse view, which guest contexts run on: those
+ * still holding a previous owner's data must be scrubbed first.
+ */
+bool arch_xenheap_needs_scrub(void)
+{
+ return sparse_dmap_configured();
+}
+
+/* Pages have been allocated from the heap as xenheap pages. */
+int arch_xenheap_pages_alloc(mfn_t mfn, unsigned int order)
+{
+ unsigned long s = mfn_x(mfn), nr = 1UL << order;
+
+ if ( !sparse_dmap_configured() )
+ return 0;
+
+ /* Xenheap pages must be reachable through the sparse view. */
+ BUG_ON(s + nr > SPARSE_DMAP_END_MFN);
+
+ if ( !sparse_dmap_root )
+ {
+ early_range_add(early_xenheap, &nr_early_xenheap,
+ ARRAY_SIZE(early_xenheap), s, s + nr);
+ return 0;
+ }
+
+ return add_to_sparse_view(sparse_dmap_root, s, nr);
+}
+
+/*
+ * Xenheap pages are about to be freed. A failure means the pages may still
+ * be reachable through the sparse view, and must not be reused.
+ */
+int arch_xenheap_pages_free(mfn_t mfn, unsigned int order)
+{
+ unsigned long s = mfn_x(mfn), nr = 1UL << order;
+
+ if ( !sparse_dmap_configured() )
+ return 0;
+
+ BUG_ON(s + nr > SPARSE_DMAP_END_MFN);
+
+ if ( !sparse_dmap_root )
+ {
+ early_range_del(early_xenheap, &nr_early_xenheap, s, s + nr);
+ return 0;
+ }
+
+ return remove_from_sparse_view(s, nr);
+}
+
+/* Memory leaves Xen's hands without passing through the heap. */
+int sparse_dmap_remove(mfn_t mfn, unsigned long nr)
+{
+ return arch_heap_pages_init(mfn, nr) ? 0 : -ENOMEM;
+}
+
+/*
+ * A page-table page of idle_pg_table's hierarchy is about to be freed. One
+ * the boot allocator provided is in the sparse view, as every boot-time
+ * allocation is: take it out before the heap may hand it on. One allocated
+ * since came from the heap and is not in the sparse view, which a lookup
+ * finds without changing, or flushing, anything. Non-zero: the page may
+ * still be mapped, and must not be freed.
+ */
+int sparse_dmap_drop_pagetable(mfn_t mfn)
+{
+ unsigned long s = mfn_x(mfn), nr = 1, n;
+ pte_attr_t flags;
+ int rc;
+
+ if ( !sparse_dmap_root || !clip_to_sparse_view(&s, &nr) ||
+ mfn_eq(xen_mapping_lookup(sparse_dmap_root,
+ (unsigned long)mfn_to_virt(s), &flags, &n),
+ INVALID_MFN) )
+ return 0;
+
+ rc = remove_from_sparse_view(s, 1);
+ if ( rc )
+ printk(XENLOG_WARNING
+ "Leaking page-table page %"PRI_mfn": still in the sparse
directmap\n",
+ mfn_x(mfn));
+
+ return rc;
+}
+
+int sparse_dmap_mirror_map(unsigned long virt, mfn_t mfn, unsigned long nr,
+ pte_attr_t flags)
+{
+ if ( !sparse_dmap_root || virt < SPARSE_DMAP_START ||
+ virt >= SPARSE_DMAP_END )
+ return 0;
+
+ nr = min(nr, (SPARSE_DMAP_END - virt) >> PAGE_SHIFT);
+
+ return map_pages_in(sparse_dmap_root, virt, mfn, nr, flags);
+}
+
+int sparse_dmap_mirror_modify(unsigned long s, unsigned long e,
+ pte_attr_t nf)
+{
+ unsigned long va;
+
+ if ( !sparse_dmap_root || e <= SPARSE_DMAP_START ||
+ s >= SPARSE_DMAP_END )
+ return 0;
+
+ s = max(s, SPARSE_DMAP_START);
+ e = min(e, SPARSE_DMAP_END);
+
+ if ( !(nf & _PAGE_PRESENT) )
+ return modify_mappings_in(sparse_dmap_root, s, e, nf);
+
+ /*
+ * A permission change applies where the sparse view maps anything: it
+ * never creates mappings, and the sparse view need not map all of [s, e).
+ */
+ for ( va = s; va < e; )
+ {
+ pte_attr_t flags;
+ unsigned long n;
+ mfn_t mfn = xen_mapping_lookup(sparse_dmap_root, va, &flags, &n);
+ int rc;
+
+ n = min(n, (e - va) >> PAGE_SHIFT);
+ if ( !mfn_eq(mfn, INVALID_MFN) &&
+ (rc = modify_mappings_in(sparse_dmap_root, va,
+ va + (n << PAGE_SHIFT), nf)) )
+ return rc;
+ va += n << PAGE_SHIFT;
+ }
+
+ return 0;
+}
+
+static int __init cf_check cmp_mfn_range(const void *a, const void *b)
+{
+ const struct mfn_range *l = a, *r = b;
+
+ if ( l->s > r->s )
+ return 1;
+ if ( l->s < r->s )
+ return -1;
+
+ return 0;
+}
+
+static void __init cf_check swap_mfn_range(void *a, void *b)
+{
+ SWAP(*(struct mfn_range *)a, *(struct mfn_range *)b);
+}
+
+/*
+ * Build the sparse view: the full directmap, minus the ranges handed to the
+ * heap so far, plus the xenheap allocations made from them. Called once the
+ * heap has taken over from the boot allocator, so that the page-tables come
+ * from the heap, and before any other CPU is up.
+ */
+void __init sparse_dmap_init(void)
+{
+ unsigned long va, mapped = 0;
+ unsigned int i, heap_idx;
+
+ if ( !sparse_dmap_configured() )
+ return;
+
+ if ( early_overflow )
+ {
+ printk(XENLOG_WARNING
+ "Sparse directmap: too many early heap ranges, disabled\n");
+ opt_sparse_dmap_pv = opt_sparse_dmap_hvm = opt_sparse_dmap_hwdom =
+ false;
+ return;
+ }
+
+ /* An L3 for every slot, so that the L4 entries never change. */
+ for ( va = SPARSE_DMAP_START; va < SPARSE_DMAP_END;
+ va += 1UL << L4_PAGETABLE_SHIFT )
+ {
+ mfn_t l3mfn;
+ void *l3t = alloc_mapped_pagetable(&l3mfn);
+
+ if ( !l3t )
+ panic("Sparse directmap: out of memory\n");
+ unmap_domain_page(l3t);
+ sparse_l4[l4_table_offset(va)] =
+ l4e_from_mfn(l3mfn, __PAGE_HYPERVISOR);
+ }
+
+ sort(early_heap, nr_early_heap, sizeof(*early_heap), cmp_mfn_range,
+ swap_mfn_range);
+
+ /*
+ * The full directmap, leaf by leaf, minus the ranges added to the
+ * heap. Leaves come in address order, which is MFN order in the
+ * directmap, and the heap ranges are sorted and disjoint, so
+ * heap_idx only moves forward.
+ */
+ for ( va = SPARSE_DMAP_START, heap_idx = 0; va < SPARSE_DMAP_END; )
+ {
+ pte_attr_t flags;
+ unsigned long n, s, e;
+
+ /* Get information about the leaf PTE at va. */
+ mfn_t mfn = xen_mapping_lookup(idle_pg_table, va, &flags, &n);
+
+ /* Clip the va range we're considering to SPARSE_DMAP_END. */
+ n = min(n, (SPARSE_DMAP_END - va) >> PAGE_SHIFT);
+ if ( mfn_eq(mfn, INVALID_MFN) )
+ {
+ va += n << PAGE_SHIFT;
+ continue;
+ }
+
+ /*
+ * Walk through the range [mfn, mfn + n) looking for ranges not
+ * handed to the heap.
+ */
+ for ( s = mfn_x(mfn), e = s + n; s < e; )
+ {
+ unsigned long upto = e;
+
+ /* Step past heap ranges that end at or before s. */
+ while ( heap_idx < nr_early_heap && early_heap[heap_idx].e <= s )
+ heap_idx++;
+
+ /* s lies in a heap range: skip to its end. */
+ if ( heap_idx < nr_early_heap && early_heap[heap_idx].s <= s )
+ {
+ s = min(e, early_heap[heap_idx].e);
+ continue;
+ }
+
+ /* Map up to the next heap range, or to the end of the leaf. */
+ if ( heap_idx < nr_early_heap )
+ upto = min(e, early_heap[heap_idx].s);
+
+ if ( map_pages_in(sparse_l4, va + ((s - mfn_x(mfn)) << PAGE_SHIFT),
+ _mfn(s), upto - s, flags) )
+ panic("Sparse directmap: out of memory\n");
+ mapped += upto - s;
+ s = upto;
+ }
+
+ va += n << PAGE_SHIFT;
+ }
+
+ /*
+ * Now add back in ranges from the heap which were allocated to
+ * the xenheap.
+ */
+ for ( i = 0; i < nr_early_xenheap; i++ )
+ {
+ unsigned long nr = early_xenheap[i].e - early_xenheap[i].s;
+
+ if ( add_to_sparse_view(sparse_l4, early_xenheap[i].s, nr) )
+ panic("Sparse directmap: out of memory\n");
+ mapped += nr;
+ }
+
+ /*
+ * Nothing in the directmap is executed: its slots are NX, as the full
+ * directmap's are.
+ */
+ if ( cpu_has_nx )
+ for ( i = l4_table_offset(SPARSE_DMAP_START);
+ i < l4_table_offset(SPARSE_DMAP_END); i++ )
+ l4e_add_flags(sparse_l4[i], _PAGE_NX_BIT);
+
+ sparse_dmap_root = sparse_l4;
+
+ printk(XENLOG_INFO "Sparse directmap: %lu pages mapped\n", mapped);
+}
diff --git a/xen/arch/x86/x86_64/mm.c b/xen/arch/x86/x86_64/mm.c
index 7b04537066..c5d9dd64a3 100644
--- a/xen/arch/x86/x86_64/mm.c
+++ b/xen/arch/x86/x86_64/mm.c
@@ -1231,8 +1231,14 @@ int memory_add(unsigned long spfn, unsigned long epfn,
unsigned int pxm)
i = virt_to_mfn(HYPERVISOR_VIRT_END - 1) + 1;
if ( spfn < i )
{
- ret = map_pages_to_xen((unsigned long)mfn_to_virt(spfn), _mfn(spfn),
- min(epfn, i) - spfn, PAGE_HYPERVISOR);
+ /*
+ * Into the full directmap only, not a sparse directmap view: the
+ * frame table and M2P carved from the start of the range are
+ * reached through their own mappings, and the rest goes to the
+ * heap, which the sparse view leaves out.
+ */
+ ret = map_pages_in(idle_pg_table, (unsigned long)mfn_to_virt(spfn),
+ _mfn(spfn), min(epfn, i) - spfn, PAGE_HYPERVISOR);
if ( ret )
goto destroy_directmap;
}
diff --git a/xen/common/Kconfig b/xen/common/Kconfig
index 5b289e444f..8f2f605e45 100644
--- a/xen/common/Kconfig
+++ b/xen/common/Kconfig
@@ -167,6 +167,9 @@ config HAS_STATIC_MEMORY
config HAS_SOFT_RESET
bool
+config HAS_SPARSE_DIRECTMAP
+ bool
+
config HAS_STACK_PROTECTOR
bool
@@ -299,6 +302,21 @@ config SPECULATIVE_HARDEN_LOCK
This option is disabled by default at run time, and needs to be
enabled on the command line.
+
+config SPARSE_DIRECTMAP
+ bool "Sparse directmap view"
+ depends on HAS_SPARSE_DIRECTMAP
+ help
+ Build support for a sparse view of the directmap, alongside the
+ full one: it maps only the memory Xen needs to reach at all times
+ (the Xen heap, Xen's own image, firmware tables), while guest
+ memory is reached through transient mappings. Guest contexts
+ selected with the `asi=` command line option run on the sparse
+ directmap view, narrowing the memory reachable from Xen's address
+ space while they run, and hence the memory disclosure surface of
+ speculative side channels.
+
+ If unsure, say N.
endmenu
menu "Other hardening"
diff --git a/xen/common/page_alloc.c b/xen/common/page_alloc.c
index fe150eccd7..69e8e982b6 100644
--- a/xen/common/page_alloc.c
+++ b/xen/common/page_alloc.c
@@ -1971,6 +1971,9 @@ static void init_heap_pages(
unsigned long i;
bool need_scrub = scrub_debug;
+ if ( !arch_heap_pages_init(page_to_mfn(pg), nr_pages) )
+ return;
+
/*
* Keep MFN 0 away from the buddy allocator to avoid crossing zone
* boundary when merging two buddies.
@@ -2540,10 +2543,33 @@ void *alloc_xenheap_pages(unsigned int order, unsigned
int memflags)
if ( !(memflags >> _MEMF_bits) )
memflags |= MEMF_bits(xenheap_bits);
- pg = alloc_domheap_pages(NULL, order, memflags | MEMF_no_scrub);
+ /*
+ * Pages still awaiting scrubbing may do as xenheap pages: the directmap
+ * maps them anyway. Not so if an arch's sparse directmap view makes them
+ * reachable where the rest of the heap's memory is not.
+ */
+ if ( !arch_xenheap_needs_scrub() )
+ memflags |= MEMF_no_scrub;
+
+ pg = alloc_domheap_pages(NULL, order, memflags);
if ( unlikely(pg == NULL) )
return NULL;
+ if ( arch_xenheap_pages_alloc(page_to_mfn(pg), order) )
+ {
+ /*
+ * The hook may have mapped part of the pages before failing: take
+ * them out again as a free would, and leak them if that fails too.
+ */
+ if ( arch_xenheap_pages_free(page_to_mfn(pg), order) )
+ printk(XENLOG_WARNING
+ "Leaking xenheap pages %"PRI_mfn"+%u: still mapped\n",
+ mfn_x(page_to_mfn(pg)), 1U << order);
+ else
+ free_domheap_pages(pg, order);
+ return NULL;
+ }
+
for ( i = 0; i < (1u << order); i++ )
pg[i].count_info |= PGC_xen_heap;
@@ -2562,6 +2588,19 @@ void free_xenheap_pages(void *v, unsigned int order)
pg = virt_to_page(v);
+ /*
+ * If the pages cannot be taken out of an arch's sparse directmap view,
+ * they may remain reachable through it: leak them rather than hand them
+ * out.
+ */
+ if ( arch_xenheap_pages_free(page_to_mfn(pg), order) )
+ {
+ printk(XENLOG_WARNING
+ "Leaking xenheap pages %"PRI_mfn"+%u: still mapped\n",
+ mfn_x(page_to_mfn(pg)), 1U << order);
+ return;
+ }
+
for ( i = 0; i < (1u << order); i++ )
pg[i].count_info &= ~PGC_xen_heap;
diff --git a/xen/include/xen/mm.h b/xen/include/xen/mm.h
index 153c57f5dd..2bcc45cf58 100644
--- a/xen/include/xen/mm.h
+++ b/xen/include/xen/mm.h
@@ -94,6 +94,41 @@ bool scrub_free_pages(void);
#define alloc_xenheap_page() (alloc_xenheap_pages(0,0))
#define free_xenheap_page(v) (free_xenheap_pages(v,0))
+/*
+ * Hooks for an architecture keeping a second, sparse view of its directmap,
+ * which maps the xenheap but not the rest of the heap's memory.
+ */
+#ifdef CONFIG_SPARSE_DIRECTMAP
+/* Memory is about to be handed to the heap. False: don't, keep it out. */
+bool arch_heap_pages_init(mfn_t mfn, unsigned long nr);
+/* Xenheap pages are to be allocated. True: scrub those needing it first. */
+bool arch_xenheap_needs_scrub(void);
+/*
+ * Xenheap pages were allocated. Non-zero: failed, possibly part-way; the
+ * pages go through arch_xenheap_pages_free() before being freed again.
+ */
+int arch_xenheap_pages_alloc(mfn_t mfn, unsigned int order);
+/* Xenheap pages are to be freed. Non-zero: failed, don't reuse them. */
+int arch_xenheap_pages_free(mfn_t mfn, unsigned int order);
+#else
+static inline bool arch_heap_pages_init(mfn_t mfn, unsigned long nr)
+{
+ return true;
+}
+static inline bool arch_xenheap_needs_scrub(void)
+{
+ return false;
+}
+static inline int arch_xenheap_pages_alloc(mfn_t mfn, unsigned int order)
+{
+ return 0;
+}
+static inline int arch_xenheap_pages_free(mfn_t mfn, unsigned int order)
+{
+ return 0;
+}
+#endif
+
/* Free an allocation, and zero the pointer to it. */
#define FREE_XENHEAP_PAGES(p, o) do { \
void *_ptr_ = (p); \
--
2.55.0
|
 |
Lists.xenproject.org is hosted with RackSpace, monitoring our |