[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[PATCH] design: design doc for shared memory on a dom0less system


  • To: <xen-devel@xxxxxxxxxxxxxxxxxxxx>, <sstabellini@xxxxxxxxxx>, <julien@xxxxxxx>
  • From: Penny Zheng <penny.zheng@xxxxxxx>
  • Date: Wed, 26 Jan 2022 10:09:43 +0000
  • Arc-authentication-results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 40.67.248.234) smtp.rcpttodomain=lists.xenproject.org smtp.mailfrom=arm.com; dmarc=pass (p=none sp=none pct=100) action=none header.from=arm.com; dkim=none (message not signed); arc=none
  • Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=mKV00kB8E2lb3zvCouyq6z5byuuEBhVPh2iHKLoEKXw=; b=Vi8xFxZBiGJyY4XIarCXauzzfIpZLJzdX0O5o7HO5YVyvF/hdTNf2GZEXRbCTzl3VW1E9X22OTnAXCC3G7zUgUFQ06AmJCKT82JlOejORSnHU++S60mQ9nQtlNJx4RBTuweRQ9B7+hc38ONeGefH2UWygYP2wMTLYLQsucL7yVffcWCHNZ2rDNg6EyUgPuwCVddnEdSFZfNxsIP7F9d2Nc60Tj6FZmTaeyqGuExzv3MYtUvRUm5DCixiqH6ceJv9lAD2N9QJkmJ6+7eVUEYHsd5j0FATDINl670rKiNlEjz7hvhBDteP1OkGTcxNYEa8ATKyXveZ/V1443l+dTKKxQ==
  • Arc-seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=alNhyWf5wX8AUKH9KTfdg1EPCrHy37vJAHENqzaDR/md3bLsAzldTnYHgqviNZfQwBropo2xViibocBTRgxaIbhXXDMMU/u+V8zmB8y5ZuY+xSZGe4Kj6IQsn+K/HnX1AwXTDA59o/L0TDxZBQ14Eu3+VuFCSCOrOXuYOYZBppgSw343fZRkgq0h51ZhGEPJhT1WPJ1gQfUoCbQ6bYHltngEGL0jt7/eEYMsjE+JndxlHRM9L0C8MMpKnue5vPp00jKm4wuIN8dDxHPWU8lZJ3cx0x5XurMFqOsrSyw84H1BGohQHdio5gcR6e+ZCbIV0TIVDe6HKxT8GtSKWA3mSg==
  • Cc: <Bertrand.Marquis@xxxxxxx>, <Wei.Chen@xxxxxxx>
  • Delivery-date: Wed, 26 Jan 2022 10:10:27 +0000
  • List-id: Xen developer discussion <xen-devel.lists.xenproject.org>
  • Nodisclaimer: true

This commit provides a design doc for static shared memory
on a dom0less system.

Signed-off-by: Penny Zheng <penny.zheng@xxxxxxx>
---
 design/shm-dom0less.md | 182 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 182 insertions(+)
 create mode 100644 design/shm-dom0less.md

diff --git a/design/shm-dom0less.md b/design/shm-dom0less.md
new file mode 100644
index 0000000..b46199d
--- /dev/null
+++ b/design/shm-dom0less.md
@@ -0,0 +1,182 @@
+# Static Shared Memory between domains on a dom0less system
+
+This design aims to provide an overview of the new feature: setting up static
+shared memory between domains on a dom0less system, through device tree
+configuration.
+
+The new feature is driven by the need of finding a way to build up
+communication channels on dom0less system, since the legacy ways including
+grant table, etc are all absent there.
+
+It was inspired by the patch serie of "xl/libxl-based shared memory", see
+[1] for more details.
+
+# Static Shared Memory Device Tree Configuration
+
+The static shared memory device tree nodes allow users to statically set up
+shared memory among a group of dom0less DomUs and Dom0, enabling domains
+to do shm-based communication.
+
+- compatible
+
+    "xen,domain-shared-memory-v1"
+
+- xen,shm-id
+
+    An u32 value represents the unique identifier of the shared memory region.
+    User valuing per shared memory region shall follow the ascending order,
+    starting from xen,shm-id = <0x0>, to the maximum identifier
+    xen,shm-id = <0x126>. The special xen,shm-id = <0x127> is reserved for
+    INVALID_SHMID.
+
+- xen,shared-mem
+
+    An array takes a physical address, which is the base address of the
+    shared memory region in host physical address space, a size, and a guest
+    physical address, as the target address of the mapping.
+
+- role(Optional)
+
+    A string property specifying the ownership of a shared memory region,
+    the value must be one of the following: "owner", or "borrower"
+    A shared memory region could be explicitly backed by one domain, which is
+    called "owner domain", and all the other domains who are also sharing
+    this region are called "borrower domain".
+    If not specified, the default value is "borrower" and owner is
+    "dom_shared", a system domain.
+
+## Example
+
+chosen {
+    #address-cells = <0x1>;
+    #size-cells = <0x1>;
+    xen,xen-bootargs = "console=dtuart dtuart=serial0 bootscrub=0";
+
+    ......
+
+    /* this is for Dom0 */
+    dom0-shared-mem@10000000 {
+        compatible = "xen,domain-shared-memory-v1";
+        xen,shm-id = <0x0>;
+        role = "owner";
+        xen,shared-mem = <0x10000000 0x10000000 0x10000000>;
+    }
+
+    domU1 {
+        compatible = "xen,domain";
+        #address-cells = <0x1>;
+        #size-cells = <0x1>;
+        memory = <0 131072>;
+        cpus = <2>;
+        vpl011;
+
+        /*
+         * shared memory region identified as 0x0(xen,shm-id = <0x0>)
+         * shared between dom0.
+         */
+        domU1-shared-mem@10000000 {
+            compatible = "xen,domain-shared-memory-v1";
+            xen,shm-id = <0x0>;
+            role = "borrower";
+            xen,shared-mem = <0x10000000 0x10000000 0x50000000>;
+        }
+
+        domU1-shared-mem@50000000 {
+            compatible = "xen,domain-shared-memory-v1";
+            xen,shm-id = <0x1>;
+            xen,shared-mem = <0x50000000 0x20000000 0x60000000>;
+        }
+
+        ......
+
+    };
+
+    domU2 {
+        compatible = "xen,domain";
+        #address-cells = <0x1>;
+        #size-cells = <0x1>;
+        memory = <0 65536>;
+        cpus = <1>;
+
+        /*
+         * shared memory region identified as 0x1(xen,shm-id = <0x1>)
+         * shared between domU1.
+         */
+        domU2-shared-mem@50000000 {
+            compatible = "xen,domain-shared-memory-v1";
+            xen,shm-id = <0x1>;
+            xen,shared-mem = <0x50000000 0x20000000 0x70000000>;
+        }
+
+        ......
+    };
+};
+
+It is the example of two static shared memory regions.
+
+In terms of shared memory region identified as 0x0, host physical address
+starting at 0x10000000 of 256MB will be reserved to be shared between Dom0
+and DomU1. It will get mapped at 0x10000000 in Dom0 guest physical address
+space, and at 0x50000000 in DomU1 guest physical address space. Dom0 is the
+owner domain, and domU1 is the borrower domain.
+
+And in terms of shared memory region identified as 0x1, host physical address
+starting at 0x50000000 of 512MB will be reserved to be shared between DomU1
+and DomU2. It will get mapped at 0x60000000 in DomU1 guest physical address
+space, and at 0x70000000 in DomU2 guest physical address space. Since no owner
+domain is explicitly defined, the default "dom_shared" is the owner domain,
+and both domU1 and domU2 are the borrower domains.
+
+# Overview of Static Shared Memory Flow
+
+Static Shared Memory working flow could be classified into the following
+steps:
+ - Carve out a range of memory in host physical address space to be used
+for sharing. Define it in device tree configuration, then parse and reserve
+it to avoid other use.
+ - Create a special domain "dom_shared". It will be the owner domain which
+is owning the statically shared pages, if "role" property is not specified.
+ - Per shared memory region could be shared with multiple domains. For
+owner domain, it acquires statically shared pages and assign them to itself,
+in the same way with static memory. And other than owner domain, the others
+who are also sharing are called "borrower domain", for which foreign memory
+map of statically shared pages is required.
+ - Expose the shared memory to the domU using the "xen,shared-memory-v1"
+reserved-memory binding. See
+Documentation/devicetree/bindings/reserved-memory/xen,shared-memory.txt
+in Linux for the corresponding device tree binding.
+
+# Memory management of Shared Memory Region
+
+Each memory page needs to have an "owner" and it is likely that in many cases
+the user don't care who the owner is, so it makes sense that users don't
+need to specify the "role" in device tree if they don't want to, in which
+scenario, a default domain shall be the owner domain.
+
+We propose a new system domain "dom_shared" to be the default domain owning all
+statically unowned shared pages, assigning it dom_id 0x7FF5(DOMID_SHARED).
+
+"dom_shared" domain shall get constructed before domain construction and after
+"setup_virt_paging", during system boot-time, so it could successfully do
+p2m initialization.
+
+Owner domain acquires statically shared pages and assign them to itself,
+while borrower domains get and take reference of them, then do foreign memory
+map of these statically shared pages.
+
+When destroying or rebooting a domain, if it is a borrower domain, other than
+removing foreign memory map of statically shared pages in P2M table, we also
+need to drop according gained reference. And if it is an owner domain, since
+statically shared pages are allocated as guest normal ram, it is not needed to
+do extra removing.
+
+However if owner domain is not the default "dom_shared" domain, but specified
+explicitly in device tree, stopping itself will make shared memory region
+unaccessible to all borrower domains, so we need to remove foreign memory map
+for all borrower domains. Notice that all borrowers domains should be stopped
+before stopping the owner domain.
+
+"dom_shared" domain is destroyed when the whole system shuts down, so its
+owning statically shared pages are only freed at system shutdown.
+
+[1] https://marc.info/?l=xen-devel&m=154404821731186
-- 
2.25.1




 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.