Xen project Mailing List

On 02/02/16 17:11, Stefano Stabellini wrote: > Haozhong, thanks for your work! > > On Mon, 1 Feb 2016, Haozhong Zhang wrote: > > 3.2 Address Mapping > > > > 3.2.1 My Design > > > > The overview of this design is shown in the following figure. > > > > Dom0 | DomU > > | > > | > > QEMU | > > +...+--------------------+...+-----+ | > > VA | | Label Storage Area | | buf | | > > +...+--------------------+...+-----+ | > > ^ ^ ^ | > > | | | | > > V | | | > > +-------+ +-------+ mmap(2) | > > | vACPI | | v_DSM | | | | +----+------------+ > > +-------+ +-------+ | | | SPA | | /dev/pmem0 | > > ^ ^ +------+ | | +----+------------+ > > --------|-----------|-----|------------|-- | ^ ^ > > | | | | | | | > > | +------+ +------------~-----~-------------+ | > > | | | | | > > XEN_DOMCTL_memory_mapping > > | | | +-----~--------------------------+ > > | | | | | > > | | +----+------------+ | > > Linux | | SPA | | /dev/pmem0 | | +------+ +------+ > > | | +----+------------+ | | ACPI | | _DSM | > > | | ^ | +------+ +------+ > > | | | | | | > > | | Dom0 Driver | hvmloader/xl | > > > > --------|----|-------------------|---------------------|----------|--------------- > > | +-------------------~---------------------~----------+ > > Xen | | | > > +------------------------~---------------------+ > > > > ---------------------------------|------------------------------------------------ > > +----------------+ > > | > > +-------------+ > > HW | NVDIMM | > > +-------------+ > > > > > > This design treats host NVDIMM devices as ordinary MMIO devices: > > (1) Dom0 Linux NVDIMM driver is responsible to detect (through NFIT) > > and drive host NVDIMM devices (implementing block device > > interface). Namespaces and file systems on host NVDIMM devices > > are handled by Dom0 Linux as well. > > > > (2) QEMU mmap(2) the pmem NVDIMM devices (/dev/pmem0) into its > > virtual address space (buf). > > > > (3) QEMU gets the host physical address of buf, i.e. the host system > > physical address that is occupied by /dev/pmem0, and calls Xen > > hypercall XEN_DOMCTL_memory_mapping to map it to a DomU. > > How is this going to work from a security perspective? Is it going to > require running QEMU as root in Dom0, which will prevent NVDIMM from > working by default on Xen? If so, what's the plan? > Oh, I forgot to address the non-root qemu issues in this design ... The default user:group of /dev/pmem0 is root:disk, and its permission is rw-rw----. We could lift the others permission to rw, so that non-root QEMU can mmap /dev/pmem0. But it looks too risky. Or, we can make a file system on /dev/pmem0, create files on it, set the owner of those files to xen-qemuuser-domid$domid, and then pass those files to QEMU. In this way, non-root QEMU should be able to mmap those files. > > > > (ACPI part is described in Section 3.3 later) > > > > Above (1)(2) have already been done in current QEMU. Only (3) is > > needed to implement in QEMU. No change is needed in Xen for address > > mapping in this design. > > > > Open: It seems no system call/ioctl is provided by Linux kernel to > > get the physical address from a virtual address. > > /proc/<qemu_pid>/pagemap provides information of mapping from > > VA to PA. Is it an acceptable solution to let QEMU parse this > > file to get the physical address? > > Does it work in a non-root scenario? > Seemingly no, according to Documentation/vm/pagemap.txt in Linux kernel: | Since Linux 4.0 only users with the CAP_SYS_ADMIN capability can get PFNs. | In 4.0 and 4.1 opens by unprivileged fail with -EPERM. Starting from | 4.2 the PFN field is zeroed if the user does not have CAP_SYS_ADMIN. | Reason: information about PFNs helps in exploiting Rowhammer vulnerability. A possible alternative is to add a new hypercall similar to XEN_DOMCTL_memory_mapping but receiving virtual address as the address parameter and translating to machine address in the hypervisor. > > > Open: For a large pmem, mmap(2) is very possible to not map all SPA > > occupied by pmem at the beginning, i.e. QEMU may not be able to > > get all SPA of pmem from buf (in virtual address space) when > > calling XEN_DOMCTL_memory_mapping. > > Can mmap flag MAP_LOCKED or mlock(2) be used to enforce the > > entire pmem being mmaped? > > Ditto > No. If I take the above alternative for the first open, maybe the new hypercall above can inject page faults into dom0 for the unmapped virtual address so as to enforce dom0 Linux to create the page mapping. > > > 3.2.2 Alternative Design > > > > Jan Beulich's comments [7] on my question "why must pmem resource > > management and partition be done in hypervisor": > > | Because that's where memory management belongs. And PMEM, > > | other than PBLK, is just another form of RAM. > > | ... > > | The main issue is that this would imo be a layering violation > > > > George Dunlap's comments [8]: > > | This is not the case for PMEM. The whole point of PMEM (correct me if > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ used as fungible ram > > | I'm wrong) is to be used for long-term storage that survives over > > | reboot. It matters very much that a guest be given the same PRAM > > | after the host is rebooted that it was given before. It doesn't make > > | any sense to manage it the way Xen currently manages RAM (i.e., that > > | you request a page and get whatever Xen happens to give you). > > | > > | So if Xen is going to use PMEM, it will have to invent an entirely new > > | interface for guests, and it will have to keep track of those > > | resources across host reboots. In other words, it will have to > > | duplicate all the work that Linux already does. What do we gain from > > | that duplication? Why not just leverage what's already implemented in > > | dom0? > > and [9]: > > | Oh, right -- yes, if the usage model of PRAM is just "cheap slow RAM", > > | then you're right -- it is just another form of RAM, that should be > > | treated no differently than say, lowmem: a fungible resource that can be > > | requested by setting a flag. > > > > However, pmem is used more as persistent storage than fungible ram, > > and my design is for the former usage. I would like to leave the > > detection, driver and partition (either through namespace or file > > systems) of NVDIMM in Dom0 Linux kernel. > > > > I notice that current XEN_DOMCTL_memory_mapping does not make santiy > > check for the physical address and size passed from caller > > (QEMU). Can QEMU be always trusted? If not, we would need to make Xen > > aware of the SPA range of pmem so that it can refuse map physical > > address in neither the normal ram nor pmem. > > Indeed > > [...] > > > > > > 3.3 Guest ACPI Emulation > > > > 3.3.1 My Design > > > > Guest ACPI emulation is composed of two parts: building guest NFIT > > and SSDT that defines ACPI namespace devices for NVDIMM, and > > emulating guest _DSM. > > > > (1) Building Guest ACPI Tables > > > > This design reuses and extends hvmloader's existing mechanism that > > loads passthrough ACPI tables from binary files to load NFIT and > > SSDT tables built by QEMU: > > 1) Because the current QEMU does not building any ACPI tables when > > it runs as the Xen device model, this design needs to patch QEMU > > to build NFIT and SSDT (so far only NFIT and SSDT) in this case. > > > > 2) QEMU copies NFIT and SSDT to the end of guest memory below > > 4G. The guest address and size of those tables are written into > > xenstore (/local/domain/domid/hvmloader/dm-acpi/{address,length}). > > > > 3) hvmloader is patched to probe and load device model passthrough > > ACPI tables from above xenstore keys. The detected ACPI tables > > are then appended to the end of existing guest ACPI tables just > > like what current construct_passthrough_tables() does. > > > > Reasons for this design are listed below: > > - NFIT and SSDT in question are quite self-contained, i.e. they do > > not refer to other ACPI tables and not conflict with existing > > guest ACPI tables in Xen. Therefore, it is safe to copy them from > > QEMU and append to existing guest ACPI tables. > > > > - A primary portion of current and future vNVDIMM implementation is > > about building ACPI tables. And this design also leave the > > emulation of _DSM to QEMU which needs to keep consistency with > > NFIT and SSDT itself builds. Therefore, reusing NFIT and SSDT from > > QEMU can ease the maintenance. > > > > - Anthony's work to pass ACPI tables from the toolstack to hvmloader > > does not move building SSDT (and NFIT) to toolstack, so this > > design can still put them in hvmloader. > > If we start asking QEMU to build ACPI tables, why should we stop at NFIT > and SSDT? for easing my development of supporting vNVDIMM in Xen ... I mean NFIT and SSDT are the only two tables needed for this purpose and I'm afraid to break exiting guests if I completely switch to QEMU for guest ACPI tables. > Once upon a time somebody made the decision that ACPI tables > on Xen should be static and included in hvmloader. That might have been > a bad decision but at least it was coherent. Loading only *some* tables > from QEMU, but not others, it feels like an incomplete design to me. > > For example, QEMU is currently in charge of emulating the PCI bus, why > shouldn't it be QEMU that generates the PRT and MCFG? > To Keir, Jan and Andrew: Are there anything related to ACPI that must be done (or are better to be done) in hvmloader? > > > (2) Emulating Guest _DSM > > > > Because the same NFIT and SSDT are used, we can leave the emulation > > of guest _DSM to QEMU. Just as what it does with KVM, QEMU registers > > the _DSM buffer as MMIO region with Xen and then all guest > > evaluations of _DSM are trapped and emulated by QEMU. > > > > 3.3.2 Alternative Design 1: switching to QEMU > > > > Stefano Stabellini's comments [10]: > > | I don't think it is wise to have two components which both think are > > | in control of generating ACPI tables, hvmloader (soon to be the > > | toolstack with Anthony's work) and QEMU. From an architectural > > | perspective, it doesn't look robust to me. > > | > > | Could we take this opportunity to switch to QEMU generating the whole > > | set of ACPI tables? > > > > So an alternative design could be switching to QEMU to generate the > > whole set of guest ACPI tables. In this way, no controversy would > > happen between multiple agents QEMU and hvmloader. (is this what > > Stefano Stabellini mean by 'robust'?) > > Right > I understand that switching to QEMU for all guest ACPI tables would probably ease the maintenance for guest ACPI tables. However, in general, is it true that we currently need to fix regressions introduced by qemu-xen whenever we upgrade it to the upstream? If yes, is it reasonable to turn to QEMU for only NVDIMM ACPI tables and fix regressions/controversy whenever upgraded qemu-xen changes NVDIMM ACPI tables? > > > However, looking at the code building ACPI tables in QEMU and > > hvmloader, they are quite different. As ACPI tables are important for > > OS to boot and operate device, it's critical to ensure ACPI tables > > built by QEMU would not break existing guests on Xen. Though I > > believe it could be done after a thorough investigation and > > adjustment, it may take quite a lot of work and tests and should be > > another project besides enabling vNVDIMM in Xen. > > > > 3.3.3 Alternative Design 2: keeping in Xen > > > > Alternative to switching to QEMU, another design would be building > > NFIT and SSDT in hvmloader or toolstack. > > > > The amount and parameters of sub-structures in guest NFIT vary > > according to different vNVDIMM configurations that can not be decided > > at compile-time. In contrast, current hvmloader and toolstack can > > only build static ACPI tables, i.e. their contents are decided > > statically at compile-time and independent from the guest > > configuration. In order to build guest NFIT at runtime, this design > > may take following steps: > > (1) xl converts NVDIMM configurations in xl.cfg to corresponding QEMU > > options, > > > > (2) QEMU accepts above options, figures out the start SPA range > > address/size/NVDIMM device handles/..., and writes them in > > xenstore. No ACPI table is built by QEMU. > > > > (3) Either xl or hvmloader reads above parameters from xenstore and > > builds the NFIT table. > > > > For guest SSDT, it would take more work. The ACPI namespace devices > > are defined in SSDT by AML, so an AML builder would be needed to > > generate those definitions at runtime. > > > > This alternative design still needs more work than the first design. > > I prefer switching to QEMU building all ACPI tables for devices that it > is emulating. However this alternative is good too because it is > coherent with the current design. > I would prefer to this one if the final conclusion is that only one agent should be allowed to build guest ACPI. As I said above, it looks like a big change to switch to QEMU for all ACPI tables and I'm afraid it would break some existing guests. Thanks, Haozhong > > > References: > > [1] ACPI Specification v6, > > http://www.uefi.org/sites/default/files/resources/ACPI_6.0.pdf > > [2] NVDIMM Namespace Specification, > > http://pmem.io/documents/NVDIMM_Namespace_Spec.pdf > > [3] NVDIMM Block Window Driver Writer's Guide, > > http://pmem.io/documents/NVDIMM_Driver_Writers_Guide.pdf > > [4] NVDIMM DSM Interface Example, > > http://pmem.io/documents/NVDIMM_DSM_Interface_Example.pdf > > [5] UEFI Specification v2.6, > > http://www.uefi.org/sites/default/files/resources/UEFI%20Spec%202_6.pdf > > [6] Intel Architecture Instruction Set Extensions Programming Reference, > > > > https://software.intel.com/sites/default/files/managed/07/b7/319433-023.pdf > > [7] http://www.gossamer-threads.com/lists/xen/devel/414945#414945 > > [8] http://www.gossamer-threads.com/lists/xen/devel/415658#415658 > > [9] http://www.gossamer-threads.com/lists/xen/devel/415681#415681 > > [10] > > http://lists.xenproject.org/archives/html/xen-devel/2016-01/msg00271.html > > _______________________________________________ Xen-devel mailing list Xen-devel@xxxxxxxxxxxxx http://lists.xen.org/xen-devel

©2013 Xen Project, A Linux Foundation Collaborative Project. All Rights Reserved.
Linux Foundation is a registered trademark of The Linux Foundation.
Xen Project is a trademark of The Linux Foundation.

Re: [Xen-devel] [RFC Design Doc] Add vNVDIMM support for Xen