Re: [Xen-devel] [Xen-Devel] [PATCH] [GSOC14] refactored mempaging code from xenpaging to libxc.

[Andres Lagar Cavilla <andres@xxxxxxxxxxxxxxxx>]

On Thu, Jun 12, 2014 at 6:22 AM, Andrew Cooper <andrew.cooper3@xxxxxxxxxx> wrote:

On 12/06/14 10:23, Dushyant Behl wrote:
> This patch is part of the work done under the gsoc project -
> Lazy Restore Using Memory Paging.
>
> This patch moves the code to initialize mempaging from xenpaging to libxc.

The source code is named libxc, but technically you are moving it into
libxenguest which is a separate library from libxenctrl.

> The code refractored from xenpaging is the code which sets up paging,
> initializes a shared ring and event channel to communicate with xen. This
> communication is done between the hypervisor and the dom0 tool which performs
> the activity of pager. The xenpaging code is changed to use the newly created
> routines and is tested to properly build and work with this code.
>
> The refractoring is done so that any tool which will act as pager in
> lazy restore or use memory paging can use a same routines to initialize mempaging.
> This refactoring will also allow any future (in-tree) tools to use mempaging.
>
> Signed-off-by: Dushyant Behl <myselfdushyantbehl@xxxxxxxxx>
> Reviewed-by: Andres Lagar-Cavilla <andres@xxxxxxxxxxxxxxxx>Â

> ---
> Âtools/libxc/Makefile        Â|  3 +
> Âtools/libxc/xc_mem_paging_setup.c  | 132 ++++++++++++++++++++++++++++++++++++
> Âtools/libxc/xenctrl.h        | Â12 ++++
> Âtools/ocaml/libs/xc/xenctrl_stubs.c | Â11 +--
> Âtools/xenpaging/Makefile      Â|  4 +-
> Âtools/xenpaging/xenpaging.c     | Â93 +++----------------------
> Â6 files changed, 167 insertions(+), 88 deletions(-)
> Âcreate mode 100644 tools/libxc/xc_mem_paging_setup.c
>
> diff --git a/tools/libxc/Makefile b/tools/libxc/Makefile
> index a74b19e..2439853 100644
> --- a/tools/libxc/Makefile
> +++ b/tools/libxc/Makefile
> @@ -69,6 +69,9 @@ GUEST_SRCS-$(CONFIG_ARM) Â Â += xc_dom_armzimageloader.c
> ÂGUEST_SRCS-y         += xc_dom_binloader.c
> ÂGUEST_SRCS-y         += xc_dom_compat_linux.c
>
> +# mem paging setup
> +GUEST_SRCS-y         += xc_mem_paging_setup.c
> +

I don't think you need the comment here. ÂIt is obvious from the name of
the file, although I would suggest you name it "xc_dom_paging.c" or
something a tad less specific.

> ÂGUEST_SRCS-$(CONFIG_X86) Â Â += xc_dom_x86.c
> ÂGUEST_SRCS-$(CONFIG_X86) Â Â += xc_cpuid_x86.c
> ÂGUEST_SRCS-$(CONFIG_X86) Â Â += xc_hvm_build_x86.c
> diff --git a/tools/libxc/xc_mem_paging_setup.c b/tools/libxc/xc_mem_paging_setup.c
> new file mode 100644
> index 0000000..125d203
> --- /dev/null
> +++ b/tools/libxc/xc_mem_paging_setup.c
> @@ -0,0 +1,132 @@
> +/*
> + * tools/libxc/xc_mem_paging_setup.c
> + *
> + * Routines to initialize memory paging. Create shared ring
> + * and event channels to communicate with the hypervisor.
> + *
> + * Copyright (c) 2014 Dushyant Behl
> + * Copyright (c) 2009 by Citrix Systems, Inc. (Patrick Colp)
> + *
> + * This library is free software; you can redistribute it and/or
> + * modify it under the terms of the GNU Lesser General Public
> + * License as published by the Free Software Foundation; either
> + * version 2.1 of the License, or (at your option) any later version.
> + *
> + * This library is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. ÂSee the GNU
> + * Lesser General Public License for more details.
> + *
> + * You should have received a copy of the GNU Lesser General Public
> + * License along with this library; if not, write to the Free Software
> + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA Â02110-1301 USA
> + */
> +
> +#include "xc_private.h"
> +#include <xen/event_channel.h>
> +#include <xen/mem_event.h>
> +
> +/*
> + * Mem paging ring and event channel setup routine.
> + * Setup a shared ring and an event channel to communicate between
> + * hypervisor and the tool performing mem paging operations.

Please document here, or in the header file, what the error semantics of
the function are. ÂLibxc is a complete mess so these things need
spelling out.

> + */
> +int xc_mem_paging_ring_setup(xc_interface *xch, domid_t domain_id,
> + Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Âvoid *ring_page, int *port,
> + Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Âuint32_t *evtchn_port,
> + Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Âxc_evtchn **xce_handle,
> + Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Âmem_event_back_ring_t *back_ring)

The parameters should be aligned with the xc_interface.

> +{
> + Â Âint rc;
> + Â Âunsigned long ring_pfn, mmap_pfn;

uint64_t's as these are hvmparams

David Vrabel's GenID series fixes the prototypes of
xc_[gs]et_hvm_param() to be sane.

Andrew, we were thinking this should be rebased onto your git branch for save restore rework, which I assume contains these GenID changes as well. Can you point to the right git coordinates? ThanksÂ

> +
> + Â Â/* Map the ring page */
> + Â Âxc_get_hvm_param(xch, domain_id, HVM_PARAM_PAGING_RING_PFN, &ring_pfn);
> + Â Âmmap_pfn = ring_pfn;
> + Â Âring_page = xc_map_foreign_batch(xch, domain_id,
> + Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â ÂPROT_READ | PROT_WRITE, &mmap_pfn, 1);

You are stuffing a pointer into an unsigned long?

ring_page is void * ... mmap_pfn is ulong ....Â

> + Â Âif ( mmap_pfn & XEN_DOMCTL_PFINFO_XTAB )

What is this check for?

foreign_batch has the old semantics of setting the MSB of the pfn in error. Maybe use map_foreign_bulk with cleaner error reporting?Â

> + Â Â{
> + Â Â Â Â/* Map failed, populate ring page */
> + Â Â Â Ârc = xc_domain_populate_physmap_exact(xch, domain_id,
> + Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â1, 0, 0, &ring_pfn);
> + Â Â Â Âif ( rc != 0 )
> + Â Â Â Â{
> + Â Â Â Â Â ÂPERROR("Failed to populate ring gfn\n");
> + Â Â Â Â Â Âreturn 1;

You are (implicitly) setting errno. ÂPlease return -1;

> + Â Â Â Â}
> +
> + Â Â Â Âmmap_pfn = ring_pfn;
> + Â Â Â Âring_page = xc_map_foreign_batch(xch, domain_id,
> + Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â ÂPROT_READ | PROT_WRITE, &mmap_pfn, 1);
> +
> + Â Â Â Âif ( mmap_pfn & XEN_DOMCTL_PFINFO_XTAB )
> + Â Â Â Â{
> + Â Â Â Â Â ÂPERROR("Could not map the ring page\n");
> + Â Â Â Â Â Âreturn 1;
> + Â Â Â Â}
> + Â Â}
> +
> + Â Â/* Initialise Xen */
> + Â Ârc = xc_mem_paging_enable(xch, domain_id, evtchn_port);
> + Â Âif ( rc != 0 )
> + Â Â{
> + Â Â Â Âswitch ( errno ) {
> + Â Â Â Â Â Âcase EBUSY:
> + Â Â Â Â Â Â Â ÂERROR("mempaging is (or was) active on this domain");
> + Â Â Â Â Â Â Â Âbreak;
> + Â Â Â Â Â Âcase ENODEV:
> + Â Â Â Â Â Â Â ÂERROR("mempaging requires Hardware Assisted Paging");
> + Â Â Â Â Â Â Â Âbreak;
> + Â Â Â Â Â Âcase EMLINK:
> + Â Â Â Â Â Â Â ÂERROR("mempaging not supported while iommu passthrough is enabled");
> + Â Â Â Â Â Â Â Âbreak;
> + Â Â Â Â Â Âcase EXDEV:
> + Â Â Â Â Â Â Â ÂERROR("mempaging not supported in a PoD guest");
> + Â Â Â Â Â Â Â Âbreak;
> + Â Â Â Â Â Âdefault:
> + Â Â Â Â Â Â Â ÂPERROR("mempaging: error initialising shared page");
> + Â Â Â Â Â Â Â Âbreak;
> + Â Â Â Â}
> + Â Â Â Âreturn 1;
> + Â Â}
> +
> + Â Â/* Open event channel */
> + Â Â*xce_handle = xc_evtchn_open(NULL, 0);
> + Â Âif ( *xce_handle == NULL )
> + Â Â{
> + Â Â Â ÂPERROR("Failed to open event channel");
> + Â Â Â Âreturn 1;
> + Â Â}
> +
> + Â Â/* Bind event notification */
> + Â Ârc = xc_evtchn_bind_interdomain(*xce_handle, domain_id, *evtchn_port);
> + Â Âif ( rc < 0 )
> + Â Â{
> + Â Â Â ÂPERROR("Failed to bind event channel");
> + Â Â Â Âreturn 1;
> + Â Â}
> + Â Â*port = rc;
> +
> + Â Â/* Initialise ring */
> + Â ÂSHARED_RING_INIT((mem_event_sring_t *)ring_page);
> + Â ÂBACK_RING_INIT(back_ring, (mem_event_sring_t *)ring_page, PAGE_SIZE);
> +
> + Â Â/* Now that the ring is set, remove it from the guest's physmap */
> + Â Âif ( xc_domain_decrease_reservation_exact(xch, domain_id, 1, 0, &ring_pfn) )
> + Â Â{
> + Â Â Â ÂPERROR("Failed to remove ring from guest physmap");
> + Â Â Â Âreturn 1;
> + Â Â}
> +
> + Â Âreturn 0;
> +}
> +
> +/*
> + * Local variables:
> + * mode: C
> + * c-file-style: "BSD"
> + * c-basic-offset: 4
> + * indent-tabs-mode: nil
> + * End:
> + */
> diff --git a/tools/libxc/xenctrl.h b/tools/libxc/xenctrl.h
> index 02129f7..c871a2f 100644
> --- a/tools/libxc/xenctrl.h
> +++ b/tools/libxc/xenctrl.h
> @@ -47,6 +47,7 @@
> Â#include <xen/xsm/flask_op.h>
> Â#include <xen/tmem.h>
> Â#include <xen/kexec.h>
> +#include <xen/mem_event.h>
>
> Â#include "xentoollog.h"
>
> @@ -2039,6 +2040,17 @@ int xc_mem_paging_prep(xc_interface *xch, domid_t domain_id, unsigned long gfn);
> Âint xc_mem_paging_load(xc_interface *xch, domid_t domain_id,
> Â Â Â Â Â Â Â Â Â Â Â Â Âunsigned long gfn, void *buffer);
>
> +/*
> + * Mem paging ring and event channel setup routine.
> + * Setup a shared ring and an event channel to communicate between
> + * hypervisor and the tool performing mem paging operations.
> + */
> +int xc_mem_paging_ring_setup(xc_interface *xch, domid_t domain_id,
> + Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Âvoid *ring_page, int *port,
> + Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Âuint32_t *evtchn_port,
> + Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Âxc_evtchn **xceh,
> + Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Âmem_event_back_ring_t *_back_ring);
> +
> Â/**
> Â * Access tracking operations.
> Â * Supported only on Intel EPT 64 bit processors.
> diff --git a/tools/ocaml/libs/xc/xenctrl_stubs.c b/tools/ocaml/libs/xc/xenctrl_stubs.c
> index ff29b47..37a4db7 100644
> --- a/tools/ocaml/libs/xc/xenctrl_stubs.c
> +++ b/tools/ocaml/libs/xc/xenctrl_stubs.c
> @@ -521,13 +521,16 @@ CAMLprim value stub_xc_evtchn_reset(value xch, value domid)
> Â Â Â CAMLreturn(Val_unit);
> Â}
>
> -
> -#define RING_SIZE 32768
> -static char ring[RING_SIZE];
> +/*
> + * The name is kept BUF_RING_SIZE, because the name RING_SIZE
> + * collides with the xen shared ring definitions in io/ring.h
> + */
> +#define BUF_RING_SIZE 32768
> +static char ring[BUF_RING_SIZE];
>
> ÂCAMLprim value stub_xc_readconsolering(value xch)
> Â{
> - Â Â unsigned int size = RING_SIZE - 1;
> + Â Â unsigned int size = BUF_RING_SIZE - 1;
> Â Â Â char *ring_ptr = ring;
> Â Â Â int retval;
>
> diff --git a/tools/xenpaging/Makefile b/tools/xenpaging/Makefile
> index 548d9dd..ea370d3 100644
> --- a/tools/xenpaging/Makefile
> +++ b/tools/xenpaging/Makefile
> @@ -1,8 +1,8 @@
> ÂXEN_ROOT=$(CURDIR)/../..
> Âinclude $(XEN_ROOT)/tools/Rules.mk
>
> -CFLAGS += $(CFLAGS_libxenctrl) $(CFLAGS_libxenstore) $(PTHREAD_CFLAGS)
> -LDLIBS += $(LDLIBS_libxenctrl) $(LDLIBS_libxenstore) $(PTHREAD_LIBS)
> +CFLAGS += $(CFLAGS_libxenctrl) $(CFLAGS_libxenguest) $(CFLAGS_libxenstore) $(PTHREAD_CFLAGS)
> +LDLIBS += $(LDLIBS_libxenctrl) $(LDLIBS_libxenguest) $(LDLIBS_libxenstore) $(PTHREAD_LIBS)

You don't introduce any libxenstore dependencies as far as I can see.

Ditto.Â

Andres

> ÂLDFLAGS += $(PTHREAD_LDFLAGS)
>
> ÂPOLICY Â Â= default
> diff --git a/tools/xenpaging/xenpaging.c b/tools/xenpaging/xenpaging.c
> index 5ef2f09..2b81408 100644
> --- a/tools/xenpaging/xenpaging.c
> +++ b/tools/xenpaging/xenpaging.c
> @@ -281,7 +281,6 @@ static struct xenpaging *xenpaging_init(int argc, char *argv[])
> Â Â Âxentoollog_logger *dbg = NULL;
> Â Â Âchar *p;
> Â Â Âint rc;
> - Â Âunsigned long ring_pfn, mmap_pfn;
>
> Â Â Â/* Allocate memory */
> Â Â Âpaging = calloc(1, sizeof(struct xenpaging));
> @@ -338,91 +337,21 @@ static struct xenpaging *xenpaging_init(int argc, char *argv[])
> Â Â Â Â Âgoto err;
> Â Â Â}
>
> - Â Â/* Map the ring page */
> - Â Âxc_get_hvm_param(xch, paging->mem_event.domain_id,
> - Â Â Â Â Â Â Â Â Â Â Â ÂHVM_PARAM_PAGING_RING_PFN, &ring_pfn);
> - Â Âmmap_pfn = ring_pfn;
> - Â Âpaging->mem_event.ring_page =
> - Â Â Â Âxc_map_foreign_batch(xch, paging->mem_event.domain_id,
> - Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â ÂPROT_READ | PROT_WRITE, &mmap_pfn, 1);
> - Â Âif ( mmap_pfn & XEN_DOMCTL_PFINFO_XTAB )
> - Â Â{
> - Â Â Â Â/* Map failed, populate ring page */
> - Â Â Â Ârc = xc_domain_populate_physmap_exact(paging->xc_handle,
> - Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Âpaging->mem_event.domain_id,
> - Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â1, 0, 0, &ring_pfn);
> - Â Â Â Âif ( rc != 0 )
> - Â Â Â Â{
> - Â Â Â Â Â ÂPERROR("Failed to populate ring gfn\n");
> - Â Â Â Â Â Âgoto err;
> - Â Â Â Â}
> -
> - Â Â Â Âmmap_pfn = ring_pfn;
> - Â Â Â Âpaging->mem_event.ring_page =
> - Â Â Â Â Â Âxc_map_foreign_batch(xch, paging->mem_event.domain_id,
> - Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â ÂPROT_READ | PROT_WRITE, &mmap_pfn, 1);
> - Â Â Â Âif ( mmap_pfn & XEN_DOMCTL_PFINFO_XTAB )
> - Â Â Â Â{
> - Â Â Â Â Â ÂPERROR("Could not map the ring page\n");
> - Â Â Â Â Â Âgoto err;
> - Â Â Â Â}
> - Â Â}
> -
> - Â Â/* Initialise Xen */
> - Â Ârc = xc_mem_paging_enable(xch, paging->mem_event.domain_id,
> - Â Â Â Â Â Â Â Â Â Â Â Â Â Â &paging->mem_event.evtchn_port);
> - Â Âif ( rc != 0 )
> - Â Â{
> - Â Â Â Âswitch ( errno ) {
> - Â Â Â Â Â Âcase EBUSY:
> - Â Â Â Â Â Â Â ÂERROR("xenpaging is (or was) active on this domain");
> - Â Â Â Â Â Â Â Âbreak;
> - Â Â Â Â Â Âcase ENODEV:
> - Â Â Â Â Â Â Â ÂERROR("xenpaging requires Hardware Assisted Paging");
> - Â Â Â Â Â Â Â Âbreak;
> - Â Â Â Â Â Âcase EMLINK:
> - Â Â Â Â Â Â Â ÂERROR("xenpaging not supported while iommu passthrough is enabled");
> - Â Â Â Â Â Â Â Âbreak;
> - Â Â Â Â Â Âcase EXDEV:
> - Â Â Â Â Â Â Â ÂERROR("xenpaging not supported in a PoD guest");
> - Â Â Â Â Â Â Â Âbreak;
> - Â Â Â Â Â Âdefault:
> - Â Â Â Â Â Â Â ÂPERROR("Error initialising shared page");
> - Â Â Â Â Â Â Â Âbreak;
> - Â Â Â Â}
> - Â Â Â Âgoto err;
> - Â Â}
> -
> - Â Â/* Open event channel */
> - Â Âpaging->mem_event.xce_handle = xc_evtchn_open(NULL, 0);
> - Â Âif ( paging->mem_event.xce_handle == NULL )
> - Â Â{
> - Â Â Â ÂPERROR("Failed to open event channel");
> - Â Â Â Âgoto err;
> - Â Â}
> -
> - Â Â/* Bind event notification */
> - Â Ârc = xc_evtchn_bind_interdomain(paging->mem_event.xce_handle,
> + Â Â/* Initialize paging by setting up ring and event channel. */
> + Â Ârc = xc_mem_paging_ring_setup(paging->xc_handle,
> Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Âpaging->mem_event.domain_id,
> - Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Âpaging->mem_event.evtchn_port);
> - Â Âif ( rc < 0 )
> - Â Â{
> - Â Â Â ÂPERROR("Failed to bind event channel");
> + Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Âpaging->mem_event.ring_page,
> + Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â&paging->mem_event.port,
> + Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â&paging->mem_event.evtchn_port,
> + Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â&paging->mem_event.xce_handle,
> + Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â&paging->mem_event.back_ring);
> + Â Âif( rc )
> + Â Â{
> + Â Â Â ÂPERROR("Could not initialize mem paging\n");
> Â Â Â Â Âgoto err;
> Â Â Â}
>
> - Â Âpaging->mem_event.port = rc;
> -
> - Â Â/* Initialise ring */
> - Â ÂSHARED_RING_INIT((mem_event_sring_t *)paging->mem_event.ring_page);
> - Â ÂBACK_RING_INIT(&paging->mem_event.back_ring,
> - Â Â Â Â Â Â Â Â Â (mem_event_sring_t *)paging->mem_event.ring_page,
> - Â Â Â Â Â Â Â Â Â PAGE_SIZE);
> -
> - Â Â/* Now that the ring is set, remove it from the guest's physmap */
> - Â Âif ( xc_domain_decrease_reservation_exact(xch,
> - Â Â Â Â Â Â Â Â Â Âpaging->mem_event.domain_id, 1, 0, &ring_pfn) )
> - Â Â Â ÂPERROR("Failed to remove ring from guest physmap");
> + Â ÂDPRINTF("ring and event channel setup successful\n");

You don't need trailing \n's in DPRINTF()s

~Andrew

>
> Â Â Â/* Get max_pages from guest if not provided via cmdline */
> Â Â Âif ( !paging->max_pages )

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel