[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] Re: [Xen-devel] RFC: QEMU bumping memory limit and domain restore
I fat-fingered Andrew's email address. Really CC him this time. On Tue, Jun 02, 2015 at 03:05:07PM +0100, Wei Liu wrote: > Previous discussion at [0]. > > For the benefit of discussion, we refer to max_memkb inside hypervisor > as hv_max_memkb (name subject to improvement). That's the maximum number > of memory a domain can use. > > Libxl doesn't know hv_max_memkb for a domain needs prior to QEMU start-up > because of optional ROMs etc. > > Libxl doesn't know the hv_max_memkb even after QEMU start-up, because > there is no mechanism to community between QEMU and libxl. This is an > area that needs improvement, we've encountered problems in this area > before. > > QEMU calls xc_domain_setmaxmem to increase hv_max_memkb by N pages. Those > pages are only accounted in hypervisor. During migration, libxl > (currently) doesn't extract that value from hypervisor. > > So now the problem is on the remote end: > > 1. Libxl indicates domain needs X pages. > 2. Domain actually needs X + N pages. > 3. Remote end tries to write N more pages and fail. > > This behaviour currently doesn't affect normal migration (that you > transfer libxl JSON to remote, construct a domain, then start QEMU) > because QEMU won't bump hv_max_memkb again. This is by design and > reflected in QEMU code. > > This behaviour affects COLO and becomes a bug in that case, because > secondary VM's QEMU doesn't go through the same start-of-day > initialisation (Hongyang, correct me if I'm wrong), i.e. no bumping > hv_max_memkb inside QEMU. > > Andrew plans to embed JSON inside migration v2 and COLO is based on > migration v2. The bug is fixed if JSON is correct in the first place. > > As COLO is not yet upstream, so this bug is not a blocker for 4.6. But > it should be fixed for the benefit of COLO. > > So here is a proof of concept patch to record and honour that value > during migration. A new field is added in IDL. Note that we don't > provide xl level config option for it and mandate it to be default value > during domain creation. This is to prevent libxl user from using it to > avoid unforeseen repercussions. > > This patch is compiled test only. If we agree this is the way to go I > will test and submit a proper patch. > > Wei. > > [0] <1428941353-18673-1-git-send-email-dslutz@xxxxxxxxxxx> > > ---8<--- > >From ab9dc179ea4ee26eb88f61f8dad36dd01b63bb6b Mon Sep 17 00:00:00 2001 > From: Wei Liu <wei.liu2@xxxxxxxxxx> > Date: Tue, 2 Jun 2015 14:53:20 +0100 > Subject: [PATCH] libxl: record and honour hv_max_memkb > > The new field hv_max_memkb in IDL is used to record "max_memkb" inside > hypervisor. That reflects the maximum memory a guest can ask for. > > This field is mandated to be default value during guest creation to > avoid unforeseen repercussions. It's only honour when restoring a guest. > > (XXX compiled test only at this stage) > > Signed-off-by: Wei Liu <wei.liu2@xxxxxxxxxx> > --- > tools/libxl/libxl.c | 17 +++++++++-------- > tools/libxl/libxl_create.c | 6 ++++++ > tools/libxl/libxl_dom.c | 9 +++++++-- > tools/libxl/libxl_types.idl | 1 + > 4 files changed, 23 insertions(+), 10 deletions(-) > > diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c > index 9117b01..72fec8b 100644 > --- a/tools/libxl/libxl.c > +++ b/tools/libxl/libxl.c > @@ -6614,6 +6614,7 @@ int libxl_retrieve_domain_configuration(libxl_ctx *ctx, > uint32_t domid, > GC_INIT(ctx); > int rc; > libxl__domain_userdata_lock *lock = NULL; > + uint64_t hv_max_memkb; > > CTX_LOCK; > > @@ -6654,6 +6655,7 @@ int libxl_retrieve_domain_configuration(libxl_ctx *ctx, > uint32_t domid, > } > libxl_uuid_copy(ctx, &d_config->c_info.uuid, &info.uuid); > libxl_dominfo_dispose(&info); > + hv_max_memkb = info.max_memkb; /* store and use later */ > } > > /* Memory limits: > @@ -6661,17 +6663,15 @@ int libxl_retrieve_domain_configuration(libxl_ctx > *ctx, uint32_t domid, > * Currently there are three memory limits: > * 1. "target" in xenstore (originally memory= in config file) > * 2. "static-max" in xenstore (originally maxmem= in config file) > - * 3. "max_memkb" in hypervisor > - * > - * The third one is not visible and currently managed by > - * toolstack. In order to rebuild a domain we only need to have > - * "target" and "static-max". > + * 3. "max_memkb" in hypervisor (corresponds to hv_max_memkb in > + * idl, not visible to xl level) > */ > { > - uint32_t target_memkb = 0, max_memkb = 0; > + uint32_t target_memkb = 0, static_max_memkb = 0; > > /* "target" */ > - rc = libxl__get_memory_target(gc, domid, &target_memkb, &max_memkb); > + rc = libxl__get_memory_target(gc, domid, &target_memkb, > + &static_max_memkb); > if (rc) { > LOG(ERROR, "fail to get memory target for domain %d", domid); > goto out; > @@ -6683,7 +6683,8 @@ int libxl_retrieve_domain_configuration(libxl_ctx *ctx, > uint32_t domid, > d_config->b_info.target_memkb = target_memkb + > d_config->b_info.video_memkb; > > - d_config->b_info.max_memkb = max_memkb; > + d_config->b_info.max_memkb = static_max_memkb; > + d_config->b_info.hv_max_memkb = hv_max_memkb; > } > > /* Devices: disk, nic, vtpm, pcidev etc. */ > diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c > index 86384d2..d35a393 100644 > --- a/tools/libxl/libxl_create.c > +++ b/tools/libxl/libxl_create.c > @@ -813,6 +813,12 @@ static void initiate_domain_create(libxl__egc *egc, > > domid = 0; > > + if (restore_fd <= 0 && > + d_config->b_info.hv_max_memkb != LIBXL_MEMKB_DEFAULT) { > + LOG(ERROR, "Setting hv_max_memkb when creating domain"); > + goto error_out; > + } > + > if (d_config->c_info.ssid_label) { > char *s = d_config->c_info.ssid_label; > ret = libxl_flask_context_to_sid(ctx, s, strlen(s), > diff --git a/tools/libxl/libxl_dom.c b/tools/libxl/libxl_dom.c > index a0c9850..5e33611 100644 > --- a/tools/libxl/libxl_dom.c > +++ b/tools/libxl/libxl_dom.c > @@ -408,8 +408,13 @@ int libxl__build_pre(libxl__gc *gc, uint32_t domid, > } > } > > - if (xc_domain_setmaxmem(ctx->xch, domid, info->target_memkb + > - LIBXL_MAXMEM_CONSTANT) < 0) { > + /* hv_max_memkb is set in restore path, honour it. */ > + if (info->hv_max_memkb == LIBXL_MEMKB_DEFAULT) > + rc = xc_domain_setmaxmem(ctx->xch, domid, info->target_memkb + > + LIBXL_MAXMEM_CONSTANT); > + else > + rc = xc_domain_setmaxmem(ctx->xch, domid, info->hv_max_memkb); > + if (rc < 0) { > LIBXL__LOG_ERRNO(ctx, LIBXL__LOG_ERROR, "Couldn't set max memory"); > return ERROR_FAIL; > } > diff --git a/tools/libxl/libxl_types.idl b/tools/libxl/libxl_types.idl > index 23f27d4..3c306c0 100644 > --- a/tools/libxl/libxl_types.idl > +++ b/tools/libxl/libxl_types.idl > @@ -377,6 +377,7 @@ libxl_domain_build_info = Struct("domain_build_info",[ > ("tsc_mode", libxl_tsc_mode), > ("max_memkb", MemKB), > ("target_memkb", MemKB), > + ("hv_max_memkb", MemKB), > ("video_memkb", MemKB), > ("shadow_memkb", MemKB), > ("rtc_timeoffset", uint32), > -- > 1.9.1 _______________________________________________ Xen-devel mailing list Xen-devel@xxxxxxxxxxxxx http://lists.xen.org/xen-devel
|
Lists.xenproject.org is hosted with RackSpace, monitoring our |