|
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] Re: [Xen-devel] [PATCH v6 00/21] libxl: domain save/restore: run in a separate process
On Thu, 2012-06-28 at 14:38 +0100, Ian Jackson wrote:
> I wrote:
> > This is v5 of my series to asyncify save/restore, rebased to tip and
> > retested. There are minor changes to 3 patches, as discussed on-list,
> > marked with "*" below:
> ...
> > * 06/21 libxl: domain save/restore: run in a separate process
> ...
> > However, first I will invite Shriram to check that Remus is still
> > working. (I can't conveniently do this with this message due to
> > shoddiness in git-send-email.)
>
> Following testing by Shriram (thanks) I have an updated version of
> 06/21. For the sake of everyone's sanity (and your MUAs) I shan't
> repost the whole series.
>
> Here is v6 of 06/21, which is simply the previous one with my earlier
> fixup patch folded in.
>
> CC Ian Campbell since he'd acked the previous one. Ian, I have left
> your ack on this version. I trust that's OK.
Absolutely fine.
Does this mean this series is now ready to go in?
I did wonder when I saw the incremental patch if some of those internal
callback pointers could perhaps be properly typed instead of void
(because they all end up taking the same pointer type), but lets not
worry about that here.
Ian.
>
> Thanks,
> Ian.
>
>
> From: Ian Jackson <ian.jackson@xxxxxxxxxxxxx>
> Subject: [PATCH] libxl: domain save/restore: run in a separate process
>
> libxenctrl expects to be able to simply run the save or restore
> operation synchronously. This won't work well in a process which is
> trying to handle multiple domains.
>
> The options are:
>
> - Block such a whole process (eg, the whole of libvirt) while
> migration completes (or until it fails).
>
> - Create a thread to run xc_domain_save and xc_domain_restore on.
> This is quite unpalatable. Multithreaded programming is error
> prone enough without generating threads in libraries, particularly
> if the thread does some very complex operation.
>
> - Fork and run the operation in the child without execing. This is
> no good because we would need to negotiate with the caller about
> fds we would inherit (and we might be a very large process).
>
> - Fork and exec a helper.
>
> Of these options the latter is the most palatable.
>
> Consequently:
>
> * A new helper program libxl-save-helper (which does both save and
> restore). It will be installed in /usr/lib/xen/bin. It does not
> link against libxl, only libxc, and its error handling does not
> need to be very advanced. It does contain a plumbing through of
> the logging interface into the callback stream.
>
> * A small ad-hoc protocol between the helper and libxl which allows
> log messages and the libxc callbacks to be passed up and down.
> Protocol doc comment is in libxl_save_helper.c.
>
> * To avoid a lot of tedium the marshalling boilerplate (stubs for the
> helper and the callback decoder for libxl) is generated with a
> small perl script.
>
> * Implement new functionality to spawn the helper, monitor its
> output, provide responses, and check on its exit status.
>
> * The functions libxl__xc_domain_restore_done and
> libxl__xc_domain_save_done now turn out to want be called in the
> same place. So make their state argument a void* so that the two
> functions are type compatible.
>
> The domain save path still writes the qemu savefile synchronously.
> This will need to be fixed in a subsequent patch.
>
> Signed-off-by: Ian Jackson <ian.jackson@xxxxxxxxxxxxx>
> Acked-by: Ian Campbell <ian.campbell@xxxxxxxxxx>
>
> -
> Changes in v6:
> * The void* passed to the callback was being treated as a
> libxl__domain_suspend_state* by the remus callbacks; this is a
> holdover from a much earlier version of the series. It is now
> properly converted to a libxl__save_helper_state and then the dss
> extracted with CONTAINER_OF.
> * The way remus works means that the toolstack save callback is
> invoked more than once, which the helper's implementation was not
> prepared to deal with. Fix this by moving the rewind of the fd
> into the helper.
>
> Changes in v5:
> * assert that preserve_fds are >2.
>
> Changes in v4:
> * Migration stream fd is handled specially by the run_helper
> function, rather than simply being a numarg. Specifically:
> - dup it to a safe fd number if necessary.
> - clear cloexec flag fd before execing helper
> * Toolstack data fd argument to run_helper replaced with
> generic preserve_fds array, which get cloexec cleared.
> * libxl__xc_domain_save uses supplied callback function pointer,
> rather than calling libxl__toolstack_save directly;
> toolstack data save callback is only supplied to libxc if
> in-libxl caller supplied a callback.
> * libxl-save-helper is not needlessly linked against libxl.
> * Code which prepares pipes for helper clarified.
> * Deal properly with, and log properly, POLLPRI/POLLERR on
> pipe to save helper.
> * Spelling fix in perl script comment.
> * In message generator, use better names for the ends of serial
> conditional here documents.
> * Makefile does $(INSTALL_DIR) $(DESTDIR)$(PRIVATE_BINDIR)
>
> Changes in v3:
> * Suppress errno value in debug message when helper reports successful
> completion.
> * Significant consequential changes to cope with changes to
> earlier patches in the series.
>
> Changes in v2:
> * Helper path can be overridden by an environment variable for testing.
> * Add a couple of debug logging messages re toolstack data.
> * Fixes from testing.
> * Helper protocol message lengths (and numbers) are 16-bit which
> more clearly avoids piling lots of junk on the stack.
> * Merged with remus changes.
> * Callback implementations in libxl now called via pointers
> so remus can have its own callbacks.
> * Better namespace prefixes on autogenerated names etc.
> * Autogenerator can generate debugging printfs too.
>
> ---
> .gitignore | 1 +
> .hgignore | 2 +
> tools/libxl/Makefile | 22 ++-
> tools/libxl/libxl_create.c | 22 ++-
> tools/libxl/libxl_dom.c | 42 +++--
> tools/libxl/libxl_internal.h | 56 +++++-
> tools/libxl/libxl_save_callout.c | 361 +++++++++++++++++++++++++++++++-
> tools/libxl/libxl_save_helper.c | 283 +++++++++++++++++++++++++
> tools/libxl/libxl_save_msgs_gen.pl | 397
> ++++++++++++++++++++++++++++++++++++
> 9 files changed, 1146 insertions(+), 40 deletions(-)
>
> diff --git a/.gitignore b/.gitignore
> index 7770e54..3451e52 100644
> --- a/.gitignore
> +++ b/.gitignore
> @@ -353,6 +353,7 @@ tools/libxl/_*.[ch]
> tools/libxl/testidl
> tools/libxl/testidl.c
> tools/libxl/*.pyc
> +tools/libxl/libxl-save-helper
> tools/blktap2/control/tap-ctl
> tools/firmware/etherboot/eb-roms.h
> tools/firmware/etherboot/gpxe-git-snapshot.tar.gz
> diff --git a/.hgignore b/.hgignore
> index 27d8f79..05304ea 100644
> --- a/.hgignore
> +++ b/.hgignore
> @@ -180,9 +180,11 @@
> ^tools/libxl/_.*\.c$
> ^tools/libxl/libxlu_cfg_y\.output$
> ^tools/libxl/xl$
> +^tools/libxl/libxl-save-helper$
> ^tools/libxl/testidl$
> ^tools/libxl/testidl\.c$
> ^tools/libxl/tmp\..*$
> +^tools/libxl/.*\.new$
> ^tools/libvchan/vchan-node[12]$
> ^tools/libaio/src/.*\.ol$
> ^tools/libaio/src/.*\.os$
> diff --git a/tools/libxl/Makefile b/tools/libxl/Makefile
> index 1d8b80a..ddc2624 100644
> --- a/tools/libxl/Makefile
> +++ b/tools/libxl/Makefile
> @@ -67,25 +67,30 @@ LIBXL_OBJS = flexarray.o libxl.o libxl_create.o
> libxl_dm.o libxl_pci.o \
> libxl_dom.o libxl_exec.o libxl_xshelp.o
> libxl_device.o \
> libxl_internal.o libxl_utils.o libxl_uuid.o \
> libxl_json.o libxl_aoutils.o \
> - libxl_save_callout.o \
> + libxl_save_callout.o _libxl_save_msgs_callout.o \
> libxl_qmp.o libxl_event.o libxl_fork.o $(LIBXL_OBJS-y)
> LIBXL_OBJS += _libxl_types.o libxl_flask.o _libxl_types_internal.o
>
> $(LIBXL_OBJS): CFLAGS += $(CFLAGS_libxenctrl) $(CFLAGS_libxenguest)
> $(CFLAGS_libxenstore) $(CFLAGS_libblktapctl) -include
> $(XEN_ROOT)/tools/config.h
>
> -AUTOINCS= libxlu_cfg_y.h libxlu_cfg_l.h _libxl_list.h _paths.h
> +AUTOINCS= libxlu_cfg_y.h libxlu_cfg_l.h _libxl_list.h _paths.h \
> + _libxl_save_msgs_callout.h _libxl_save_msgs_helper.h
> AUTOSRCS= libxlu_cfg_y.c libxlu_cfg_l.c
> +AUTOSRCS += _libxl_save_msgs_callout.c _libxl_save_msgs_helper.c
> LIBXLU_OBJS = libxlu_cfg_y.o libxlu_cfg_l.o libxlu_cfg.o \
> libxlu_disk_l.o libxlu_disk.o libxlu_vif.o libxlu_pci.o
> $(LIBXLU_OBJS): CFLAGS += $(CFLAGS_libxenctrl) # For xentoollog.h
>
> -CLIENTS = xl testidl
> +CLIENTS = xl testidl libxl-save-helper
>
> XL_OBJS = xl.o xl_cmdimpl.o xl_cmdtable.o xl_sxp.o
> $(XL_OBJS): CFLAGS += $(CFLAGS_libxenctrl) # For xentoollog.h
> $(XL_OBJS): CFLAGS += $(CFLAGS_libxenlight)
> $(XL_OBJS): CFLAGS += -include $(XEN_ROOT)/tools/config.h # libxl_json.h
> needs it.
>
> +SAVE_HELPER_OBJS = libxl_save_helper.o _libxl_save_msgs_helper.o
> +$(SAVE_HELPER_OBJS): CFLAGS += $(CFLAGS_libxenctrl)
> +
> testidl.o: CFLAGS += $(CFLAGS_libxenctrl) $(CFLAGS_libxenlight)
> testidl.c: libxl_types.idl gentest.py libxl.h $(AUTOINCS)
> $(PYTHON) gentest.py libxl_types.idl testidl.c.new
> @@ -117,6 +122,12 @@ _libxl_list.h:
> $(XEN_INCLUDE)/xen-external/bsd-sys-queue-h-seddery $(XEN_INCLUDE
> perl $^ --prefix=libxl >$@.new
> $(call move-if-changed,$@.new,$@)
>
> +_libxl_save_msgs_helper.c _libxl_save_msgs_callout.c \
> +_libxl_save_msgs_helper.h _libxl_save_msgs_callout.h: \
> + libxl_save_msgs_gen.pl
> + $(PERL) -w $< $@ >$@.new
> + $(call move-if-changed,$@.new,$@)
> +
> libxl.h: _libxl_types.h
> libxl_json.h: _libxl_types_json.h
> libxl_internal.h: _libxl_types_internal.h _paths.h
> @@ -159,6 +170,9 @@ libxlutil.a: $(LIBXLU_OBJS)
> xl: $(XL_OBJS) libxlutil.so libxenlight.so
> $(CC) $(LDFLAGS) -o $@ $(XL_OBJS) libxlutil.so $(LDLIBS_libxenlight)
> $(LDLIBS_libxenctrl) -lyajl $(APPEND_LDFLAGS)
>
> +libxl-save-helper: $(SAVE_HELPER_OBJS) libxenlight.so
> + $(CC) $(LDFLAGS) -o $@ $(SAVE_HELPER_OBJS) $(LDLIBS_libxenctrl)
> $(LDLIBS_libxenguest) $(APPEND_LDFLAGS)
> +
> testidl: testidl.o libxlutil.so libxenlight.so
> $(CC) $(LDFLAGS) -o $@ testidl.o libxlutil.so $(LDLIBS_libxenlight)
> $(LDLIBS_libxenctrl) $(APPEND_LDFLAGS)
>
> @@ -169,7 +183,9 @@ install: all
> $(INSTALL_DIR) $(DESTDIR)$(INCLUDEDIR)
> $(INSTALL_DIR) $(DESTDIR)$(BASH_COMPLETION_DIR)
> $(INSTALL_DIR) $(DESTDIR)$(XEN_RUN_DIR)
> + $(INSTALL_DIR) $(DESTDIR)$(PRIVATE_BINDIR)
> $(INSTALL_PROG) xl $(DESTDIR)$(SBINDIR)
> + $(INSTALL_PROG) libxl-save-helper $(DESTDIR)$(PRIVATE_BINDIR)
> $(INSTALL_PROG) libxenlight.so.$(MAJOR).$(MINOR) $(DESTDIR)$(LIBDIR)
> ln -sf libxenlight.so.$(MAJOR).$(MINOR)
> $(DESTDIR)$(LIBDIR)/libxenlight.so.$(MAJOR)
> ln -sf libxenlight.so.$(MAJOR) $(DESTDIR)$(LIBDIR)/libxenlight.so
> diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
> index 9c3c671..7b92539 100644
> --- a/tools/libxl/libxl_create.c
> +++ b/tools/libxl/libxl_create.c
> @@ -662,7 +662,8 @@ static void domcreate_bootloader_done(libxl__egc *egc,
> libxl_domain_build_info *const info = &d_config->b_info;
> const int restore_fd = dcs->restore_fd;
> libxl__domain_build_state *const state = &dcs->build_state;
> - struct restore_callbacks *const callbacks = &dcs->callbacks;
> + libxl__srm_restore_autogen_callbacks *const callbacks =
> + &dcs->shs.callbacks.restore.a;
>
> if (rc) domcreate_rebuild_done(egc, dcs, rc);
>
> @@ -702,7 +703,6 @@ static void domcreate_bootloader_done(libxl__egc *egc,
> pae = libxl_defbool_val(info->u.hvm.pae);
> no_incr_generationid =
> !libxl_defbool_val(info->u.hvm.incr_generationid);
> callbacks->toolstack_restore = libxl__toolstack_restore;
> - callbacks->data = gc;
> break;
> case LIBXL_DOMAIN_TYPE_PV:
> hvm = 0;
> @@ -722,10 +722,24 @@ static void domcreate_bootloader_done(libxl__egc *egc,
> libxl__xc_domain_restore_done(egc, dcs, rc, 0, 0);
> }
>
> -void libxl__xc_domain_restore_done(libxl__egc *egc,
> - libxl__domain_create_state *dcs,
> +void libxl__srm_callout_callback_restore_results(unsigned long store_mfn,
> + unsigned long console_mfn, unsigned long genidad, void *user)
> +{
> + libxl__save_helper_state *shs = user;
> + libxl__domain_create_state *dcs = CONTAINER_OF(shs, *dcs, shs);
> + STATE_AO_GC(dcs->ao);
> + libxl__domain_build_state *const state = &dcs->build_state;
> +
> + state->store_mfn = store_mfn;
> + state->console_mfn = console_mfn;
> + state->vm_generationid_addr = genidad;
> + shs->need_results = 0;
> +}
> +
> +void libxl__xc_domain_restore_done(libxl__egc *egc, void *dcs_void,
> int ret, int retval, int errnoval)
> {
> + libxl__domain_create_state *dcs = dcs_void;
> STATE_AO_GC(dcs->ao);
> libxl_ctx *ctx = libxl__gc_owner(gc);
> char **vments = NULL, **localents = NULL;
> diff --git a/tools/libxl/libxl_dom.c b/tools/libxl/libxl_dom.c
> index c44dec0..0e0dbee 100644
> --- a/tools/libxl/libxl_dom.c
> +++ b/tools/libxl/libxl_dom.c
> @@ -467,16 +467,20 @@ static inline char *restore_helper(libxl__gc *gc,
> uint32_t domid,
> }
>
> int libxl__toolstack_restore(uint32_t domid, const uint8_t *buf,
> - uint32_t size, void *data)
> + uint32_t size, void *user)
> {
> - libxl__gc *gc = data;
> - libxl_ctx *ctx = gc->owner;
> + libxl__save_helper_state *shs = user;
> + libxl__domain_create_state *dcs = CONTAINER_OF(shs, *dcs, shs);
> + STATE_AO_GC(dcs->ao);
> + libxl_ctx *ctx = CTX;
> int i, ret;
> const uint8_t *ptr = buf;
> uint32_t count = 0, version = 0;
> struct libxl__physmap_info* pi;
> char *xs_path;
>
> + LOG(DEBUG,"domain=%"PRIu32" toolstack data size=%"PRIu32, domid, size);
> +
> if (size < sizeof(version) + sizeof(count)) {
> LIBXL__LOG(ctx, LIBXL__LOG_ERROR, "wrong size");
> return -1;
> @@ -529,9 +533,10 @@ static void domain_suspend_done(libxl__egc *egc,
> /*----- callbacks, called by xc_domain_save -----*/
>
> int libxl__domain_suspend_common_switch_qemu_logdirty
> - (int domid, unsigned int enable, void *data)
> + (int domid, unsigned enable, void *user)
> {
> - libxl__domain_suspend_state *dss = data;
> + libxl__save_helper_state *shs = user;
> + libxl__domain_suspend_state *dss = CONTAINER_OF(shs, *dss, shs);
> STATE_AO_GC(dss->ao);
> char *path;
> bool rc;
> @@ -597,9 +602,10 @@ int libxl__domain_resume_device_model(libxl__gc *gc,
> uint32_t domid)
> return 0;
> }
>
> -int libxl__domain_suspend_common_callback(void *data)
> +int libxl__domain_suspend_common_callback(void *user)
> {
> - libxl__domain_suspend_state *dss = data;
> + libxl__save_helper_state *shs = user;
> + libxl__domain_suspend_state *dss = CONTAINER_OF(shs, *dss, shs);
> STATE_AO_GC(dss->ao);
> unsigned long hvm_s_state = 0, hvm_pvdrv = 0;
> int ret;
> @@ -739,9 +745,9 @@ static inline char *save_helper(libxl__gc *gc, uint32_t
> domid,
> }
>
> int libxl__toolstack_save(uint32_t domid, uint8_t **buf,
> - uint32_t *len, void *data)
> + uint32_t *len, void *dss_void)
> {
> - libxl__domain_suspend_state *dss = data;
> + libxl__domain_suspend_state *dss = dss_void;
> STATE_AO_GC(dss->ao);
> int i = 0;
> char *start_addr = NULL, *size = NULL, *phys_offset = NULL, *name = NULL;
> @@ -810,6 +816,8 @@ int libxl__toolstack_save(uint32_t domid, uint8_t **buf,
> ptr += sizeof(struct libxl__physmap_info) + namelen;
> }
>
> + LOG(DEBUG,"domain=%"PRIu32" toolstack data size=%"PRIu32, domid, *len);
> +
> return 0;
> }
>
> @@ -823,7 +831,8 @@ static int libxl__remus_domain_suspend_callback(void
> *data)
>
> static int libxl__remus_domain_resume_callback(void *data)
> {
> - libxl__domain_suspend_state *dss = data;
> + libxl__save_helper_state *shs = data;
> + libxl__domain_suspend_state *dss = CONTAINER_OF(shs, *dss, shs);
> STATE_AO_GC(dss->ao);
>
> /* Resumes the domain and the device model */
> @@ -836,7 +845,8 @@ static int libxl__remus_domain_resume_callback(void *data)
>
> static int libxl__remus_domain_checkpoint_callback(void *data)
> {
> - libxl__domain_suspend_state *dss = data;
> + libxl__save_helper_state *shs = data;
> + libxl__domain_suspend_state *dss = CONTAINER_OF(shs, *dss, shs);
> STATE_AO_GC(dss->ao);
>
> /* This would go into tailbuf. */
> @@ -864,7 +874,8 @@ void libxl__domain_suspend(libxl__egc *egc,
> libxl__domain_suspend_state *dss)
> const int live = dss->live;
> const int debug = dss->debug;
> const libxl_domain_remus_info *const r_info = dss->remus;
> - struct save_callbacks *const callbacks = &dss->callbacks;
> + libxl__srm_save_autogen_callbacks *const callbacks =
> + &dss->shs.callbacks.save.a;
>
> switch (type) {
> case LIBXL_DOMAIN_TYPE_HVM: {
> @@ -925,8 +936,7 @@ void libxl__domain_suspend(libxl__egc *egc,
> libxl__domain_suspend_state *dss)
> callbacks->suspend = libxl__domain_suspend_common_callback;
>
> callbacks->switch_qemu_logdirty =
> libxl__domain_suspend_common_switch_qemu_logdirty;
> - callbacks->toolstack_save = libxl__toolstack_save;
> - callbacks->data = dss;
> + dss->shs.callbacks.save.toolstack_save = libxl__toolstack_save;
>
> libxl__xc_domain_save(egc, dss, vm_generationid_addr);
> return;
> @@ -935,10 +945,10 @@ void libxl__domain_suspend(libxl__egc *egc,
> libxl__domain_suspend_state *dss)
> domain_suspend_done(egc, dss, rc);
> }
>
> -void libxl__xc_domain_save_done(libxl__egc *egc,
> - libxl__domain_suspend_state *dss,
> +void libxl__xc_domain_save_done(libxl__egc *egc, void *dss_void,
> int rc, int retval, int errnoval)
> {
> + libxl__domain_suspend_state *dss = dss_void;
> STATE_AO_GC(dss->ao);
>
> /* Convenience aliases */
> diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
> index 7cf1b04..1a7b526 100644
> --- a/tools/libxl/libxl_internal.h
> +++ b/tools/libxl/libxl_internal.h
> @@ -54,6 +54,7 @@
>
> #include "libxl.h"
> #include "_paths.h"
> +#include "_libxl_save_msgs_callout.h"
>
> #if __GNUC__ > 3 || (__GNUC__ == 3 && __GNUC_MINOR__ >= 1)
> #define _hidden __attribute__((visibility("hidden")))
> @@ -1773,6 +1774,51 @@ _hidden void
> libxl__datacopier_kill(libxl__datacopier_state *dc);
> _hidden int libxl__datacopier_start(libxl__datacopier_state *dc);
>
>
> +/*----- Save/restore helper (used by creation and suspend) -----*/
> +
> +typedef struct libxl__srm_save_callbacks {
> + libxl__srm_save_autogen_callbacks a;
> + int (*toolstack_save)(uint32_t domid, uint8_t **buf,
> + uint32_t *len, void *data);
> +} libxl__srm_save_callbacks;
> +
> +typedef struct libxl__srm_restore_callbacks {
> + libxl__srm_restore_autogen_callbacks a;
> +} libxl__srm_restore_callbacks;
> +
> +/* a pointer to this struct is also passed as "user" to the
> + * save callout helper callback functions */
> +typedef struct libxl__save_helper_state {
> + /* public, caller of run_helper initialises */
> + libxl__ao *ao;
> + uint32_t domid;
> + union {
> + libxl__srm_save_callbacks save;
> + libxl__srm_restore_callbacks restore;
> + } callbacks;
> + int (*recv_callback)(const unsigned char *msg, uint32_t len, void *user);
> + void (*completion_callback)(libxl__egc *egc, void *caller_state,
> + int rc, int retval, int errnoval);
> + void *caller_state;
> + int need_results; /* set to 0 or 1 by caller of run_helper;
> + * if set to 1 then the ultimate caller's
> + * results function must set it to 0 */
> + /* private */
> + int rc;
> + int completed; /* retval/errnoval valid iff completed */
> + int retval, errnoval; /* from xc_domain_save / xc_domain_restore */
> + libxl__carefd *pipes[2]; /* 0 = helper's stdin, 1 = helper's stdout */
> + libxl__ev_fd readable;
> + libxl__ev_child child;
> + const char *stdin_what, *stdout_what;
> + FILE *toolstack_data_file;
> +
> + libxl__egc *egc; /* valid only for duration of each event callback;
> + * is here in this struct for the benefit of the
> + * marshalling and xc callback functions */
> +} libxl__save_helper_state;
> +
> +
> /*----- Domain suspend (save) state structure -----*/
>
> typedef struct libxl__domain_suspend_state libxl__domain_suspend_state;
> @@ -1798,7 +1844,7 @@ struct libxl__domain_suspend_state {
> int xcflags;
> int guest_responded;
> int interval; /* checkpoint interval (for Remus) */
> - struct save_callbacks callbacks;
> + libxl__save_helper_state shs;
> };
>
>
> @@ -1910,7 +1956,7 @@ struct libxl__domain_create_state {
> libxl__stub_dm_spawn_state dmss;
> /* If we're not doing stubdom, we use only dmss.dm,
> * for the non-stubdom device model. */
> - struct restore_callbacks callbacks;
> + libxl__save_helper_state shs;
> };
>
> /*----- Domain suspend (save) functions -----*/
> @@ -1926,8 +1972,7 @@ _hidden void libxl__xc_domain_save(libxl__egc*,
> libxl__domain_suspend_state*,
> /* If rc==0 then retval is the return value from xc_domain_save
> * and errnoval is the errno value it provided.
> * If rc!=0, retval and errnoval are undefined. */
> -_hidden void libxl__xc_domain_save_done(libxl__egc*,
> - libxl__domain_suspend_state*,
> +_hidden void libxl__xc_domain_save_done(libxl__egc*, void *dss_void,
> int rc, int retval, int errnoval);
>
> _hidden int libxl__domain_suspend_common_callback(void *data);
> @@ -1945,8 +1990,7 @@ _hidden void libxl__xc_domain_restore(libxl__egc *egc,
> /* If rc==0 then retval is the return value from xc_domain_save
> * and errnoval is the errno value it provided.
> * If rc!=0, retval and errnoval are undefined. */
> -_hidden void libxl__xc_domain_restore_done(libxl__egc *egc,
> - libxl__domain_create_state *dcs,
> +_hidden void libxl__xc_domain_restore_done(libxl__egc *egc, void *dcs_void,
> int rc, int retval, int errnoval);
>
>
> diff --git a/tools/libxl/libxl_save_callout.c
> b/tools/libxl/libxl_save_callout.c
> index 1b481ab..a6abcda 100644
> --- a/tools/libxl/libxl_save_callout.c
> +++ b/tools/libxl/libxl_save_callout.c
> @@ -16,6 +16,30 @@
>
> #include "libxl_internal.h"
>
> +/* stream_fd is as from the caller (eventually, the application).
> + * It may be 0, 1 or 2, in which case we need to dup it elsewhere.
> + * The actual fd value is not included in the supplied argnums; rather
> + * it will be automatically supplied by run_helper as the 2nd argument.
> + *
> + * preserve_fds are fds that the caller is intending to pass to the
> + * helper so which need cloexec clearing. They may not be 0, 1 or 2.
> + * An entry may be -1 in which case it will be ignored.
> + */
> +static void run_helper(libxl__egc *egc, libxl__save_helper_state *shs,
> + const char *mode_arg,
> + int stream_fd,
> + const int *preserve_fds, int num_preserve_fds,
> + const unsigned long *argnums, int num_argnums);
> +
> +static void helper_failed(libxl__egc*, libxl__save_helper_state *shs, int
> rc);
> +static void helper_stdout_readable(libxl__egc *egc, libxl__ev_fd *ev,
> + int fd, short events, short revents);
> +static void helper_exited(libxl__egc *egc, libxl__ev_child *ch,
> + pid_t pid, int status);
> +static void helper_done(libxl__egc *egc, libxl__save_helper_state *shs);
> +
> +/*----- entrypoints -----*/
> +
> void libxl__xc_domain_restore(libxl__egc *egc, libxl__domain_create_state
> *dcs,
> int hvm, int pae, int superpages,
> int no_incr_generationid)
> @@ -27,22 +51,337 @@ void libxl__xc_domain_restore(libxl__egc *egc,
> libxl__domain_create_state *dcs,
> const int restore_fd = dcs->restore_fd;
> libxl__domain_build_state *const state = &dcs->build_state;
>
> - int r = xc_domain_restore(CTX->xch, restore_fd, domid,
> - state->store_port, &state->store_mfn,
> - state->store_domid, state->console_port,
> - &state->console_mfn, state->console_domid,
> - hvm, pae, superpages, no_incr_generationid,
> - &state->vm_generationid_addr, &dcs->callbacks);
> - libxl__xc_domain_restore_done(egc, dcs, 0, r, errno);
> + unsigned cbflags = libxl__srm_callout_enumcallbacks_restore
> + (&dcs->shs.callbacks.restore.a);
> +
> + const unsigned long argnums[] = {
> + domid,
> + state->store_port,
> + state->store_domid, state->console_port,
> + state->console_domid,
> + hvm, pae, superpages, no_incr_generationid,
> + cbflags,
> + };
> +
> + dcs->shs.ao = ao;
> + dcs->shs.domid = domid;
> + dcs->shs.recv_callback = libxl__srm_callout_received_restore;
> + dcs->shs.completion_callback = libxl__xc_domain_restore_done;
> + dcs->shs.caller_state = dcs;
> + dcs->shs.need_results = 1;
> + dcs->shs.toolstack_data_file = 0;
> +
> + run_helper(egc, &dcs->shs, "--restore-domain", restore_fd, 0,0,
> + argnums, ARRAY_SIZE(argnums));
> }
>
> void libxl__xc_domain_save(libxl__egc *egc, libxl__domain_suspend_state *dss,
> unsigned long vm_generationid_addr)
> {
> STATE_AO_GC(dss->ao);
> - int r;
> + int r, rc, toolstack_data_fd = -1;
> + uint32_t toolstack_data_len = 0;
> +
> + /* Resources we need to free */
> + uint8_t *toolstack_data_buf = 0;
> +
> + unsigned cbflags = libxl__srm_callout_enumcallbacks_save
> + (&dss->shs.callbacks.save.a);
> +
> + if (dss->shs.callbacks.save.toolstack_save) {
> + r = dss->shs.callbacks.save.toolstack_save
> + (dss->domid, &toolstack_data_buf, &toolstack_data_len, dss);
> + if (r) { rc = ERROR_FAIL; goto out; }
> +
> + dss->shs.toolstack_data_file = tmpfile();
> + if (!dss->shs.toolstack_data_file) {
> + LOGE(ERROR, "cannot create toolstack data tmpfile");
> + rc = ERROR_FAIL;
> + goto out;
> + }
> + toolstack_data_fd = fileno(dss->shs.toolstack_data_file);
> +
> + r = libxl_write_exactly(CTX, toolstack_data_fd,
> + toolstack_data_buf, toolstack_data_len,
> + "toolstack data tmpfile", 0);
> + if (r) { rc = ERROR_FAIL; goto out; }
> + }
> +
> + const unsigned long argnums[] = {
> + dss->domid, 0, 0, dss->xcflags, dss->hvm, vm_generationid_addr,
> + toolstack_data_fd, toolstack_data_len,
> + cbflags,
> + };
> +
> + dss->shs.ao = ao;
> + dss->shs.domid = dss->domid;
> + dss->shs.recv_callback = libxl__srm_callout_received_save;
> + dss->shs.completion_callback = libxl__xc_domain_save_done;
> + dss->shs.caller_state = dss;
> + dss->shs.need_results = 0;
> +
> + free(toolstack_data_buf);
> +
> + run_helper(egc, &dss->shs, "--save-domain", dss->fd,
> + &toolstack_data_fd, 1,
> + argnums, ARRAY_SIZE(argnums));
> + return;
> +
> + out:
> + free(toolstack_data_buf);
> + if (dss->shs.toolstack_data_file) fclose(dss->shs.toolstack_data_file);
> +
> + libxl__xc_domain_save_done(egc, dss, rc, 0, 0);
> +}
> +
> +
> +/*----- helper execution -----*/
> +
> +static void run_helper(libxl__egc *egc, libxl__save_helper_state *shs,
> + const char *mode_arg, int stream_fd,
> + const int *preserve_fds, int num_preserve_fds,
> + const unsigned long *argnums, int num_argnums)
> +{
> + STATE_AO_GC(shs->ao);
> + const char *args[4 + num_argnums];
> + const char **arg = args;
> + int i, rc;
> +
> + /* Resources we must free */
> + libxl__carefd *childs_pipes[2] = { 0,0 };
> +
> + /* Convenience aliases */
> + const uint32_t domid = shs->domid;
> +
> + shs->rc = 0;
> + shs->completed = 0;
> + shs->pipes[0] = shs->pipes[1] = 0;
> + libxl__ev_fd_init(&shs->readable);
> + libxl__ev_child_init(&shs->child);
> +
> + shs->stdin_what = GCSPRINTF("domain %"PRIu32" save/restore helper"
> + " stdin pipe", domid);
> + shs->stdout_what = GCSPRINTF("domain %"PRIu32" save/restore helper"
> + " stdout pipe", domid);
> +
> + *arg++ = getenv("LIBXL_SAVE_HELPER") ?: LIBEXEC "/" "libxl-save-helper";
> + *arg++ = mode_arg;
> + const char **stream_fd_arg = arg++;
> + for (i=0; i<num_argnums; i++)
> + *arg++ = GCSPRINTF("%lu", argnums[i]);
> + *arg++ = 0;
> + assert(arg == args + ARRAY_SIZE(args));
> +
> + libxl__carefd_begin();
> + int childfd;
> + for (childfd=0; childfd<2; childfd++) {
> + /* Setting up the pipe for the child's fd childfd */
> + int fds[2];
> + if (libxl_pipe(CTX,fds)) { rc = ERROR_FAIL; goto out; }
> + int childs_end = childfd==0 ? 0 /*read*/ : 1 /*write*/;
> + int our_end = childfd==0 ? 1 /*write*/ : 0 /*read*/;
> + childs_pipes[childfd] = libxl__carefd_record(CTX, fds[childs_end]);
> + shs->pipes[childfd] = libxl__carefd_record(CTX, fds[our_end]);
> + }
> + libxl__carefd_unlock();
> +
> + pid_t pid = libxl__ev_child_fork(gc, &shs->child, helper_exited);
> + if (!pid) {
> + if (stream_fd <= 2) {
> + stream_fd = dup(stream_fd);
> + if (stream_fd < 0) {
> + LOGE(ERROR,"dup migration stream fd");
> + exit(-1);
> + }
> + }
> + libxl_fd_set_cloexec(CTX, stream_fd, 0);
> + *stream_fd_arg = GCSPRINTF("%d", stream_fd);
> +
> + for (i=0; i<num_preserve_fds; i++)
> + if (preserve_fds[i] >= 0) {
> + assert(preserve_fds[i] > 2);
> + libxl_fd_set_cloexec(CTX, preserve_fds[i], 0);
> + }
> +
> + libxl__exec(gc,
> + libxl__carefd_fd(childs_pipes[0]),
> + libxl__carefd_fd(childs_pipes[1]),
> + -1,
> + args[0], (char**)args, 0);
> + }
> +
> + libxl__carefd_close(childs_pipes[0]);
> + libxl__carefd_close(childs_pipes[1]);
> +
> + rc = libxl__ev_fd_register(gc, &shs->readable, helper_stdout_readable,
> + libxl__carefd_fd(shs->pipes[1]),
> POLLIN|POLLPRI);
> + if (rc) goto out;
> + return;
> +
> + out:
> + libxl__carefd_close(childs_pipes[0]);
> + libxl__carefd_close(childs_pipes[1]);
> + helper_failed(egc, shs, rc);;
> +}
> +
> +static void helper_failed(libxl__egc *egc, libxl__save_helper_state *shs,
> + int rc)
> +{
> + STATE_AO_GC(shs->ao);
> +
> + if (!shs->rc)
> + shs->rc = rc;
> +
> + libxl__ev_fd_deregister(gc, &shs->readable);
> +
> + if (!libxl__ev_child_inuse(&shs->child)) {
> + helper_done(egc, shs);
> + return;
> + }
> +
> + int r = kill(shs->child.pid, SIGKILL);
> + if (r) LOGE(WARN, "failed to kill save/restore helper [%lu]",
> + (unsigned long)shs->child.pid);
> +}
> +
> +static void helper_stdout_readable(libxl__egc *egc, libxl__ev_fd *ev,
> + int fd, short events, short revents)
> +{
> + libxl__save_helper_state *shs = CONTAINER_OF(ev, *shs, readable);
> + STATE_AO_GC(shs->ao);
> + int rc, errnoval;
> +
> + if (revents & (POLLERR|POLLPRI)) {
> + LOG(ERROR, "%s signaled POLLERR|POLLPRI (%#x)",
> + shs->stdout_what, revents);
> + rc = ERROR_FAIL;
> + out:
> + /* this is here because otherwise we bypass the decl of msg[] */
> + helper_failed(egc, shs, rc);
> + return;
> + }
> +
> + uint16_t msglen;
> + errnoval = libxl_read_exactly(CTX, fd, &msglen, sizeof(msglen),
> + shs->stdout_what, "ipc msg header");
> + if (errnoval) { rc = ERROR_FAIL; goto out; }
> +
> + unsigned char msg[msglen];
> + errnoval = libxl_read_exactly(CTX, fd, msg, msglen,
> + shs->stdout_what, "ipc msg body");
> + if (errnoval) { rc = ERROR_FAIL; goto out; }
> +
> + shs->egc = egc;
> + shs->recv_callback(msg, msglen, shs);
> + shs->egc = 0;
> + return;
> +}
> +
> +static void helper_exited(libxl__egc *egc, libxl__ev_child *ch,
> + pid_t pid, int status)
> +{
> + libxl__save_helper_state *shs = CONTAINER_OF(ch, *shs, child);
> + STATE_AO_GC(shs->ao);
> +
> + /* Convenience aliases */
> + const uint32_t domid = shs->domid;
> +
> + const char *what =
> + GCSPRINTF("domain %"PRIu32" save/restore helper", domid);
> +
> + if (status) {
> + libxl_report_child_exitstatus(CTX, XTL_ERROR, what, pid, status);
> + shs->rc = ERROR_FAIL;
> + }
> +
> + if (shs->need_results) {
> + if (!shs->rc)
> + LOG(ERROR,"%s exited without providing results",what);
> + shs->rc = ERROR_FAIL;
> + }
> +
> + if (!shs->completed) {
> + if (!shs->rc)
> + LOG(ERROR,"%s exited without signaling completion",what);
> + shs->rc = ERROR_FAIL;
> + }
> +
> + helper_done(egc, shs);
> + return;
> +}
> +
> +static void helper_done(libxl__egc *egc, libxl__save_helper_state *shs)
> +{
> + STATE_AO_GC(shs->ao);
> +
> + libxl__ev_fd_deregister(gc, &shs->readable);
> + libxl__carefd_close(shs->pipes[0]); shs->pipes[0] = 0;
> + libxl__carefd_close(shs->pipes[1]); shs->pipes[1] = 0;
> + assert(!libxl__ev_child_inuse(&shs->child));
> + if (shs->toolstack_data_file) fclose(shs->toolstack_data_file);
> +
> + shs->egc = egc;
> + shs->completion_callback(egc, shs->caller_state,
> + shs->rc, shs->retval, shs->errnoval);
> + shs->egc = 0;
> +}
> +
> +/*----- generic helpers for the autogenerated code -----*/
> +
> +const libxl__srm_save_autogen_callbacks*
> +libxl__srm_callout_get_callbacks_save(void *user)
> +{
> + libxl__save_helper_state *shs = user;
> + return &shs->callbacks.save.a;
> +}
> +
> +const libxl__srm_restore_autogen_callbacks*
> +libxl__srm_callout_get_callbacks_restore(void *user)
> +{
> + libxl__save_helper_state *shs = user;
> + return &shs->callbacks.restore.a;
> +}
> +
> +void libxl__srm_callout_sendreply(int r, void *user)
> +{
> + libxl__save_helper_state *shs = user;
> + libxl__egc *egc = shs->egc;
> + STATE_AO_GC(shs->ao);
> + int errnoval;
> +
> + errnoval = libxl_write_exactly(CTX, libxl__carefd_fd(shs->pipes[0]),
> + &r, sizeof(r), shs->stdin_what,
> + "callback return value");
> + if (errnoval)
> + helper_failed(egc, shs, ERROR_FAIL);
> +}
> +
> +void libxl__srm_callout_callback_log(uint32_t level, uint32_t errnoval,
> + const char *context, const char *formatted, void *user)
> +{
> + libxl__save_helper_state *shs = user;
> + STATE_AO_GC(shs->ao);
> + xtl_log(CTX->lg, level, errnoval, context, "%s", formatted);
> +}
> +
> +void libxl__srm_callout_callback_progress(const char *context,
> + const char *doing_what, unsigned long done,
> + unsigned long total, void *user)
> +{
> + libxl__save_helper_state *shs = user;
> + STATE_AO_GC(shs->ao);
> + xtl_progress(CTX->lg, context, doing_what, done, total);
> +}
> +
> +int libxl__srm_callout_callback_complete(int retval, int errnoval,
> + void *user)
> +{
> + libxl__save_helper_state *shs = user;
> + STATE_AO_GC(shs->ao);
>
> - r = xc_domain_save(CTX->xch, dss->fd, dss->domid, 0, 0, dss->xcflags,
> - &dss->callbacks, dss->hvm, vm_generationid_addr);
> - libxl__xc_domain_save_done(egc, dss, 0, r, errno);
> + shs->completed = 1;
> + shs->retval = retval;
> + shs->errnoval = errnoval;
> + libxl__ev_fd_deregister(gc, &shs->readable);
> + return 0;
> }
> diff --git a/tools/libxl/libxl_save_helper.c b/tools/libxl/libxl_save_helper.c
> new file mode 100644
> index 0000000..772251a
> --- /dev/null
> +++ b/tools/libxl/libxl_save_helper.c
> @@ -0,0 +1,283 @@
> +/*
> + * Copyright (C) 2012 Citrix Ltd.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU Lesser General Public License as published
> + * by the Free Software Foundation; version 2.1 only. with the special
> + * exception on linking described in file LICENSE.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> + * GNU Lesser General Public License for more details.
> + */
> +
> +/*
> + * The libxl-save-helper utility speaks a protocol to its caller for
> + * the callbacks. The protocol is as follows.
> + *
> + * The helper talks on stdin and stdout, in binary in machine
> + * endianness. The helper speaks first, and only when it has a
> + * callback to make. It writes a 16-bit number being the message
> + * length, and then the message body.
> + *
> + * Each message starts with a 16-bit number indicating which of the
> + * messages it is, and then some arguments in a binary marshalled form.
> + * If the callback does not need a reply (it returns void), the helper
> + * just continues. Otherwise the helper waits for its caller to send a
> + * single int which is to be the return value from the callback.
> + *
> + * Where feasible the stubs and callbacks have prototypes identical to
> + * those required by xc_domain_save and xc_domain_restore, so that the
> + * autogenerated functions can be used/provided directly.
> + *
> + * The actual messages are in the array @msgs in libxl_save_msgs_gen.pl
> + */
> +
> +#include "libxl_osdeps.h"
> +
> +#include <stdlib.h>
> +#include <unistd.h>
> +#include <assert.h>
> +#include <inttypes.h>
> +
> +#include "libxl.h"
> +
> +#include "xenctrl.h"
> +#include "xenguest.h"
> +#include "_libxl_save_msgs_helper.h"
> +
> +/*----- globals -----*/
> +
> +static const char *program = "libxl-save-helper";
> +static xentoollog_logger *logger;
> +static xc_interface *xch;
> +
> +/*----- error handling -----*/
> +
> +static void fail(int errnoval, const char *fmt, ...)
> + __attribute__((noreturn,format(printf,2,3)));
> +static void fail(int errnoval, const char *fmt, ...)
> +{
> + va_list al;
> + va_start(al,fmt);
> + xtl_logv(logger,XTL_ERROR,errnoval,program,fmt,al);
> + exit(-1);
> +}
> +
> +static int read_exactly(int fd, void *buf, size_t len)
> +/* returns 0 if we get eof, even if we got it midway through; 1 if ok */
> +{
> + while (len) {
> + ssize_t r = read(fd, buf, len);
> + if (r<=0) return r;
> + assert(r <= len);
> + len -= r;
> + buf = (char*)buf + r;
> + }
> + return 1;
> +}
> +
> +static void *xmalloc(size_t sz)
> +{
> + if (!sz) return 0;
> + void *r = malloc(sz);
> + if (!r) { perror("memory allocation failed"); exit(-1); }
> + return r;
> +}
> +
> +/*----- logger -----*/
> +
> +typedef struct {
> + xentoollog_logger vtable;
> +} xentoollog_logger_tellparent;
> +
> +static void tellparent_vmessage(xentoollog_logger *logger_in,
> + xentoollog_level level,
> + int errnoval,
> + const char *context,
> + const char *format,
> + va_list al)
> +{
> + char *formatted;
> + int r = vasprintf(&formatted, format, al);
> + if (r < 0) { perror("memory allocation failed during logging");
> exit(-1); }
> + helper_stub_log(level, errnoval, context, formatted, 0);
> + free(formatted);
> +}
> +
> +static void tellparent_progress(struct xentoollog_logger *logger_in,
> + const char *context,
> + const char *doing_what, int percent,
> + unsigned long done, unsigned long total)
> +{
> + helper_stub_progress(context, doing_what, done, total, 0);
> +}
> +
> +static void tellparent_destroy(struct xentoollog_logger *logger_in)
> +{
> + abort();
> +}
> +
> +static xentoollog_logger_tellparent *createlogger_tellparent(void)
> +{
> + xentoollog_logger_tellparent newlogger;
> + return XTL_NEW_LOGGER(tellparent, newlogger);
> +}
> +
> +/*----- helper functions called by autogenerated stubs -----*/
> +
> +unsigned char * helper_allocbuf(int len, void *user)
> +{
> + return xmalloc(len);
> +}
> +
> +static void transmit(const unsigned char *msg, int len, void *user)
> +{
> + while (len) {
> + int r = write(1, msg, len);
> + if (r<0) { perror("write"); exit(-1); }
> + assert(r >= 0);
> + assert(r <= len);
> + len -= r;
> + msg += r;
> + }
> +}
> +
> +void helper_transmitmsg(unsigned char *msg_freed, int len_in, void *user)
> +{
> + assert(len_in < 64*1024);
> + uint16_t len = len_in;
> + transmit((const void*)&len, sizeof(len), user);
> + transmit(msg_freed, len, user);
> + free(msg_freed);
> +}
> +
> +int helper_getreply(void *user)
> +{
> + int v;
> + int r = read_exactly(0, &v, sizeof(v));
> + if (r<=0) exit(-2);
> + return v;
> +}
> +
> +/*----- other callbacks -----*/
> +
> +static int toolstack_save_fd;
> +static uint32_t toolstack_save_len;
> +
> +static int toolstack_save_cb(uint32_t domid, uint8_t **buf,
> + uint32_t *len, void *data)
> +{
> + assert(toolstack_save_fd > 0);
> +
> + int r = lseek(toolstack_save_fd, 0, SEEK_SET);
> + if (r) fail(errno,"rewind toolstack data tmpfile");
> +
> + *buf = xmalloc(toolstack_save_len);
> + r = read_exactly(toolstack_save_fd, *buf, toolstack_save_len);
> + if (r<0) fail(errno,"read toolstack data");
> + if (r==0) fail(0,"read toolstack data eof");
> +
> + *len = toolstack_save_len;
> + return 0;
> +}
> +
> +static void startup(const char *op) {
> + logger = (xentoollog_logger*)createlogger_tellparent();
> + if (!logger) {
> + fprintf(stderr, "%s: cannot initialise logger\n", program);
> + exit(-1);
> + }
> +
> + xtl_log(logger,XTL_DEBUG,0,program,"starting %s",op);
> +
> + xch = xc_interface_open(logger,logger,0);
> + if (!xch) fail(errno,"xc_interface_open failed");
> +}
> +
> +static void complete(int retval) {
> + int errnoval = retval ? errno : 0; /* suppress irrelevant errnos */
> + xtl_log(logger,XTL_DEBUG,errnoval,program,"complete r=%d",retval);
> + helper_stub_complete(retval,errnoval,0);
> + exit(0);
> +}
> +
> +static struct save_callbacks helper_save_callbacks;
> +static struct restore_callbacks helper_restore_callbacks;
> +
> +int main(int argc, char **argv)
> +{
> + int r;
> +
> +#define NEXTARG (++argv, assert(*argv), *argv)
> +
> + const char *mode = *++argv;
> + assert(mode);
> +
> + if (!strcmp(mode,"--save-domain")) {
> +
> + int io_fd = atoi(NEXTARG);
> + uint32_t dom = strtoul(NEXTARG,0,10);
> + uint32_t max_iters = strtoul(NEXTARG,0,10);
> + uint32_t max_factor = strtoul(NEXTARG,0,10);
> + uint32_t flags = strtoul(NEXTARG,0,10);
> + int hvm = atoi(NEXTARG);
> + unsigned long genidad = strtoul(NEXTARG,0,10);
> + toolstack_save_fd = atoi(NEXTARG);
> + toolstack_save_len = strtoul(NEXTARG,0,10);
> + unsigned cbflags = strtoul(NEXTARG,0,10);
> + assert(!*++argv);
> +
> + if (toolstack_save_fd >= 0)
> + helper_save_callbacks.toolstack_save = toolstack_save_cb;
> +
> + helper_setcallbacks_save(&helper_save_callbacks, cbflags);
> +
> + startup("save");
> + r = xc_domain_save(xch, io_fd, dom, max_iters, max_factor, flags,
> + &helper_save_callbacks, hvm, genidad);
> + complete(r);
> +
> + } else if (!strcmp(mode,"--restore-domain")) {
> +
> + int io_fd = atoi(NEXTARG);
> + uint32_t dom = strtoul(NEXTARG,0,10);
> + unsigned store_evtchn = strtoul(NEXTARG,0,10);
> + domid_t store_domid = strtoul(NEXTARG,0,10);
> + unsigned console_evtchn = strtoul(NEXTARG,0,10);
> + domid_t console_domid = strtoul(NEXTARG,0,10);
> + unsigned int hvm = strtoul(NEXTARG,0,10);
> + unsigned int pae = strtoul(NEXTARG,0,10);
> + int superpages = strtoul(NEXTARG,0,10);
> + int no_incr_genidad = strtoul(NEXTARG,0,10);
> + unsigned cbflags = strtoul(NEXTARG,0,10);
> + assert(!*++argv);
> +
> + helper_setcallbacks_restore(&helper_restore_callbacks, cbflags);
> +
> + unsigned long store_mfn = 0;
> + unsigned long console_mfn = 0;
> + unsigned long genidad = 0;
> +
> + startup("restore");
> + r = xc_domain_restore(xch, io_fd, dom, store_evtchn, &store_mfn,
> + store_domid, console_evtchn, &console_mfn,
> + console_domid, hvm, pae, superpages,
> + no_incr_genidad, &genidad,
> + &helper_restore_callbacks);
> + helper_stub_restore_results(store_mfn,console_mfn,genidad,0);
> + complete(r);
> +
> + } else {
> + assert(!"unexpected mode argument");
> + }
> +}
> +
> +/*
> + * Local variables:
> + * mode: C
> + * c-basic-offset: 4
> + * indent-tabs-mode: nil
> + * End:
> + */
> diff --git a/tools/libxl/libxl_save_msgs_gen.pl
> b/tools/libxl/libxl_save_msgs_gen.pl
> new file mode 100755
> index 0000000..c45986e
> --- /dev/null
> +++ b/tools/libxl/libxl_save_msgs_gen.pl
> @@ -0,0 +1,397 @@
> +#!/usr/bin/perl -w
> +
> +use warnings;
> +use strict;
> +use POSIX;
> +
> +our $debug = 0; # produce copious debugging output at run-time?
> +
> +our @msgs = (
> + # flags:
> + # s - applicable to save
> + # r - applicable to restore
> + # c - function pointer in callbacks struct rather than fixed function
> + # x - function pointer is in struct {save,restore}_callbacks
> + # and its null-ness needs to be passed through to the helper's xc
> + # W - needs a return value; callback is synchronous
> + [ 1, 'sr', "log", [qw(uint32_t level
> + uint32_t errnoval
> + STRING context
> + STRING formatted)] ],
> + [ 2, 'sr', "progress", [qw(STRING context
> + STRING doing_what),
> + 'unsigned long', 'done',
> + 'unsigned long', 'total'] ],
> + [ 3, 'scxW', "suspend", [] ],
> + [ 4, 'scxW', "postcopy", [] ],
> + [ 5, 'scxW', "checkpoint", [] ],
> + [ 6, 'scxW', "switch_qemu_logdirty", [qw(int domid
> + unsigned enable)] ],
> + # toolstack_save done entirely `by hand'
> + [ 7, 'rcxW', "toolstack_restore", [qw(uint32_t domid
> + BLOCK tsdata)] ],
> + [ 8, 'r', "restore_results", ['unsigned long', 'store_mfn',
> + 'unsigned long', 'console_mfn',
> + 'unsigned long', 'genidad'] ],
> + [ 9, 'srW', "complete", [qw(int retval
> + int errnoval)] ],
> +);
> +
> +#----------------------------------------
> +
> +our %cbs;
> +our %func;
> +our %func_ah;
> +our @outfuncs;
> +our %out_decls;
> +our %out_body;
> +our %msgnum_used;
> +
> +die unless @ARGV==1;
> +die if $ARGV[0] =~ m/^-/;
> +
> +our ($intendedout) = @ARGV;
> +
> +$intendedout =~ m/([a-z]+)\.([ch])$/ or die;
> +my ($want_ah, $ch) = ($1, $2);
> +
> +my $declprefix = '';
> +
> +foreach my $ah (qw(callout helper)) {
> + $out_body{$ah} .=
> + <<END_BOTH.($ah eq 'callout' ? <<END_CALLOUT : <<END_HELPER);
> +#include "libxl_osdeps.h"
> +
> +#include <assert.h>
> +#include <string.h>
> +#include <stdint.h>
> +#include <limits.h>
> +END_BOTH
> +
> +#include "libxl_internal.h"
> +
> +END_CALLOUT
> +
> +#include "_libxl_save_msgs_${ah}.h"
> +#include <xenctrl.h>
> +#include <xenguest.h>
> +
> +END_HELPER
> +}
> +
> +die $want_ah unless defined $out_body{$want_ah};
> +
> +sub f_decl ($$$$) {
> + my ($name, $ah, $c_rtype, $c_decl) = @_;
> + $out_decls{$name} = "${declprefix}$c_rtype $name$c_decl;\n";
> + $func{$name} = "$c_rtype $name$c_decl\n{\n" . ($func{$name} || '');
> + $func_ah{$name} = $ah;
> +}
> +
> +sub f_more ($$) {
> + my ($name, $addbody) = @_;
> + $func{$name} ||= '';
> + $func{$name} .= $addbody;
> + push @outfuncs, $name;
> +}
> +
> +our $libxl = "libxl__srm";
> +our $callback = "${libxl}_callout_callback";
> +our $receiveds = "${libxl}_callout_received";
> +our $sendreply = "${libxl}_callout_sendreply";
> +our $getcallbacks = "${libxl}_callout_get_callbacks";
> +our $enumcallbacks = "${libxl}_callout_enumcallbacks";
> +sub cbtype ($) { "${libxl}_".$_[0]."_autogen_callbacks"; };
> +
> +f_decl($sendreply, 'callout', 'void', "(int r, void *user)");
> +
> +our $helper = "helper";
> +our $encode = "${helper}_stub";
> +our $allocbuf = "${helper}_allocbuf";
> +our $transmit = "${helper}_transmitmsg";
> +our $getreply = "${helper}_getreply";
> +our $setcallbacks = "${helper}_setcallbacks";
> +
> +f_decl($allocbuf, 'helper', 'unsigned char *', '(int len, void *user)');
> +f_decl($transmit, 'helper', 'void',
> + '(unsigned char *msg_freed, int len, void *user)');
> +f_decl($getreply, 'helper', 'int', '(void *user)');
> +
> +sub typeid ($) { my ($t) = @_; $t =~ s/\W/_/; return $t; };
> +
> +$out_body{'callout'} .= <<END;
> +static int bytes_get(const unsigned char **msg,
> + const unsigned char *const endmsg,
> + void *result, int rlen)
> +{
> + if (endmsg - *msg < rlen) return 0;
> + memcpy(result,*msg,rlen);
> + *msg += rlen;
> + return 1;
> +}
> +
> +END
> +$out_body{'helper'} .= <<END;
> +static void bytes_put(unsigned char *const buf, int *len,
> + const void *value, int vlen)
> +{
> + assert(vlen < INT_MAX/2 - *len);
> + if (buf)
> + memcpy(buf + *len, value, vlen);
> + *len += vlen;
> +}
> +
> +END
> +
> +foreach my $simpletype (qw(int uint16_t uint32_t unsigned), 'unsigned long')
> {
> + my $typeid = typeid($simpletype);
> + $out_body{'callout'} .= <<END;
> +static int ${typeid}_get(const unsigned char **msg,
> + const unsigned char *const endmsg,
> + $simpletype *result)
> +{
> + return bytes_get(msg, endmsg, result, sizeof(*result));
> +}
> +
> +END
> + $out_body{'helper'} .= <<END;
> +static void ${typeid}_put(unsigned char *const buf, int *len,
> + const $simpletype value)
> +{
> + bytes_put(buf, len, &value, sizeof(value));
> +}
> +
> +END
> +}
> +
> +$out_body{'callout'} .= <<END;
> +static int BLOCK_get(const unsigned char **msg,
> + const unsigned char *const endmsg,
> + const uint8_t **result, uint32_t *result_size)
> +{
> + if (!uint32_t_get(msg,endmsg,result_size)) return 0;
> + if (endmsg - *msg < *result_size) return 0;
> + *result = (const void*)*msg;
> + *msg += *result_size;
> + return 1;
> +}
> +
> +static int STRING_get(const unsigned char **msg,
> + const unsigned char *const endmsg,
> + const char **result)
> +{
> + const uint8_t *data;
> + uint32_t datalen;
> + if (!BLOCK_get(msg,endmsg,&data,&datalen)) return 0;
> + if (datalen == 0) return 0;
> + if (data[datalen-1] != '\\0') return 0;
> + *result = (const void*)data;
> + return 1;
> +}
> +
> +END
> +$out_body{'helper'} .= <<END;
> +static void BLOCK_put(unsigned char *const buf,
> + int *len,
> + const uint8_t *bytes, uint32_t size)
> +{
> + uint32_t_put(buf, len, size);
> + bytes_put(buf, len, bytes, size);
> +}
> +
> +static void STRING_put(unsigned char *const buf,
> + int *len,
> + const char *string)
> +{
> + size_t slen = strlen(string);
> + assert(slen < INT_MAX / 4);
> + assert(slen < (uint32_t)0x40000000);
> + BLOCK_put(buf, len, (const void*)string, slen+1);
> +}
> +
> +END
> +
> +foreach my $sr (qw(save restore)) {
> + f_decl("${getcallbacks}_${sr}", 'callout',
> + "const ".cbtype($sr)." *",
> + "(void *data)");
> +
> + f_decl("${receiveds}_${sr}", 'callout', 'int',
> + "(const unsigned char *msg, uint32_t len, void *user)");
> +
> + f_decl("${enumcallbacks}_${sr}", 'callout', 'unsigned',
> + "(const ".cbtype($sr)." *cbs)");
> + f_more("${enumcallbacks}_${sr}", " unsigned cbflags = 0;\n");
> +
> + f_decl("${setcallbacks}_${sr}", 'helper', 'void',
> + "(struct ${sr}_callbacks *cbs, unsigned cbflags)");
> +
> + f_more("${receiveds}_${sr}",
> + <<END_ALWAYS.($debug ? <<END_DEBUG : '').<<END_ALWAYS);
> + const unsigned char *const endmsg = msg + len;
> + uint16_t mtype;
> + if (!uint16_t_get(&msg,endmsg,&mtype)) return 0;
> +END_ALWAYS
> + fprintf(stderr,"libxl callout receiver: got len=%u
> mtype=%u\\n",len,mtype);
> +END_DEBUG
> + switch (mtype) {
> +
> +END_ALWAYS
> +
> + $cbs{$sr} = "typedef struct ".cbtype($sr)." {\n";
> +}
> +
> +foreach my $msginfo (@msgs) {
> + my ($msgnum, $flags, $name, $args) = @$msginfo;
> + die if $msgnum_used{$msgnum}++;
> +
> + my $f_more_sr = sub {
> + my ($contents_spec, $fnamebase) = @_;
> + $fnamebase ||= "${receiveds}";
> + foreach my $sr (qw(save restore)) {
> + $sr =~ m/^./;
> + next unless $flags =~ m/$&/;
> + my $contents = (!ref $contents_spec) ? $contents_spec :
> + $contents_spec->($sr);
> + f_more("${fnamebase}_${sr}", $contents);
> + }
> + };
> +
> + $f_more_sr->(" case $msgnum: { /* $name */\n");
> + if ($flags =~ m/W/) {
> + $f_more_sr->(" int r;\n");
> + }
> +
> + my $c_rtype_helper = $flags =~ m/W/ ? 'int' : 'void';
> + my $c_rtype_callout = $flags =~ m/W/ ? 'int' : 'void';
> + my $c_decl = '(';
> + my $c_callback_args = '';
> +
> + f_more("${encode}_$name",
> + <<END_ALWAYS.($debug ? <<END_DEBUG : '').<<END_ALWAYS);
> + unsigned char *buf = 0;
> + int len = 0, allocd = 0;
> +
> +END_ALWAYS
> + fprintf(stderr,"libxl-save-helper: encoding $name\\n");
> +END_DEBUG
> + for (;;) {
> + uint16_t_put(buf, &len, $msgnum /* $name */);
> +END_ALWAYS
> +
> + my @args = @$args;
> + my $c_recv = '';
> + my ($argtype, $arg);
> + while (($argtype, $arg, @args) = @args) {
> + my $typeid = typeid($argtype);
> + my $c_args = "$arg";
> + my $c_get_args = "&$arg";
> + if ($argtype eq 'STRING') {
> + $c_decl .= "const char *$arg, ";
> + $f_more_sr->(" const char *$arg;\n");
> + } elsif ($argtype eq 'BLOCK') {
> + $c_decl .= "const uint8_t *$arg, uint32_t ${arg}_size, ";
> + $c_args .= ", ${arg}_size";
> + $c_get_args .= ",&${arg}_size";
> + $f_more_sr->(" const uint8_t *$arg;\n".
> + " uint32_t ${arg}_size;\n");
> + } else {
> + $c_decl .= "$argtype $arg, ";
> + $f_more_sr->(" $argtype $arg;\n");
> + }
> + $c_callback_args .= "$c_args, ";
> + $c_recv.=
> + " if (!${typeid}_get(&msg,endmsg,$c_get_args)) return
> 0;\n";
> + f_more("${encode}_$name", " ${typeid}_put(buf, &len,
> $c_args);\n");
> + }
> + $f_more_sr->($c_recv);
> + $c_decl .= "void *user)";
> + $c_callback_args .= "user";
> +
> + $f_more_sr->(" if (msg != endmsg) return 0;\n");
> +
> + my $c_callback;
> + if ($flags !~ m/c/) {
> + $c_callback = "${callback}_$name";
> + } else {
> + $f_more_sr->(sub {
> + my ($sr) = @_;
> + $cbs{$sr} .= " $c_rtype_callout (*${name})$c_decl;\n";
> + return
> + " const ".cbtype($sr)." *const cbs =\n".
> + " ${getcallbacks}_${sr}(user);\n";
> + });
> + $c_callback = "cbs->${name}";
> + }
> + my $c_make_callback = "$c_callback($c_callback_args)";
> + if ($flags !~ m/W/) {
> + $f_more_sr->(" $c_make_callback;\n");
> + } else {
> + $f_more_sr->(" r = $c_make_callback;\n".
> + " $sendreply(r, user);\n");
> + f_decl($sendreply, 'callout', 'void', '(int r, void *user)');
> + }
> + if ($flags =~ m/x/) {
> + my $c_v = "(1u<<$msgnum)";
> + my $c_cb = "cbs->$name";
> + $f_more_sr->(" if ($c_cb) cbflags |= $c_v;\n", $enumcallbacks);
> + $f_more_sr->(" $c_cb = (cbflags & $c_v) ? ${encode}_${name} :
> 0;\n",
> + $setcallbacks);
> + }
> + $f_more_sr->(" return 1;\n }\n\n");
> + f_decl("${callback}_$name", 'callout', $c_rtype_callout, $c_decl);
> + f_decl("${encode}_$name", 'helper', $c_rtype_helper, $c_decl);
> + f_more("${encode}_$name",
> +" if (buf) break;
> + buf = ${helper}_allocbuf(len, user);
> + assert(buf);
> + allocd = len;
> + len = 0;
> + }
> + assert(len == allocd);
> + ${transmit}(buf, len, user);
> +");
> + if ($flags =~ m/W/) {
> + f_more("${encode}_$name",
> + (<<END_ALWAYS.($debug ? <<END_DEBUG : '').<<END_ALWAYS));
> + int r = ${helper}_getreply(user);
> +END_ALWAYS
> + fprintf(stderr,"libxl-save-helper: $name got reply %d\\n",r);
> +END_DEBUG
> + return r;
> +END_ALWAYS
> + }
> +}
> +
> +print "/* AUTOGENERATED by $0 DO NOT EDIT */\n\n" or die $!;
> +
> +foreach my $sr (qw(save restore)) {
> + f_more("${enumcallbacks}_${sr}",
> + " return cbflags;\n");
> + f_more("${receiveds}_${sr}",
> + " default:\n".
> + " return 0;\n".
> + " }");
> + $cbs{$sr} .= "} ".cbtype($sr).";\n\n";
> + if ($ch eq 'h') {
> + print $cbs{$sr} or die $!;
> + print "struct ${sr}_callbacks;\n";
> + }
> +}
> +
> +if ($ch eq 'c') {
> + foreach my $name (@outfuncs) {
> + next unless defined $func{$name};
> + $func{$name} .= "}\n\n";
> + $out_body{$func_ah{$name}} .= $func{$name};
> + delete $func{$name};
> + }
> + print $out_body{$want_ah} or die $!;
> +} else {
> + foreach my $name (sort keys %out_decls) {
> + next unless $func_ah{$name} eq $want_ah;
> + print $out_decls{$name} or die $!;
> + }
> +}
> +
> +close STDOUT or die $!;
> --
> tg: (52b6131..) t/xen/xc.save-restore-protocol (depends on:
> t/xen/xl.ao.suspend.pre)
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel
|
 |
Lists.xenproject.org is hosted with RackSpace, monitoring our |