[Xen-devel] libxl/xl memory paging/sharing/ballooning/etc proposal.

[Ian Campbell <Ian.Campbell@xxxxxxxxxx>]

I think we've roughly reached consensus on what this could look like.
I've combined the various aspects of the proposal into the below.

In hindsight I should probably have written this as a proposed patch to
libxl.h and xl.cfg.pod.5 etc but I've written it now so here it is.

There are a few outstanding issues / decisions marked with XXX

Ian J -- I've CC'd you since I propose some new uses of the events
subsystem (as well as being interested in your overall opinion)

high level libxl interface
---------------------------

libxl_domain_set_memory_policy(ctx, domid, const char *actor, const char 
*params):
        
        Sets the name of the "actor" which will implement memory policy
        for this domain by writing the name of the actor to
                /libxl/X/memory-policy/actor
        and the params to
                /libxl/X/memory-policy/actor-params
                        
        actor-params is defined per-actor. normally be a comma separated
        key=value list.

        If libxl_domain_set_memory_policy is not called for a domain
        then the default is considered to be "" i.e. none.

        XXX should the actor config also be part of struct
        libxl_domain_build_info? (I expect so)

libxl_domain_get_memory_policy(ctx, domid, char **actor, char **params)

        Retrieves the current policy and params (if non NULL etc).

libxl_domain_set_memory_target(ctx, domid, target_memkb, bool relative,
bool enforce):

        Replaces existing libxl_domain_memory_target() function.
                
        If memory-policy/actor != "":
                updates /libxl/X/memory-policy/<target,enforce>
        else:
                behaves as libxl_domain_memory_target today, e.g. sets
                balloon target in xenstore and iff enforcing then sets
                maxmem -- probably this is just a call to
                libxl_domain_set_balloon_target() (see below)
                        
        target in xenstore is absolute and may be calculated from the
        existing target and provided target_memkb if relative == true
        
        enforce in xenstore is the one of the strings "true" or "false".
        (XXX or 0 1? what do we normally do? does it matter?)
        
New libxl_event_type DOMAIN_MEMORY_TARGET_CHANGED

        This event is triggered when any of /libxl/X/memory-policy/*
        change.
        
        Parameters in libxl_event.u are "char *actor", "char
        *actor_params", "bool target" and "bool enforce" as read from
        XS.
        
        Associated functions libxl_evenable/disable_blah are provided
        and take an optional "char * actor" to match against. if actor
        == NULL then events will be generated for any actor name,
        otherwise only the changes while the actor is the given actor
        will occur (this includes setting the actor to that value)

On domain build libxl shall behave as if libxl_domain_set_memory_target
had been called with:
        target_memkb == libxl_domain_build_info.target_memkb
(XXX I think that's the right one).

low level libxl interface
-------------------------

It is expected that toolstacks primary user interface will be via the
above high level interface. The lowlevel interface is provided in order
to allow implementation of memory policy actors using libxl and to allow
toolstacks to optionally provide an interface for expert users to shoot
themselves in the foot with ( ;-) )

libxl_domain_(set|get)_balloon_target(ctx, domid, target_memkb, relative, 
enforce):
        
        Read/write /local/domain/X/memory/target to control the in guest
        balloon driver. This is basically the existing
        libxl_domain_memory_target with a new name
                
        As before this is a request to the guest balloon driver to try
        and use this much actual RAM. This is an existing guest
        interface which we cannot change.

libxl_domain_(set|get)_paging_target(ctx, domid, target_memkb, relative)

        Read/write whichever XS path controls the xenpaging daemon's
        target. (Olaf?). I think this path should not be
        under /local/domain since it should not be guest visible
        (probably it already isn't).

        It is an error to call this for a PV guest (ERROR_??) for a
        guest which does not have paging enabled
        (ERROR_PAGING_DISABLED?).
        
libxl_domain_(enable|disable)_paging(ctx, domid)

        Idempotently starts or stops the xenpaging daemon for a domain.

libxl_domain_(enable|disable)_sharing(ctx, domid)

        Does what it says on the tin.

actor behaviour
---------------

An actor should use the libxl event system to monitor for
DOMAIN_MEMORY_TARGET_CHANGED events. An actor must ignore any event for
which actor != their name.

On receiving such an event the actor should apply its policy and
configuration in order (try and) cause the domain to use "target" amount
of actual RAM. This may involve calling the libxl low level interfaces,
including enabled paging and/or sharing as it desires.

It is an actor specific policy whether it guarantees to meet the target
(e.g. a balloon only actor would not be able to make that guarantee)

An actor might be either be a single daemon with host wide knowledge or
a daemon per domain (or anything in between). It is expected in the
multiple daemon case that all the daemons will either co-ordinate as
necessary or implement a policy where coordination is not necessary.

Normally the same actor would be used for all domains but we do not rule
out the possibility for non-homogeneous configurations (but it is up to
the user to make sure the actors don't have conflicting, potentially
catastrophic, interactions...)

XXX I think we need a DOMAIN_INTRODUCE event to allow for a system wide
actor. This seems like a generally useful event anyhow.

xl actor configuration
----------------------

xl will read its default actor from its configuration file. Since it is
usually expected to use the same actor for all domains it would normally
be /etc/xen/xl.conf, but this will be overrideable in a guest config
file, likewise for the policy params. xl will call
libxl_domain_set_memory_policy with the configured value while building
a domain (XXX or use fields in libxl_domain_build_config if we do
that). 

If the configuration key is absent then the default value used is
"xl" (described below). A user can explicitly supply actor = "" to get
the default libxl behaviour.

XXX Config key names TBD, "memory_policy" and "memory_policy_args"?

xl high level interface
-----------------------

xl mem-set <target>

        Pretty much same as today. Calls libxl_domain_set_memory_target
        with enforcing == 1. (but note that the behaviour of
        libxl_domain_set_memory_target has changed)

xl mem-policy-set <actor> <param-string>

        Calls libxl_domain_set_memory_policy to set the new actor.

"xl" actor behaviour
--------------------

Each xl daemon instance will listen for DOMAIN_MEMORY_TARGET_CHANGED
events for the domain which it is managing. When the event is triggered
(and actor == "xl") then xl will:

        if actor != "xl": return
        
        call libxl_domain_set_balloon_target(ctx, domid, target,
        enforcing)
        
        if domain == pv: return # cannot do more for a PV guest
        
        XXX if not enforcing: return ??? (not sure about this, we don't
        actually expose the ability to set enforcing in the xl
        interface)
        
        XXX if actor-params:paging!=true return ??? I think we should do
        this to enable memory_policy_params="paging=true|false" based
        control of paging.
        
        delay for a configurable number of seconds (configurable via
        actor-params:delay=N, default 5s?) XXX does libxl provide a
        guest accessible timeout event? Would be usefully convenient.

        XXX libxl_domain_paging_enable here or on domain build / policy
        set? libxl_domain_paging_enable is idempotent so calling it here
        is always safe?
        
        call libxl_domain_set_paging_target(ctx, domid, target)

The "xl" actor explicitly does not deal with page sharing. It is assumed
that page sharing requires system wide knowledge and policy which is out
of xl's scope and hence a different actor would be needed.

xl low level interface
----------------------

All of the following setters functions should error out unless actor ==
"xl-manual". The getter could return the current value regardless.

xl mem-balloon-(set|get)-target
xl mem-paging-(set|get)-target

        Set or get the relevant targets by calling the equivalent libxl
        function. 
        
        Enables or disables paging as necessary, i.e. target == 0 =>
        disable
        
NB if you call "xl mem-set" when actor == "xl-manual" then it won't do
anything since nothing is reacting as that actor.

XXX We could alternatively have xl also implement an "xl-manual" actor
which immediately sets both ballooning and paging targets to the given
value. Probably a good idea?

XXX Maybe "xl" (the default) should be called "xl-auto" ?

If you "xl create -e" (or -F) then you won't get a monitoring
daemon and therefore you won't get any actor implementing the memory
policy, just like you won't get automatic reboot control. I think this
is expected and reasonable.



_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel