Re: [Xen-users] Transcendent Memory ("tmem") -capable kernel now publicly released

[Dan Magenheimer <dan.magenheimer@xxxxxxxxxx>]

> From: Phillip Susi [mailto:psusi@xxxxxxxxxx]
> Sent: Monday, April 16, 2012 12:41 PM
> To: Dan Magenheimer
> Cc: eva; xen-users@xxxxxxxxxxxxx
> Subject: Re: [Xen-users] Transcendent Memory ("tmem") -capable kernel now 
> publicly released
> 
> On 4/16/2012 12:40 PM, Dan Magenheimer wrote:
> > Hi Psusi (Phillip?) - Just saw your tmem post... sorry I don't keep
> > up with xen-users.   If you are looking at implementing Xen tmem
> > support on/for a future Ubuntu release, let me know if I can help and
> > see:
> > http://oss.oracle.com/git/?p=linux-2.6-unbreakable.git;a=summary
> 
> The last time I played with it I think I managed to get it working after
> adding the "tmem" command line arguments to xen and the guest kernel.
> What I have not been able to do is figure out how to monitor its
> usage/effectiveness.  The output of xm tmem was completely
> indecipherable.  Could you shed some light on that?

Hi Phillip --

That weird ASCII text is intended to allow the hypervisor to
communicate with userland in a fully-backwards-and-forwards-compatible
way.  It must be parsed by a userland program, i.e.:

# xm tmem-list --long --all | /usr/sbin/xen-tmem-list-parse

I usually surround the above with 'watch -d "<above commands>"'
in a big window to watch things change.  Dunno if ubuntu
has the watch command though.

(If the parsing program isn't at that location, it may not be
built or maybe is installed elsewhere on a Debian-ish system.
See the source for it in xen source in xen/tools/misc.)

The parsed output has a huge amount of detail so is still mostly
undecipherable to mortals.  The key data can also be watched in
xentop (aka "xm top")... press "t" for tmem data (which will
show up only if any of the key values are non-zero).  Also
in xentop, you can watch selfballooning working.  (Note
"xm list" has had a bug forever so doesn't show "current memory"
just the memory the domain was launched with.)
 
> Also I was wondering about how pages move back and forth between
> cleancache and pagecache.  Are they copied or moved back and forth?  In
> other words, is the data placed in cleancache only once the local
> pagecache discards it, or as soon as it is read from disk, and when it
> is later requested, is it copied back from cleancache to the local
> pagecache, thus resulting in the data being duplicated in memory, once
> for the guest, and once in tmem?

For ephemeral pools (i.e. via cleancache), "gets" are destructive,
so the page of data is moved (removed from tmem) on a successful get.
So no duplication.  For persistent pools (i.e. via frontswap),
"gets" are non-destructive, so the page of data is copied from tmem
on a successful get, which mimics a swap device, thus the name
"persistent"; a "flush" is required to destroy persistent pages of data.

See 
http://oss.oracle.com/projects/tmem/dist/documentation/api/tmemspec-v001.pdf 
(which is old, but still 99% correct) and you may also be interested
in this: http://lwn.net/Articles/454795/ )

Dan

P.S. Although Xen tmem will work without frontswap, I don't recommend it
(esp selfballooning) without frontswap.  Selfballooning can be aggressive
and may sometimes cause swapping, which is absorbed by frontswap
instead of going to a swap disk.  That's why the Oracle UEK2 kernel
includes frontswap even though it isn't upstream yet, see
http://lwn.net/Articles/465317/ (we'll be trying upstream again soon).

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxx
http://lists.xen.org/xen-users