Re: [Xen-devel] LTTng Xen port : finally in a repository near you

[Mathieu Desnoyers <compudj@xxxxxxxxxxxxxxxxxx>]

* Mark Williamson (mark.williamson@xxxxxxxxxxxx) wrote:
> Mathieu,
> 
> Nice one, thank you for getting involved in this.  Are you planning to 
> eventually submit this for inclusion in the Xen mainline?
> 

I did the original port, but my current focus is more oriented towards
having LTTng included in Linux mainline. As you probably guess, it is
changing quite a bit in the process. I plan to first get the kernel
tracing polished, and afterward to port this "polished" tracer to
userspace tracing and hypervisor. Since a first port has been done, it
will not represent too much work to do it again.

> Is there anything we (the community) can do to help?  E.g. patch review / 
> advise / etc?
> 

You could go through a code review of the Xen port I have done. Not the
tracing infrastructure itself, since it is scheduled for major changes,
but mostly the low-level calls I did in lttd-xen, lttctl-xen and the Xen
ltt-tracer.



Note that you have to expect the current "Linux Kernel Markers" to be
ported to the Xen hypervisor eventually.

Some advice about how I currently (fail) to do the teardown of traces
correctly: I depend on synchronize_sched() under Linux to prodive a
race-less efficient teardown (using read-copy-update for my data
structures). It is a core concept I depend on to provide fast and
efficient tracing. Has RCU finally been implement in Xen since the last
time I checked?

Thanks,

Mathieu

> Cheers,
> Mark
> 
> On Monday 25 June 2007, INAKOSHI Hiroya wrote:
> > Hi Mathieu,
> >
> > I am interested in LTTng-xen because I thought that it would be nice if
> > I can get traces on both xen and guest linux at the same time.  I
> > reviewed LTTng-xen and found that
> >
> > * LTTng and LTTng-xen have a quite similar structure,
> > * a trace buffer resides in a hypervisor for LTTng-xen,
> > * it is currently impossible to get traces from guest linux because
> > there is no LTTng for 2.6.18-xen kernel, as you mentioned.
> >
> > I had coarsely ported LTTng to 2.6.18-xen, though it is only for
> > i386.  Now I can get traces on xen and guest linux simultaneously, even
> > though they put records in different trace buffers.  Then I thought that
> > it would be more useful if they put records in xen's trace buffer and I
> > can analyze events from xen and linux guests with a single lttd and
> > lttctl running on Domain-0.  Do you have an opinion about that?
> >
> > Regards,
> > Hiroya
> >
> > Mathieu Desnoyers wrote:
> > > Hello,
> > >
> > > I made a working version of the LTTng tracer for xen-unstable for x86.
> > > Here is the pointer to my repository so you can try it out :
> > >
> > > hg clone http://ltt.polymtl.ca/cgi-bin/hgweb.cgi xen-unstable-lttng.hg
> > >
> > > Basic usage :
> > >
> > > (see lttctl-xen -h)
> > >
> > > lttctl-xen -c
> > >
> > > (in a different console)
> > > lttd-xen -t /tmp/xentrace1
> > >
> > > (in the 1st console)
> > > lttctl-xen -s
> > >
> > > (tracing is active)
> > >
> > > lttctl-xen -q
> > > lttctl-xen -r
> > >
> > > lttd-xen should automatically quit after writing the last buffers as
> > > soon as lttctl-xen -r is issued.
> > >
> > > Then, you must copy the XML facilities :
> > >
> > > (see the http://ltt.polymtl.ca > QUICKSTART to see how to install the
> > > ltt-control package which contains the XML facilities in your system)
> > >
> > > lttctl-xen -e -t /tmp/xentrace1
> > >
> > > View in the visualiser : (see the QUICKSTART to see how to install it)
> > >
> > > lttv -m textDump -t /tmp/xentrace1
> > >
> > > (not tested yet) : to visualize a dom0 trace with the xen hypervisor
> > > information, one would have to collect the dom0 kernel trace and the Xen
> > > trace and open them together with :
> > > lttv -m textDump -t /tmp/xentrace1 -t /tmp/dom0trace
> > >
> > > The current Linux kernel instrumentation is for 2.6.20. A backport might
> > > be needed to 2.6.18 if there is no proper Xen support in 2.6.20 (I have
> > > not followed the recent developments).
> > >
> > >
> > > Currently broken/missing :
> > >
> > > - Ressources are not freed when the trace channels are destroyed. So you
> > >   basically have to reboot between taking different traces.
> > > - My code in the hypervisor complains to the console that subbuffers
> > >   have not been fully read when the trace channels are destroyed. The
> > >   error printing is just done too fast : lttd-xen is still there and
> > >   reading the buffers at that point. It will get fixed with proper
> > >   ressource usage tracking of both Xen and lttd-xen (same as the first
> > >   point above).
> > > - x86_64 not tested, powerpc local.h and ltt.h missing (should be ripped
> > >   from my Linux kernel LTTng).
> > >
> > >
> > > Cheers,
> > >
> > > Mathieu
> > >
> > > * Mathieu Desnoyers (compudj@xxxxxxxxxxxxxxxxxx) wrote:
> > >> Hi,
> > >>
> > >> My name is Mathieu Desnoyers, I am the current maintainer of the Linux
> > >> Trace Toolkit project, known as LTTng. This is a tracer for the 2.6
> > >> Linux kernels oriented towards high performance and real-time
> > >> applications.
> > >>
> > >> I have read your tracing thread and I am surprised to see how much
> > >> things you would like in a tracer are already implemented and tested in
> > >> LTTng. I am currently porting my tracer to Xen, so I think it might be
> > >> useful for you to know what it provides. My goal is to do not duplicate
> > >> the effort and save everyone some time.
> > >>
> > >> Here follows some key features of LTTng :
> > >>
> > >> Architecture independant data types
> > >> Extensible event records
> > >> Self-describing traces
> > >> Variable size records
> > >> Fast (200 ns per event record)
> > >> Highly reentrant
> > >> Does not disable interrupts
> > >> Does not take lock on the critical path
> > >> Supports NMI tracing
> > >> Analysis/visualization tool (LTTV)
> > >>
> > >> Looking at the integration of the existing LTTng implementation into
> > >> Xen, I came up with those two points for my Christmas whichlist :
> > >>
> > >> Additionnal functionnalities that would be nice to have in Xen :
> > >>
> > >> - RCU-style updates : would allow freeing the buffers without impact on
> > >> tracing. * I guess I could currently use :
> > >>       for_each_domain( d )
> > >>         for_each_vcpu( d, v )
> > >>           vcpu_sleep_sync(v);
> > >>       I think it will have a huge impact on the system, but it would
> > >> only be performed before trace buffers free.
> > >>
> > >> - Polling for data in Xen from a dom0 process.
> > >>   Xentrace currently polls the hypervisor each 100ms to see if there is
> > >> data that needs to be consumed. Instead of an active polling, it would
> > >> be nice to use the dom0 OS capability to put a process to sleep while
> > >> waiting for a resource. It would imply creating a module, loaded in
> > >> dom0, that would wait for a specific virq coming from the Hypervisor to
> > >> wake up such processes. We could think of exporting a complete poll()
> > >> interface through sysfs or procfs that would be a directory filled with
> > >> the resources exported from the Hypervisor to dom0 (which could include
> > >> wait for resource freed, useful when shutting down a domU instead of
> > >> busy looping). It would help dom0 to schedule other processes while a
> > >> process is waiting for the Hypervisor.
> > >>
> > >>
> > >> You might also be interested in looking at :
> > >> - the website (http://ltt.polymtl.ca)
> > >> - LTTng Xen port design document (this one is different from the one
> > >> posted by Jimi)
> > >>  
> > >> (http://ltt.polymtl.ca/svn/ltt/branches/poly/doc/developer/lttng-xen.txt
> > >>) - OLS 2006 paper "The LTTng tracer : A Low Impact Performance and
> > >> Behavior Monitor for GNU/Linux"
> > >>   (http://ltt.polymtl.ca/papers/desnoyers-ols2006.pdf)
> > >>
> > >>
> > >> Questions and constructive comments are welcome.
> > >>
> > >> Mathieu
> > >>
> > >>
> > >> OpenPGP public key:             
> > >> http://krystal.dyndns.org:8080/key/compudj.gpg Key fingerprint:     8CD5
> > >> 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68
> > >>
> > >> _______________________________________________
> > >> Xen-devel mailing list
> > >> Xen-devel@xxxxxxxxxxxxxxxxxxx
> > >> http://lists.xensource.com/xen-devel
> >
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@xxxxxxxxxxxxxxxxxxx
> > http://lists.xensource.com/xen-devel
> 
> 
> 
> -- 
> Dave: Just a question. What use is a unicyle with no seat?  And no pedals!
> Mark: To answer a question with a question: What use is a skateboard?
> Dave: Skateboards have wheels.
> Mark: My wheel has a wheel!
> 

-- 
Mathieu Desnoyers
Computer Engineering Ph.D. Student, Ecole Polytechnique de Montreal
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel