Re: [Xen-devel] LTTng Xen port : finally in a repository near you

[INAKOSHI Hiroya <inakoshi.hiroya@xxxxxxxxxxxxxx>]

Hi Mathieu,

I am interested in LTTng-xen because I thought that it would be nice if
I can get traces on both xen and guest linux at the same time.  I
reviewed LTTng-xen and found that

* LTTng and LTTng-xen have a quite similar structure,
* a trace buffer resides in a hypervisor for LTTng-xen,
* it is currently impossible to get traces from guest linux because
there is no LTTng for 2.6.18-xen kernel, as you mentioned.

I had coarsely ported LTTng to 2.6.18-xen, though it is only for
i386.  Now I can get traces on xen and guest linux simultaneously, even
though they put records in different trace buffers.  Then I thought that
it would be more useful if they put records in xen's trace buffer and I
can analyze events from xen and linux guests with a single lttd and
lttctl running on Domain-0.  Do you have an opinion about that?

Regards,
Hiroya


Mathieu Desnoyers wrote:
> Hello,
> 
> I made a working version of the LTTng tracer for xen-unstable for x86.
> Here is the pointer to my repository so you can try it out :
> 
> hg clone http://ltt.polymtl.ca/cgi-bin/hgweb.cgi xen-unstable-lttng.hg
> 
> Basic usage :
> 
> (see lttctl-xen -h)
> 
> lttctl-xen -c
> 
> (in a different console)
> lttd-xen -t /tmp/xentrace1
> 
> (in the 1st console)
> lttctl-xen -s
> 
> (tracing is active)
> 
> lttctl-xen -q
> lttctl-xen -r
> 
> lttd-xen should automatically quit after writing the last buffers as
> soon as lttctl-xen -r is issued.
> 
> Then, you must copy the XML facilities :
> 
> (see the http://ltt.polymtl.ca > QUICKSTART to see how to install the
> ltt-control package which contains the XML facilities in your system)
> 
> lttctl-xen -e -t /tmp/xentrace1
> 
> View in the visualiser : (see the QUICKSTART to see how to install it)
> 
> lttv -m textDump -t /tmp/xentrace1
> 
> (not tested yet) : to visualize a dom0 trace with the xen hypervisor
> information, one would have to collect the dom0 kernel trace and the Xen
> trace and open them together with :
> lttv -m textDump -t /tmp/xentrace1 -t /tmp/dom0trace
> 
> The current Linux kernel instrumentation is for 2.6.20. A backport might
> be needed to 2.6.18 if there is no proper Xen support in 2.6.20 (I have
> not followed the recent developments).
> 
> 
> Currently broken/missing :
> 
> - Ressources are not freed when the trace channels are destroyed. So you
>   basically have to reboot between taking different traces.
> - My code in the hypervisor complains to the console that subbuffers
>   have not been fully read when the trace channels are destroyed. The
>   error printing is just done too fast : lttd-xen is still there and
>   reading the buffers at that point. It will get fixed with proper
>   ressource usage tracking of both Xen and lttd-xen (same as the first
>   point above).
> - x86_64 not tested, powerpc local.h and ltt.h missing (should be ripped
>   from my Linux kernel LTTng).
> 
> 
> Cheers,
> 
> Mathieu
> 
> 
> 
> * Mathieu Desnoyers (compudj@xxxxxxxxxxxxxxxxxx) wrote:
>> Hi,
>>
>> My name is Mathieu Desnoyers, I am the current maintainer of the Linux Trace
>> Toolkit project, known as LTTng. This is a tracer for the 2.6 Linux kernels
>> oriented towards high performance and real-time applications.
>>
>> I have read your tracing thread and I am surprised to see how much things
>> you would like in a tracer are already implemented and tested in LTTng. I am
>> currently porting my tracer to Xen, so I think it might be useful for you to
>> know what it provides. My goal is to do not duplicate the effort and save
>> everyone some time.
>>
>> Here follows some key features of LTTng :
>>
>> Architecture independant data types
>> Extensible event records
>> Self-describing traces
>> Variable size records
>> Fast (200 ns per event record)
>> Highly reentrant
>> Does not disable interrupts
>> Does not take lock on the critical path
>> Supports NMI tracing
>> Analysis/visualization tool (LTTV)
>>
>> Looking at the integration of the existing LTTng implementation into Xen, I
>> came up with those two points for my Christmas whichlist :
>>
>> Additionnal functionnalities that would be nice to have in Xen :
>>
>> - RCU-style updates : would allow freeing the buffers without impact on 
>> tracing.
>>     * I guess I could currently use :
>>       for_each_domain( d )
>>         for_each_vcpu( d, v )
>>           vcpu_sleep_sync(v);
>>       I think it will have a huge impact on the system, but it would only be
>>       performed before trace buffers free.
>>
>> - Polling for data in Xen from a dom0 process.
>>   Xentrace currently polls the hypervisor each 100ms to see if there is data
>>   that needs to be consumed. Instead of an active polling, it would be nice 
>> to
>>   use the dom0 OS capability to put a process to sleep while waiting for a
>>   resource. It would imply creating a module, loaded in dom0, that would wait
>>   for a specific virq coming from the Hypervisor to wake up such processes.
>>   We could think of exporting a complete poll() interface through sysfs or
>>   procfs that would be a directory filled with the resources exported from 
>> the
>>   Hypervisor to dom0 (which could include wait for resource freed, useful 
>> when
>>   shutting down a domU instead of busy looping). It would help dom0 to 
>> schedule
>>   other processes while a process is waiting for the Hypervisor.
>>
>>
>> You might also be interested in looking at :
>> - the website (http://ltt.polymtl.ca)
>> - LTTng Xen port design document (this one is different from the one posted 
>> by
>>   Jimi)
>>   (http://ltt.polymtl.ca/svn/ltt/branches/poly/doc/developer/lttng-xen.txt)
>> - OLS 2006 paper "The LTTng tracer : A Low Impact Performance and Behavior
>>   Monitor for GNU/Linux"
>>   (http://ltt.polymtl.ca/papers/desnoyers-ols2006.pdf)
>>
>>
>> Questions and constructive comments are welcome.
>>
>> Mathieu
>>
>>
>> OpenPGP public key:              
>> http://krystal.dyndns.org:8080/key/compudj.gpg
>> Key fingerprint:     8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68 
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@xxxxxxxxxxxxxxxxxxx
>> http://lists.xensource.com/xen-devel
>>
> 




_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel