RE: [Xen-devel] Xenoprof in an HVM domain

["Santos, Jose Renato G" <joserenato.santos@xxxxxx>]

 Stephan,

Thanks for your comments.
Please see my comments embedded in the text below,

>> Another important point is that I do not think that 
>> per-guest measurements should include VMM-level execution, 
>> unlike for a system-wide measurement. That is true for both 
>> para-virtualized and fully virtualized (VT/Pacifica) guests. 
>> This is important for sampling. I am not sure tools would 
>> know what to do with samples they cannot attribute to code 
>> they know about. Furthermore, the goal of virtualization is 
>> to HIDE to guests applications the fact that they run 
>> virtualized. What would we make an exception for monitoring 
>> tools? Note that this implies that the VMM must turn off/on 
>> monitoring upon entry/exit.

I am worried that this would cause a high performance hit.
Also, I am not sure that this is the right thing to do.
After all when Xen is running on behalf of the domain, it is using CPU
clocks, causing cache misses etc...(and all this on behalf of the
domain).
Also note that if counters are virtualized and are programmed to
generate a "virtual interrupt" on overflow, the virtual interrupt
handler in the guest is the one which should sample the PC, not Xen.
Therefore the domain will not see any sample associated with Xen code.
For overflows inside Xen, it will look like for the guest as the
overflow happened on the instruction that caused the entry into
hypervisor.
I particularly think we should avoid stopping/starting counters on
entry/exit, but I am open to be convinced otherwise. I am curious about
the opnion of others on this issue, specially the core Xen team.

>> 
>> For system-wide monitoring, you do need visibility into the 
>> VMM.  Yet monitoring is driven from a guest domain, most 
>> likely domain0. On counter overflow, the VMM receives the 
>> PMU interrupt and the corresponding interrupted IP (IIP). 
>> That information must somehow be conveyed to the monitoring 
>> tool. It is not possible to simply pass the interrupt to 
>> domain0 (controlling domain for the monitoring session). To 
>> solve this problem, XenOprofile uses a in-VMM buffer where 
>> the "samples" are first saved. Then there needs to be a 
>> communication channel with controlling domain to send 
>> notification when the buffer becomes full. There needs to be 
>> one such buffer per virtual-CPU. Those buffers only need to 
>> be visible to domain0. The whole mechanism should NOT 
>> require any special code in the guest domains, except for 
>> domain0. 

That is not quite correct. Indeed you do not need to run any code in the
guest if you are interested only in kernel and Xen samples. This the
passive domain mode supported by Xenoprof (Not yet available on the
public tree, but it will be there soon, thanks to Xiaowei Yang). Dom0
collects and interpret samples for passive domains which does not need
to run any profiling code. However if you want to interpret user level
samples, you need to interpret these samples on the domain that hosts
the code, since that is the only place where the current memory mapping
necessary to map a particular PC to the user level symbol is known.
Xenoprof enable you to get complete profiling information (including
user level code in all domains). For that you need to run oprofile on
all domains for which you want user level profiling. This is the active
domain mode in Xenoprof. By choosing between passive and active domain
modes, the user can trade off complexity of monitoring with detail in
measurements.  

>> fully virtualized guests be they Linux, Windows or something 
>> else. In XenOprofile, I understand the buffer is shared via 
>> remapping. I think the interface to setup/control the buffer 
>> needs to be more generic. For instance, certain measurements 
>> may need to record in the buffer more than just the IIP.
>> they may need to also save certain counters values. The 
>> controlling domain needs some interface to express what 
>> needs to be recorded in each sample. Furthermore, it also 
>> needs to know how to resume after an overflow, i.e., what 
>> sampling period to reload in the overflowed counter. All 
>> this information must be passed to the VMM because there is 
>> not intervention from the controlling domain until the 
>> buffer fills up. Once again, this is not something new. We 
>> have the equivalent mechanism in perfmon2 simply because we 
>> support an in-kernel sampling buffer.

I think we first need to capture a list of use case scenarios before we
define how the xenoprof interface should be extended. We do not want to
create a complex interface if it ends up never being used. If you know
specific applications/scenarios that need to be supported , this could
help us identify the limitations of the current interface and determine
how it should be extended.

>> 
>> The next step is to see how the PMU can be shared between a 
>> system-wide usage and a per-guest usage. On some PMU models, 
>> this may not be possible due to hardware limitations, i.e., 
>> fully independence of the counters. This gets into a new 
>> level of complexity which has to be managed by the VMM. 
>> Basically, this requires a VMM PMU register allocator per 
>> virtual-CPU. This also implies that consumers cannot expect 
>> to systematically have access to the full PMU each time they 
>> ask for it.
>> Note that it may be acceptable for the time being to say 
>> that system-wide and per-guest are mutually exclusive.
>> 
>> Hope this helps.
>> 
>> 
>> > The current model supported by Xenoprof is system-wide profiling, 
>> > where counters are used to profile the collection of 
>> domains and Xen together.
>> > This is usefull for Xen developers to optimize Xen and 
>> > para-virtualized kernels running on Xen.
>> > 
>> > Ideally we would like to have support for both system-wide 
>> profiling 
>> > (for Xen developers) and independent guest profiling with 
>> perf counter 
>> > virtualization (for Xen users). Adding perf counter 
>> virtualization is 
>> > in our to do list. If anybody is interested in working on 
>> this please 
>> > let me know.
>> > We would appreciate any help we could get.
>> > 
>> > Thanks
>> > Renato
>> 
>> -- 
>> 
>> -Stephane
>> 
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@xxxxxxxxxxxxxxxxxxx
>> http://lists.xensource.com/xen-devel
>> 

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel