Re: [Xen-devel] Performance evaluation and Questions: Eliminating Xen (RTDS) scheduler overhead on dedicated CPU

[Konrad Rzeszutek Wilk <konrad.wilk@xxxxxxxxxx>]

Andrii,

I've a question to you at the bottom that I hope you can answer.

> >> 3) There exist some constant virtualization overhead (see the case
> >> when the the task's execution time is very small 9264 cycles). I don't
> >> know where this kind of constant virtualization overhead comes from
> >> and if we can eliminate/bound this kind of overhead. Do you have any
> >> suggestion/advice on this?
> >
> > How are you testing this -- running RDTSC?  Do you know what TSC mode
> > you're running in?  If you're trapping on TSCs, that might account for
> > some of the overhead for very small cycles.
> 
> I'm using rtdsc to read the tsc counter in the test program in domU.
> I didn't configure the TSC mode for domU, so I think it should be the
> default mode (i.e., tsc_mode=1 emulated mode?).  Do you think this
> could be the source of the ~1000 cycles overhead for task with small
> execution time?

It should be the =2, which is native (no emulation). At least
that is what libxl sets by default.

> 
> You mentioned "If you're trapping on TSCs", when does this (trapping)
> may happen? Is it related to always emulated mode or never emulated
> mode or the PVRDTSCP mode? Do you have any suggestion on how I should
> investigate this?  (I had a look at

We would trap (VMEXIT) and if you look at the EIP of the guest the virtual
address should correspond to the 'rdstcp' opcode.

> http://xenbits.xen.org/docs/4.3-testing/misc/tscmode.txt, but didn't
> find the idea how to look into this. :-( )
> 
> >
> > Other than that, nothing comes to mind off the top of my head.
> 
> I see.

I had dug through the path of the different scenarios when the
guest does an HALT operation - and the codepaths we select.

You might want to see 
http://mid.gmane.org/20140423212824.GB12560@xxxxxxxxxxxxxxxxxxx
and http://mid.gmane.org/20140506173627.GA6942@xxxxxxxxxxxxxxxxxxx

The point is that you probably want to run 'xentrace' and use
the xenformat to get the raw state of all the traces.

Then comes the hard part of figuring out what happens in between
the traces. I figured out that my problem was due to three
softirqs being scheduled - TIMER, TASKLET, SCHEDULE and TASKLET
was taking an global spinlock.

In your case you probably don't have the TASKLET.

Since you are looking this from the perspective of the guest
you can setup an page between hypervisor and your guest - and
add an marker in there (set a bit). When the hypervisor
goes in the VMEXIT it can check if the marker is there and
do an trace op - and also one when it is right about to
go back to the guest.

Andrii, CC-ed here, I believe did something like that?

> 
> Thank you very much for your advice and time!
> 
> Best,
> 
> Meng
> 
> >
> >  -George
> >
> 
> 
> 
> -- 
> 
> 
> -----------
> Meng Xu
> PhD Student in Computer and Information Science
> University of Pennsylvania

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel