Re: [Xen-devel] Delays on usleep calls

[Dario Faggioli <dario.faggioli@xxxxxxxxxx>]

On lun, 2014-01-20 at 16:10 +0200, Pavlo Suikov wrote:
> Hi,
> 
Hi again Pavlo,

> yet another question on soft real time under Xen. Setup looks like
> this:
> 
> 
> 
> Xen: 4.4-rc1 with credit scheduler
> 
> dom0: Linux 3.8.13 with CFS scheduler
> 
> domU: Android 4.3 with Linux kernel 3.8.13 with CFS scheduler
> 
x86 or ARM host?

If x86, is DomU HVM or PV?

Also, how many pCPUs and vCPUs do the host and the various guests have?
Are you using any vCPU-to-pCPU pinning?

> Test program makes nothing but sleeping for 30 (5, 500) ms then
> printing timestamp in an endless loop. 
>
Ok, so something similar to cyclictest, right?

https://rt.wiki.kernel.org/index.php/Cyclictest

I'm also investigating on running it in a bunch of consideration... I'll
have results that we can hopefully compare to yours very soon.

What about giving a try to it yourself? I think standardizing on one (a
set of) specific tool could be a good thing.

> Results on a guest OS being run without hypervisor are pretty correct,
> while on a guest OS under a hypervisor (both in dom0 and domU) we
> observe regular delay of 5-15 ms no matter what sleep time is.
> Configuring scheduler to different weights for dom0/domU has no effect
> whatsoever. 
> 
Mmm... Can you show us at least part of the numbers? From my experience,
it's really easy to mess up with terminology in this domain (delay,
latency, jitter, etc.).

AFAIUI, you're saying that you're asking for a sleep time of X, and
you're being waking up in the interval [x+5ms, x+15ms], is that the
case?

> If setup looks like this (the only change is the Xen scheduler):
> 
> 
> 
> Xen: 4.4-rc1 with sEDF scheduler
> dom0: Linux 3.8.13 with CFS scheduler
> 
> domU: Android 4.3 with Linux kernel 3.8.13 with CFS scheduler
> 
> 
> we observe the same delay but only in domU; dom0 measurements are far
> more correct.
> 
It would be nice to know, in addition to the information above (arch,
CPUs, pinning, etc), something about how sEDF is being used in this
particular case too. Can you post the output of the following commands:

# xl list -n

# xl vcpu-list

# xl sched-sedf


That being said, sEDF is quite broken, unless used in a very specific
way. Also, even if it would be 100% working, you usecase seems to not
need sEDF (or any real-time scheduling solution) _yet_, as it does not
include any kind of other load, wrt the usleeping ask in the Dom0 or
DomU, is that the case? Or you have some other load in the system while
performing these measurements? If the latter, what and where?

What I mean is as follows. As far as Xen is concerned, if you have a
bunch of VMs, with various and different kind of load running into them,
and you want to make sure that one of them receives at least some
specific pCPU time (or things like that), then it is important what
scheduler you pick in Xen. If you only have, say, Dom0 and one DomU, and
you are only running your workload (the usleeping task) inside the DomU,
then vCPU scheduling happening in Xen should not make that much
difference. Of course, in order to be sure of that, we'd need to know
more about the configuration, as I already asked above.

Oh, and now that I think about it, something that present in credit and
not in sEDF that might be worth checking is the scheduling rate limiting
thing.

You can fin out more about it in this blog post:
http://blog.xen.org/index.php/2012/04/10/xen-4-2-new-scheduler-parameters-2/

Discussion about it started in this thread (and then continued in a few
other ones):
http://thr3ads.net/xen-devel/2011/10/1129938-PATCH-scheduler-rate-controller

In the code, look for ratelimit_us, in particular inside
xen/common/sched_credit.c, to figure out even better what this does.

It probably isn't the source of your problems (the 'delay' you're seeing
looks too big), but since it's quite cheap to check... :-)

Let us know.

Regards,
Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

Attachment: signature.asc
Description: This is a digitally signed message part

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel