|
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] Re: [Xen-devel] long tail latency caused by rate-limit in Xen credit2
Hi Dario,
Besides the sockperf benchmark, I also did another test with memcached
as the I/O application. The client side I use cloudsuite 3.0 as the
request generator. The command from the client side is as follows:
$ ./loader -a ../twitter_dataset/twitter_dataset_30x -s servers.txt -g
0.8 -T 20 -t 20 -c 20 -w 4 -e -r 5000
Here -r 5000 means the request per second is 5000. Such number of
requests makes the CPU utilization as around 20%, less the deserved
CPU shares(50%), which removes the consideration of CPU availability.
memcached results:
1, I/O-VM alone
timeDiff, rps, requests, gets, sets, hits,
misses,
avg_lat, 90th, 95th, 99th, std, min,
max, avgGetSize
20.000020, 4994.9, 99899, 79832, 20067, 66628,
13204,
0.175332, 0.194000, 0.209100, 0.279100, 0.063429, 0.123000,
11.734000, 668.072465
2, I/O-VM co-run with CPU-VM, with default ratelimit
timeDiff, rps, requests, gets, sets, hits,
misses,
avg_lat, 90th, 95th, 99th, std, min,
max, avgGetSize
20.000020, 5002.7, 100054, 79912, 20142, 66758,
13154,
0.818998, 1.300000, 1.600000, 2.100000, 0.418125, 0.137000,
10.176000, 667.906672
3, I/O-VM co-run with CPU-VM, ratelimit disable
timeDiff, rps, requests, gets, sets, hits,
misses,
avg_lat, 90th, 95th, 99th, std, min,
max, avgGetSize
20.000020, 4997.1, 99943, 79800, 20143, 66856,
12944,
0.186571, 0.215100, 0.228100, 0.286100, 0.067099, 0.130000,
7.483000, 670.413409
The above results as well as sockperf results reveal that it is better
to disable the ratelimit in consolidated environment, especially the
I/O-intensive workloads VMs share the physical resources with
CPU-intensive workloads VMs. Otherwise, the average latency as well as
the tail latency of applications might increase significantly.
Best
Tony
On Tue, Jun 13, 2017 at 4:15 PM, T S <suokunstar@xxxxxxxxx> wrote:
> On Tue, Jun 13, 2017 at 3:51 PM, Dario Faggioli
> <dario.faggioli@xxxxxxxxxx> wrote:
>> On Tue, 2017-06-13 at 14:59 -0500, T S wrote:
>>> Hi all,
>>>
>> Hi,
>>
>> Nice to hear from you again... You guys always have interesting things
>> to say/report/etc., about scheduling... I truly appreciate what you
>> do! :-)
>>
>
> Thank you for your reply, Dario. : )
>
>
>>> When I was executing the latency-sensitive applications in VMs on the
>>> latest Xen,
>>> I found the rate limit will cause the long tail latency for VMs
>>> sharing CPU with other VMs.
>>>
>> Yeah, I totally can see how this can be the case.
>>
>> Personally, I'm not a fan of context switch rate limiting. Not at all.
>> But it has proven to be useful in some workloads, so it's good for it
>> to be there.
>>
>> I think the scenario you describe is one of those cases where rate
>> limiting is better be disabled. :-)
>>
>>> (1) Problem description
>>>
>>> [snip]
>>>
>>> (2) Problem analysis
>>>
>>> ------------Analysis----------------
>>> I read the source code in Xen credit2 scheduler. The vCPU priority
>>> used in credit1 such as OVER, UNDER, BOOST, is all removed and all
>>> the
>>> vCPUs are just ordered by their credit. I traced vCPU credit and the
>>> I/O-VM vCPU credit is always larger than the CPU-VM credit. So the
>>> order of I/O-VM vCPU is always ahead of the CPU-VM vCPU.
>>>
>>> Next, I traced the time gap between vCPU wake and vCPU scheduler
>>> function. I found that if the I/O-VM run alone, the time gap is about
>>> 3,000ns; however, if the I/O-VM co-run with CPU-VM on the same core,
>>> the time gap enlarged to 1,000,000ns and that happened in every vCPU
>>> scheduling. That reminded me the ratelimit in the Xen credit
>>> scheduler. The default ratelimit in Xen is 1000us.
>>>
>>> As I modified the the ratelimit to 100us in the terminal:
>>> $ sudo /usr/local/sbin/xl sched-credit2 -s -r 100
>>>
>>> The average latency is reduced from 300+us to 200+us and the tail
>>> latency is also reduced.
>>>
>> Ok, good to hear that things are behaving as expected. :-D
>>
>>> [another snip]
>>>
>>> However, the minimum value of ratelimit is 100us which means there
>>> still exists the gap between the mix running VMs case and the running
>>> alone VM case. (P.S. the valid range of ratelimit is from 100 to
>>> 500000us). To mitigate the latency, the users have to run the I/O VMs
>>> on a dedicated core but that will waste lots of CPU resources on the
>>> other hand.
>>>
>>> As an experiment test, I modified the Xen source code to allow the
>>> ratelimit could be set as 0. As below, here is the result when I set
>>> the ratelimit to 0. Both average latency and tail latency when
>>> co-running with CPU-VMs is at the same magnitude and range of that in
>>> I/O-VM running alone.
>>>
>> Wait... it is already possible to disable ratelimiting. I mean, you're
>> right, you can't set it to 50us, because, if it's not 0, then it have
>> to be > 100us.
>>
>> But you can do:
>>
>> $ sudo /usr/local/sbin/xl sched-credit2 -s -r 0
>>
>> and it will be disabled.
>>
>> That was possible last time I tried. If it's not right now, then you've
>> found a bug (I'll double check this tomorrow morning).
>>
>
> Yes. You are right. I just double check on my clean Xen 4.8.1. I can
> disable the ratelimit by
> $ sudo /usr/local/sbin/xl sched-credit2 -s -r 0
>
>>> sockperf: ====> avg-lat= 71.766 (std-dev=1.618)
>>> sockperf: # dropped messages = 0; # duplicated messages = 0; #
>>> out-of-order messages = 0
>>> sockperf: Summary: Latency is 71.766 usec
>>> sockperf: Total 1999 observations; each percentile contains 19.99
>>> observations
>>> sockperf: ---> <MAX> observation = 99.257
>>> sockperf: ---> percentile 99.999 = 99.257
>>> sockperf: ---> percentile 99.990 = 99.257
>>> sockperf: ---> percentile 99.900 = 84.155
>>> sockperf: ---> percentile 99.000 = 78.873
>>> sockperf: ---> percentile 90.000 = 73.920
>>> sockperf: ---> percentile 75.000 = 72.546
>>> sockperf: ---> percentile 50.000 = 71.458
>>> sockperf: ---> percentile 25.000 = 70.518
>>> sockperf: ---> <MIN> observation = 63.150
>>>
>> Well, not too bad, considering it's running concurrently with another
>> VM. It means the scheduler is doing a good job "prioritizing" I/O the
>> bound workload.
>>
>>> Similar problem could also be found in credit1 scheduler.
>>>
>> And again, it should be possible to disable ratelimiting while on
>> Credit1 as well, in a similar manner (I'll check this too).
>>
>
> I just checked on credit1 in Xen 4.8.1. I can disable the ratelimit by
> $ sudo /usr/local/sbin/xl sched-credit -s -r 0.
>
> Thanks.
>
>> Thanks and Regards,
>> Dario
>> --
>> <<This happens because I choose it to happen!>> (Raistlin Majere)
>> -----------------------------------------------------------------
>> Dario Faggioli, Ph.D, http://about.me/dario.faggioli
>> Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
>
>
>
> --
>
> **********************************
>> Tony Suo
>> Computer Science, University of Texas at Arlington
> **********************************
--
**********************************
> Tony
> Email: kun.suo@xxxxxxxxxxxx
> Computer Science, University of Texas at Arlington
**********************************
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel
|
 |
Lists.xenproject.org is hosted with RackSpace, monitoring our |