Re: [Xen-devel] Introduce rt real-time scheduler for Xen

[Dario Faggioli <dario.faggioli@xxxxxxxxxx>]

On ven, 2014-07-11 at 00:49 -0400, Meng Xu wrote:
> This serie of patches adds rt real-time scheduler to Xen.
> 
He Meng, Sisu!

Nice to see you here on xen-devel with this nice code-drop! :-P

> In summary, It supports:
> 1) Preemptive Global Earliest Deadline First scheduling policy by using a 
> global RunQ for the scheduler;
> 2) Assign/display each VCPU's parameters of each domain;
> 3) Supports CPU Pool
> 
Great, thanks for doing the effort of extracting this from your code
base, and submit it here. :-)

Having look at the series carefully, I think it's a nice piece of work
already. There's quite a few modification and cleanups to do, and I
think there's room for quite a bit of improvement, but I really like the
fact that all the features are basically there already.

In particular, proper SMP support, per-VCPU scheduling parameters, and a
sane and theoretically sound budgetting scheme is what we're missing in
SEDF[*], and we need these things badly!

[*] Josh's RFC is improving this, but only wrt to the latter one (sane
scheduling algorithm).

> -----------------------------------------------------------------------------------------------------------------------------
> One scenario to show the functionality of this rt scheduler is as follows:
> //list each vcpu's parameters of each domain in cpu pools using rt scheduler
> #xl sched-rt
> Cpupool Pool-0: sched=EDF
> Name                                ID VCPU Period Budget
> Domain-0                             0    0     10     10
> Domain-0                             0    1     20     20
> Domain-0                             0    2     30     30
> Domain-0                             0    3     10     10
> litmus1                              1    0     10      4
> litmus1                              1    1     10      4
> 
> [...]
>
Thanks for showing this also.

> -----------------------------------------------------------------------------------------------------------------------------
> The differences between this new rt real-time scheduler and the sedf 
> scheduler are as follows:
> 1) rt scheduler supports global EDF scheduling, while sedf only supports 
> partitioned scheduling. With the support of vcpu mask, rt scheduler can also 
> be used as partitioned scheduling by setting each VCPUâs cpumask to a 
> specific cpu.
>
Which is be biggest and most important difference. In fact, although the
implementation of this scheduler can be improved (AFAICT) wrt this
aspect too, adding SMP support to SEDF would be much much harder...

> 2) rt scheduler supports setting and getting each VCPUâs parameters of a 
> domain. A domain can have multiple vcpus with different parameters, rt 
> scheduler can let user get/set the parameters of each VCPU of a specific 
> domain; (sedf scheduler does not support it now)
> 3) rt scheduler supports cpupool.
>
Right. Well, to be fair, SEDF supports cpupools as well. :-)

> 4) rt scheduler uses deferrable server to burn/replenish budget of a VCPU, 
> while sedf uses constrant bandwidth server to burn/replenish budget of a 
> VCPU. This is just two options of implementing a global EDF real-time 
> scheduler and both optionsâ real-time performance have already been proved in 
> academic.
> 
So, can you put some links to some of your works on top of RT-Xen, which
is from which this scheduler comes from? Or, if that's not possible, at
least the titles?

I really don't expect people to jump on research papers, but the I've
seen a few, and the experimental sections were nice to read and quite
useful.

> -----------------------------------------------------------------------------------------------------------------------------
> TODO:
>
Allow me to add a few items here, in some sort of priority order (at
least mine one):

  *) Deal with budget overrun in the algorithm [medium]
  *) Split runnable and depleted (=no budget left) VCPU queues [easy]
> 1) Improve the code of getting/setting each VCPUâs parameters. [easy]
>     Right now, it create an array with LIBXL_XEN_LEGACY_MAX_VCPUS (i.e., 32) 
> elements to bounce all VCPUsâ parameters of a domain between xen tool and xen 
> to get all VCPUsâ parameters of a domain. It is unnecessary to have 
> LIBXL_XEN_LEGACY_MAX_VCPUS elements for this array.
>     The current work is to first get the exact number of VCPUs of a domain 
> and then create an array with that exact number of elements to bounce between 
> xen tool and xen.
> 2) Provide microsecond time precision in xl interface instead of millisecond 
> time precision. [easy]
>     Right now, rt scheduler let user to specify each VCPUâs parameters 
> (period, budget) in millisecond (i.e., ms). In some real-time application, 
> user may want to specify VCPUsâ parameters in  microsecond (i.e., us). The 
> next work is to let user specify VCPUsâ parameters in microsecond and count 
> the time in microsecond (or nanosecond) in xen rt scheduler as well.
>
  *) Subject Dom0 to the EDF+DS scheduling, as all other domains [easy]
      We can discuss what default Dom0 parameters should be, but we
      certainly want it to be scheduled as all other domains, and not
      getting too much of a special treatment.

> 3) Add Xen trace into the rt scheduler. [easy]
>     We will add a few xentrace tracepoints, like TRC_CSCHED2_RUNQ_POS in 
> credit2 scheduler, in rt scheduler, to debug via tracing.
>
  *) Try using timers for replenishment, instead of scanning the full
     runqueue every now and then [medium]

> 4) Method of improving the performance of rt scheduler [future work]
>     VCPUs of the same domain may preempt each other based on the preemptive 
> global EDF scheduling policy. This self-switch issue does not bring benefit 
> to the domain but introduce more overhead. When this situation happens, we 
> can simply promote the current running lower-priority VCPUâs priority and let 
> it  borrow budget from higher priority VCPUs to avoid such self-swtich issue.
> 
> Timeline of implementing the TODOs:
> We plan to finish the TODO 1), 2) and 3) within 3-4 weeks (or earlier).
> Because TODO 4) will make the scheduling policy not pure GEDF, (people who 
> wants the real GEDF may not be happy with this.) we look forward to hearing 
> peopleâs opinions.
>
That one is definitely something we can concentrate on later.

> -----------------------------------------------------------------------------------------------------------------------------
> Special huge thanks to Dario Faggioli for his helpful and detailed comments 
> on the preview version of this rt scheduler. :-)
> 
:-)

Regards,
Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

Attachment: signature.asc
Description: This is a digitally signed message part

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel