Re: [Xen-devel] [RFC] [Design] Better XL support of RTDS scheduler for Xen 4.6

[Dario Faggioli <dario.faggioli@xxxxxxxxxx>]

> 2015-02-21 10:51 GMT-05:00 Meng Xu <xumengpanda@xxxxxxxxx>:
> > Hi all,
> >
Hey Meng... First of all, thanks for keeping up working on this, we really need 
it!

> > This is for Xen 4.6: Enabling XL to set the parameters of each
> > individual VCPU of a domain for RTDS scheduler.
> >
Right.

> > [Goal]
> > Right now, xl sched-rtds tool can only set the VCPUs of a domain to
> > the same parameter although the scheduler did support VCPUs with
> > different parameters. This work is to provide the âxl sched-rtdsâ tool
> > the ability to configure the VCPUs of a domain with different
> > parameters.
> >
Yes, for RTDS, we need this, much more than for the other available
schedulers we have in Xen.

The main reason, IMO, is that I see it quite common for someone to want
to pin tasks to (v)cpus inside a real-time enabled VM, and if they do
this, they expect to be able to specify the scheduling parameters for
each one of the vcpus (or for a group of vcpus), depending on what tasks
inside the guest they are pinning where.

> > [Design sketch]
> > We submitted the patch along with the RTDS scheduler before last year.
> > The thread can be found at
> > http://lists.xen.org/archives/html/xen-devel/2014-08/msg02245.html.
> >
> > In order to implement the get function, which return the parameters of
> > all VCPUs of the same domain, we can bounce an array that has all
> > parameters of all VCPUs of the domain from hypervisor to userspace and
> > output it. (This part didn't cause much buzz. :-) )
> > The other choice is using a hypercall to get the information of each
> > VCPU. If there are x VCPUs in a domain, we will issue x hypercalls to
> > get all information of this domain. This choice causes more overhead
> > since the number of hypercalls is linear to the number of VCPUs in a
> > domain. The first choice only needs one hypercall for a domain to get
> > all needed information.
> >
ISTR that at some point we agreed on the fact that, even if getting and
setting vcpu params won't happen in hot paths (neither from the tools
nor from the hypervisor point of view), batching hypercall is always
good, an we should always batch as far as we can.

That of course does not mean that we the API, at all levels, need to
always deal with all vcpus. As you know very well, there is the hcall
interface, where we said we want to batch; then there is the libxc
interface, which can well diverge, but it's usually better map quite
closely to the hcall one; and then there is libxl, on top of libxl, and
things can be significantly different here.

So, I think your design proposal/code should explicitly state what the
plan is at each level.

> > In order to implement the set function, which set the parameters of
> > one specific VCPU of a domain, we have two different approaches to
> > implement it:
> > Choice 1): When users set the parameters of k^th VCPUs of a domain,
> > which has x VCPUs in total, the toolstack passes "only" the modified
> > VCPUsâ information from userspace to hypervisor via hypercall;  So
> > this implementation does "not" need an array to bounce between
> > userspace and hypervisor. (The implementation in lilbxc was at
> > http://lists.xen.org/archives/html/xen-devel/2014-08/msg02248.html)
> > Choice 2): When users set the parameters of k^th VCPUs of a domain,
> > which has x VCPUs in total, the toolstack will create an array with
> > length x and set the information (period, budget) of the k^th element
> > in the array. The other elements in the array are marked as 0. After
> > the toolstack bounce the array into the hypervisor, it will only set
> > the parameters of the k^th VCPU.
> >
Get and set interface should match, at each level, so we should decide
once and for both of them, bearing in mind all the pros and the cons of
each solution together, not split the two decisions.

> > My questions are:
> > 1) Should we allow users to set the parameters of "several" VCPUs with
> > one command:
> > For example, should we allow users to set the parameters of VCPU 1 and
> > 2 of domain litmus1 with one xl sched-rtds command?
> > # xl sched-rtds -d litmus1 -v 1 -p 20000 -b 10000 -v 2 -p 30000 -b 20000
> > IMO, we'd better not allow them to do this because 1) setting the
> > parameters of VCPUs of a domain should not be very frequent; 2) the
> > command will become quite long when users want to set the parameters
> > of many VCPUs of a domain.
> >
I actually see some value in supporting such syntax. It's not something
critical, though, I think. For sure it doesn't have much to do with the
decision on how the interface should look like at the various level.

This means you can just go ahead and send code implementing only one '-v
xxx -p yy y -b zzz', and we can extend it later, adding the support for
the list.

> > 2) Which design choice should we choose to implement the set function,
> > Choice 1 or Choice 2? or do you have better suggestion?
> >
If we batch for get, I think we should batch for set. As I said, I
personally see some value in an interface allowing setting all the
parameters of all the vcpus at the same time. In fact, this is probably
better, even as far as real-time theory is concerned: as I'm sure you
know better than me, parameter changing can be tricky for the algorithm,
so it's probably better to encourage the user to change the params all
together, synchronously, rather than having to deal with a stream of
adjustments scattered in time.

So, my opinion, choice 2).

If I can add something, the big challenge here is how to make the
interfaces good and homogeneous enough with what we already have in
place for other schedulers. I really suggest you to address this too in
this design conversation and/or in RFC/code.

Thanks and Regards,
Dario

Attachment: signature.asc
Description: This is a digitally signed message part

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel