Re: [Xen-devel] [PATCH v1 1/4] xen: add real time scheduler rt

[George Dunlap <George.Dunlap@xxxxxxxxxxxxx>]

On Wed, Sep 3, 2014 at 4:13 PM, Meng Xu <xumengpanda@xxxxxxxxx> wrote:
> Hi George,
>
>
> 2014-09-03 10:20 GMT-04:00 George Dunlap <George.Dunlap@xxxxxxxxxxxxx>:
>
>> On Sun, Aug 24, 2014 at 11:58 PM, Meng Xu <mengxu@xxxxxxxxxxxxx> wrote:
>> > diff --git a/xen/include/public/domctl.h b/xen/include/public/domctl.h
>> > index 5b11bbf..27d01c1 100644
>> > --- a/xen/include/public/domctl.h
>> > +++ b/xen/include/public/domctl.h
>> > @@ -339,6 +339,19 @@ struct xen_domctl_max_vcpus {
>> >  typedef struct xen_domctl_max_vcpus xen_domctl_max_vcpus_t;
>> >  DEFINE_XEN_GUEST_HANDLE(xen_domctl_max_vcpus_t);
>> >
>> > +/*
>> > + * This structure is used to pass to rt scheduler from a
>> > + * privileged domain to Xen
>> > + */
>> > +struct xen_domctl_sched_rt_params {
>> > +    /* get vcpus' info */
>> > +    uint64_t period; /* s_time_t type */
>> > +    uint64_t budget;
>> > +    uint16_t index;
>> > +    uint16_t padding[3];
>>
>> Why the padding?
>>
>
> I did this because of Jan's comment "Also, you need to pad the structure to
> a multiple of 8 bytes, or
> its layout will differ between 32- and 64-bit (tool stack) callers." I think
> what he said make sense so I added the padding here. :-)
>
> Here is the link: http://marc.info/?l=xen-devel&m=140661680931179&w=2

Right. :-)  I personally prefer to handle that by re-arranging the
elements rather than adding padding, unless absolutely necessary.  In
this case that shouldn't be too hard, particularly once we pare the
interface down so we only have one interface (either all one vcpu at a
time, or all batched vcpus).

> I think it's a better idea to
>  pass in an array with information about vcpus to get/set vcpus'
> information.
>
> I only need to change the code related to setting a vcpu's information.
> I have a question:
> When we set a vcpu's information by using an array, we have two choices:
>
> a) just create an array with one vcpu element, and specify the index of the
> vcpu to modify; The concern to this method is that we only uses one element
> of this array, so is it a good idea to use an array with only one element?
> b) create an array with all vcpus of this domain, modify the parameters of
> the vcpu users want to change, and then bounce the array to hypervisor to
> reset these vcpus' parameters. The concern to this method is that we don't
> need any other vcpus' information to set a specific vcpu's parameters.
> Bouncing the whole array with all vcpus information seems expensive and
> unnecessary?
>
> Do you have any suggestion/advice/preference on this?
>
> I don't really like about the idea of reading the vcpu's information
> one-by-one. :-) If a domain has many vcpus, say 12 vcpus, we will issue 12
> hypercalls to get all vcpus' information of this domain. Because we only
> need to issue one hypercall to get all information we want, the extra
> hypercalls causes more overhead. This did simplify the implementation, but
> may cause more overhead.

For convenience for users, I think it's definitely the case that libxl
should provide an interface to get and set all the vcpu parameters at
once.  Then it can either batch them all into a single hypercall (if
that's what we decide), or it can make the individual calls for each
vcpu.

But I don't really expect this to be a performance-critical operation.
Twelve hypercalls 1000 times a second is obviously way too many; but
12 hypercalls even once a second isn't really a big deal.

The main reason I would think to batch the hypercalls is for
consistency: it seems like you may want to change the period / budget
of vcpus atomically, rather than setting one, possibly having dom0
de-scheduled for a few hundred milliseconds, and then setting another.
Same thing goes for reading: I would think you would want a consistent
"snapshot" of some existing state, rather than having the possibility
of reading half the state, then having someone change it, and then
reading the other half.

Re the per-vcpu settings, though: Is it really that common for RT
domains to want different parameters for different vcpus?  Are these
parameters exposed to the guest in any way, so that it can make more
reasonable decisions as to where to run what kinds of workloads?

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel