Re: [Xen-devel] [PATCH v5 20/19] docs: add "sched-gran" boot parameter documentation

[George Dunlap <george.dunlap@xxxxxxxxxx>]

On 9/30/19 12:26 PM, Jürgen Groß wrote:
> On 30.09.19 13:20, Jan Beulich wrote:
>> On 30.09.2019 13:13, Jürgen Groß wrote:
>>> On 30.09.19 13:02, Jan Beulich wrote:
>>>> On 30.09.2019 12:51, Jürgen Groß wrote:
>>>>> On 30.09.19 12:25, Jan Beulich wrote:
>>>>>> On 30.09.2019 12:09, Juergen Gross wrote:
>>>>>>> Add documentation for the new "sched-gran" hypervisor boot
>>>>>>> parameter.
>>>>>>>
>>>>>>> Signed-off-by: Juergen Gross <jgross@xxxxxxxx>
>>>>>>> ---
>>>>>>>     docs/misc/xen-command-line.pandoc | 21 +++++++++++++++++++++
>>>>>>>     1 file changed, 21 insertions(+)
>>>>>>>
>>>>>>> diff --git a/docs/misc/xen-command-line.pandoc
>>>>>>> b/docs/misc/xen-command-line.pandoc
>>>>>>> index fc64429064..c855246050 100644
>>>>>>> --- a/docs/misc/xen-command-line.pandoc
>>>>>>> +++ b/docs/misc/xen-command-line.pandoc
>>>>>>> @@ -1782,6 +1782,27 @@ Set the timeslice of the credit1
>>>>>>> scheduler, in milliseconds.  The
>>>>>>>     default is 30ms.  Reasonable values may include 10, 5, or
>>>>>>> even 1 for
>>>>>>>     very latency-sensitive workloads.
>>>>>>>     +### sched-gran (x86)
>>>>>>> +> `= cpu | core | socket`
>>>>>>> +
>>>>>>> +> Default: `sched-gran=cpu`
>>>>>>> +
>>>>>>> +Set the scheduling granularity. In case the granularity is
>>>>>>> larger than 1 (e.g.
>>>>>>> +`core`on a SMT-enabled system, or `socket`) multiple vcpus are
>>>>>>> assigned
>>>>>>> +statically to a "scheduling unit" which will then be subject to
>>>>>>> scheduling.
>>>>>>> +This assignment of vcpus to scheduling units is fixed.
>>>>>>> +
>>>>>>> +`cpu`: Vcpus will be scheduled individually on single cpus.
>>>>>>> +
>>>>>>> +`core`: As many vcpus as there are hyperthreads on a physical
>>>>>>> core are
>>>>>>> +scheduled together on a physical core.
>>>>>>> +
>>>>>>> +`socket`: As many vcpus as there are hyperthreads on a physical
>>>>>>> sockets are
>>>>>>> +scheduled together on a physical socket.
>>>>>>
>>>>>> I'd prefer if this didn't end up Intel-centric; ideally it also
>>>>>> wouldn't be
>>>>>> x86-specific. AMD has introduced hyperthreading in Fam17 only;
>>>>>> Fam15 used
>>>>>> "compute units", grouping together "cores". Internally the Intel side
>>>>>> "core vs hyperthread" is represented in the same variables
>>>>>> (cpu_sibling_mask
>>>>>> in particular) as the AMD side "compute unit vs core".
>>>>>
>>>>> Yes, it is a mess.
>>>>>
>>>>>> Therefore it may be better to talk here about e.g. "smallest
>>>>>> topological
>>>>>> sub-unit" and only say "e.g. a hyperthread to make a connection to
>>>>>> common
>>>>>> x86 / Intel terminology". Of course the AMD side alternative use
>>>>>> of the
>>>>>> variables also renders the actual command line option
>>>>>> "sched-gran=core"
>>>>>> not overly fortunate. Perhaps we'd want to also use more abstract
>>>>>> terms
>>>>>> here, e.g. topological "levels"?
>>>>>
>>>>> I think regarding usage of "hyperthreads" I'll go with:
>>>>>
>>>>> +`cpu`: Vcpus will be scheduled individually on single cpus (e.g. a
>>>>> + hyperthread using x86/Intel terminology)
>>>>> +
>>>>> + `core`: As many vcpus as there are cpus on a physical core are
>>>>> + scheduled together on a physical core.
>>>>> ...
>>>>>
>>>>> I think using "core" is fine. We have it in multiple places in the
>>>>> hypervisor which are _not_ specific to Intel.
>>>>
>>>> Well, what we have in hypervisor sources is one thing - we can
>>>> settle on any convention we want there. It's the user (admin)
>>>> interface (i.e. the command line option name and description
>>>> here) which we may want to be a little more careful with. But
>>>> yes, I can see how we use "core" already in similar contexts
>>>> in the command line option doc, first and foremost on
>>>> "credit2_runqueue". (In retrospect I think this might have been
>>>> a mistake though.)
>>>
>>> So what do you suggest?
>>>
>>> <Irony on>
>>> "topology-level-just-above-the-smallest-topological-sub-unit"?
>>> <Irony-off>
>>>
>>> I can't think of any sensible terminology not resulting in something
>>> which is much harder to understand than "core".
>>
>> Ideally I'd like us to have an arch-independent way of
>> expressing things - "socket" and "node" look to be common enough,
>> so perhaps wouldn't need further abstraction, but sub-socket
>> granularities could perhaps be expressed as "level1" or "level2"?
>> And then there could be context sensitive meanings of "core",
>> "cu", and perhaps (in the future) "die".

Words like "core" should have a consistent  meaning.

I did a quick search and couldn't really find any useful resources
describing the difference.

I think we have a couple of options (not necessarily all of these are
exclusive):

* Use "core / thread" for both, and document the rough mapping of these
onto AMD terminologies.

* Use "core / thread" for Intel, and AMD-specific terminology for AMD.

* Add higher-level terms, like "secure" and "performance" (or
"smallest"), so that an administrator can say, "Give me the smallest
granularity which is still secure", and "Give me the best performance
regardless of security".  If HT is ever fixed on future processors, then
those processors in the fleet will have thread-based scheduling, and
insecure processors will have core-based scheduling.

Fundamentally, either the topology levels are similar enough that a
single setting is sensible to use across both, or they are not.  If they
are similar enough, then I think using "core / thread" and mapping them
is probably the best option.  If they are not similar enough, then
things like "level1" and "level2" aren't actually useful anyway, because
what they mean on different systems is too divergent; i.e., in all
likelihood you'd want "level2" on Intels and "level1" on AMD anyway.

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel