Re: [Xen-devel] [PATCH v2] Xen sched: Fix multiple runqueues in credit2

[Dario Faggioli <dario.faggioli@xxxxxxxxxx>]

On Fri, 2014-02-07 at 23:03 -1000, Justin Weaver wrote:
> This patch attempts to address the issue of the Xen Credit 2
> Scheduler only creating one vCPU run queue on multiple physical
> processor systems. It should be creating one run queue per
> physical processor.
> 
> CPU 0 does not get a starting callback, so it is hard coded to run
> queue 0. At the time this happens, socket information is not
> available for CPU 0.
> 
> Socket information is available for each individual CPU when each
> gets the STARTING callback (socket information is also available
> for CPU 0 by that time). Each are assigned to a run queue
> based on their socket.
> 
Right. Don't forget to put your 'Signed-off-by: xxx' here, as an
indication that you certify the patch under the "Developer's Certificate
of Origin":

http://wiki.xenproject.org/wiki/Submitting_Xen_Patches#Signing_off_a_patch

> ---
> Changes from v1:
> * moved comments to the top of the section in one long comment block
> * collapsed code to improve readability
> * fixed else if indentation style
> * updated comment about the runqueue plan
>
Thanks for all this.

> diff --git a/xen/common/sched_credit2.c b/xen/common/sched_credit2.c
> index 4e68375..3ff46a3 100644

>  /*
> @@ -1945,6 +1945,8 @@ static void deactivate_runqueue(struct csched_private 
> *prv, int rqi)
>  static void init_pcpu(const struct scheduler *ops, int cpu)
>  {

> @@ -1959,15 +1961,26 @@ static void init_pcpu(const struct scheduler *ops, 
> int cpu)
>          return;
>      }
>  
> -    /* Figure out which runqueue to put it in */
> +    /*
> +     * Choose which run queue to add cpu to based on its socket.
> +     * If it's CPU 0, hard code it to run queue 0 (it doesn't get a STARTING
> +     * callback and socket information is not yet available for it).
> +     * If cpu is on the same socket as CPU 0, add it to run queue 0 with CPU 
> 0.
> +     * Else If cpu is on socket 0, add it to a run queue based on the socket
> +     * CPU 0 is actually on.
> +     * Else add it to a run queue based on its own socket. 
> +     */
> +
>
I don't think you need this extra blank line here.

>      rqi = 0;
> +    cpu_socket = cpu_to_socket(cpu);
> +    cpu0_socket = cpu_to_socket(0);
>  
> -    /* Figure out which runqueue to put it in */
> -    /* NB: cpu 0 doesn't get a STARTING callback, so we hard-code it to 
> runqueue 0. */
> -    if ( cpu == 0 )
> -        rqi = 0;
> +    if ( cpu == 0 || cpu_socket == cpu0_socket )
> +        rqi = 0;    
> +    else if ( cpu_socket == 0 )
> +        rqi = cpu0_socket;
>      else
> -        rqi = cpu_to_socket(cpu);
> +        rqi = cpu_socket;
>
This looks much better, thanks.

So, what about Juergen's comments about cpupools? In case you don't
know, Xen allows for the pCPUs to be pooled in pretty much any way. A
pool is basically a set of pCPUs, and each pool has its own scheduler.
You can have, for instance, pCPUs 0-7 in pool0, with the credit1
scheduler, and pCPUs 8-15 in pool1, with credit2.

More info about cpupools here:
http://wiki.xenproject.org/wiki/Xen_4.2:_cpupools
http://wiki.xenproject.org/wiki/Cpupools_Howto

I agree with Juergen that it would be wrong (perhaps just pointless, but
I lean toward the former) to have pCPUs from different cpupools in the
same runqueue. As he says, right now, one runqueue per cpupool is what
is created... I think we need to check more thoroughly what happens
when, after your bugfix, we start creating cpupools.

Can you look into that and report it here? (don't forget to keep Juergen
in Cc).

Thanks again and Regards,
Dario

-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

Attachment: signature.asc
Description: This is a digitally signed message part

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel