[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] Re: [Xen-devel] About vcpu wakeup and runq tickling in credit
On 16/11/12 12:00, Dario Faggioli wrote: On Fri, 2012-11-16 at 11:53 +0100, Dario Faggioli wrote:On Thu, 2012-11-15 at 12:18 +0000, George Dunlap wrote:Maybe what we should do is do the wake-up based on who is likely to run on the current cpu: i.e., if "current" is likely to be pre-empted, look at idlers based on "current"'s mask; if "new" is likely to be put on the queue, look at idlers based on "new"'s mask.Ok, find attached the two (trivial) patches that I produced and am testing in these days. Unfortunately, early results shows that I/we might be missing something.I'm just came to thinking that this approach, although more, say, correct, could have a bad impact on caches and locality in general. One thing that xenalyze will already tell you is statistics on how a vcpu migrates over pcpus. For example: cpu affinity: 242 7009916158 {621089444|5643356292|19752063006} [0]: 15 6940230676 {400952|5643531152|27013831272} [1]: 19 6366861827 {117462|5031404806|19751998114} [2]: 31 6888557514 {1410800684|5643015454|19752100009} [3]: 18 7790887470 {109764|5920027975|25395539566} ...The general format is: "$number $average_cycles {5th percentile|50th percentile|95th percentile}". The first line includes samples from *all* cpus (i.e,. so it migrated a total of 242 times, averaging 7 billion cycles each time); the subsequent numbers show statistics on specific pcpus (i.e., it had 15 sessions on pcpu 0, averaging 6.94 billion cycles, &c). You should be able to use this to do a basic verification of your hypothesis that vcpus are migrating more often. In fact, suppose a new vcpu N wakes up on pcpu #x where another vcpu C is running, with prio(N)>prio(C). What upstream does is asking to #x and to all the idlers that can execute N to reschedule. Doing both is, I think, wrong, as there's the chance of ending up with N being scheduled on #x and C being runnable but not running (in #x's runqueue) even if there are idle cpus that could run it, as they're not poked (as already and repeatedly said). What the patches do, in this case (remember (prio(N)>prio(C)), is asking #x and all the idlers that can run C to reschedule, the effect being that N will likely run on #x, after a context switch, and C will run somewhere else, after a migration, potentially wasting its cache-hotness (it is running after all!). It looks like we can do better... Something like the below: + if there are no idlers where N can run, ask #x and the idlers where C can run to reschedule (exactly what the patches do, although, they do that _unconditionally_), as there isn't anything else we can do to try to make sure they both will run; + if *there*are* idlers where N can run, _do_not_ ask #x to reschedule and only poke them to come pick N up. In fact, in this case, it is not necessary to send C away for having both the vcpus ruunning, and it seems better to have N experience the migration as, since it's waking-up, it's more likely for him than for C to be cache-cold. I think that makes a lot of sense -- look forward to seeing the results. :-)There may be some other tricks we could look at. For example, if N and C are both going to do a significant chunk of computation, then this strategy will work best. But suppose that C does a significant junk of computation, but N is only going to run for a few hundred microseconds and then go to sleep again? In that case, it may be easier to just run N on the current processor and not bother with IPIs and such; C will run again in a few microseconds. Conversely, if N will do a significant chunk of work but C is fairly short, we might as well let C continue running, as N will shortly get to run. How to know if the next time this vcpu runs will be long or short? We could try tracking the runtimes of the last N (maybe 3 or 5) this was scheduled, and using that to predict the results. Do you have traces for any of those runs you did? I might just take a look at them and see if I can make an analysis of cache "temperature" wrt scheduling. :-) -George -George _______________________________________________ Xen-devel mailing list Xen-devel@xxxxxxxxxxxxx http://lists.xen.org/xen-devel
|
Lists.xenproject.org is hosted with RackSpace, monitoring our |