[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] alloc_heap_pages is low efficient with more CPUs

  • To: Jan Beulich <JBeulich@xxxxxxxx>
  • From: Keir Fraser <keir@xxxxxxx>
  • Date: Tue, 16 Oct 2012 09:53:08 +0100
  • Cc: tupeng212 <tupeng212@xxxxxxxxx>, xen-devel <xen-devel@xxxxxxxxxxxxx>
  • Delivery-date: Tue, 16 Oct 2012 08:54:02 +0000
  • List-id: Xen developer discussion <xen-devel.lists.xen.org>
  • Thread-index: Ac2re6my8tVS/Z1CVUa03bWv7r++Fg==
  • Thread-topic: [Xen-devel] alloc_heap_pages is low efficient with more CPUs

On 16/10/2012 09:28, "Jan Beulich" <JBeulich@xxxxxxxx> wrote:

>> It's just the small factors multiplying up. A 40G domain is 10M page
>> allocations, each of which does 64x per-cpu cpumask bitops and timestamp
>> compares. That's going on for a billion (10^9) times round
>> tlbflush_filter()s loop. Each iteration need only take a few CPU cycles for
>> the effect to actually be noticeable. If the stuff being touched were not in
>> the cache (which of course it is, since this path is so hot and not touching
>> much memory) it would probably take an hour to create the domain!
> Which means that your TODO remark in the change description
> is more than a nice-to-have, since
> - in the fallback case using 4k allocations your change doesn't
>   win anything
> - even larger domains would still suffer from this very problem
>   (albeit in 4.2 we try 1G allocations first, so your change should
>   have an even more visible effect there, as long as falling back
>   to smaller chunks isn't needed)

Agreed. This is a simple starting point which is also easy to backport
though, and solves the immediate observed problem.

> One question of course is whether, for sufficiently large allocation
> requests on sufficiently large systems, trying to avoid the TLB
> flush is worth it at all, considering that determining where to skip
> the flushes is costing so much more than the flushes themselves.

It might be a simpler option. I'll see how doing it the filtering way looks.

 -- Keir

Xen-devel mailing list



Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.