Re: [Xen-devel] [PATCH v2] Xen: Spread boot time page scrubbing across all available CPU's

[Konrad Rzeszutek Wilk <konrad@xxxxxxxxxx>]

On Tue, Apr 01, 2014 at 03:29:05PM -0400, Konrad Rzeszutek Wilk wrote:
> On Mon, Sep 30, 2013 at 01:35:17PM +0100, Malcolm Crossley wrote:
> > The page scrubbing is done in 128MB chunks in lockstep across all the CPU's.
> > This allows for the boot CPU to hold the heap_lock whilst each chunk is 
> > being
> > scrubbed and then release the heap_lock when all CPU's are finished scrubing
> > their individual chunk. This allows for the heap_lock to not be held
> > continously and for pending softirqs are to be serviced periodically across
> > all CPU's.
> > 
> > The page scrub memory chunks are allocated to the CPU's in a NUMA aware
> > fashion to reduce Socket interconnect overhead and improve performance.
> > 
> > This patch reduces the boot page scrub time on a 128GB 64 core AMD Opteron
> > 6386 machine from 49 seconds to 3 seconds.
> > 
> > Changes in v2
> >  - Reduced default chunk size to 128MB
> >  - Added code to scrub NUMA nodes with no active CPU linked to them
> >  - Be robust to boot CPU not being linked to a NUMA node
> 
> Is there an updated version of this patch?
> 
> My recollection is that Jan wanted this patch to spread the work
> only on the cores and not the threads. Were there any other concerns?

And for fun I tried it on a large box and it does make a difference.

With the patch (attached) where we only use the cores and skip
the SMT threads the boot time scrubbing for 1.5TB is 66 seconds
while for all of the CPUs (including threads) is 127 seconds.

Sorry about the quality of the patch - was poking at it while
on conference calls so please treat it as HACK/RFC.

Jan/Malcom, were there any other issues that needed to be addressed?

Thanks.

Attachment: nosmt.patch
Description: Text document

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel