RE: [Xen-devel] 10 million cycles disappearing

[Dan Magenheimer <dan.magenheimer@xxxxxxxxxx>]

The problem still occurs on latest tip (c/s 19515). :-(


> -----Original Message-----
> From: Tian, Kevin [mailto:kevin.tian@xxxxxxxxx]
> Sent: Tuesday, April 07, 2009 6:33 PM
> To: Dan Magenheimer; Xen-Devel (E-mail); jbeulich@xxxxxxxxxx
> Subject: RE: [Xen-devel] 10 million cycles disappearing
>
>
> I remember that some previous post (from Jan?) solved some heavy
> operation incurred in some special action of writable page able. Not
> recall the detail, but your description about subop with memory
> allocation leads me to that part...
>
> Thanks,
> Kevin
>
> >From: Dan Magenheimer
> >Sent: 2009年4月8日 7:54
> >
> >I've been seeing a possible performance problem off and on
> >and I've spent some time tracking it but haven't made much
> >progress and have to give up for now, so I thought I'd at
> >least document what I know and see if it sounds familiar
> >to anyone.
> >
> >The problem: Something in Xen seems to periodically take about
> >10M cycles.  I think it is an interrupt and I think it is
> >taking a lock related to memory allocation and holding
> >it for a LONG time (i.e. 10M cycles or close).
> >
> >I am measuring inside a hypercall using TSC, taking a TSC
> >reading at entry to the hypercall code and at exit.  Xen
> >is not pre-emptive, so it can't be switching context or
> >something, right?  Nearly all of the readings are less
> >than 100K cycles, but some samples are "huge" and
> >usually at 9M-10M cycles.  Since I am recording the max
> >difference between the TSCs, the max "huge" grows over
> >a long period of time, but eventually converges close
> >to 10M (and this is a 3Ghz processor).  I can see
> >it grow using "watch".  And I've NEVER seen a reading
> >over 10M.
> >
> >I am able to disable interrupts and still take
> >measurements.  Roughly half of the measurements
> >occur when doing a hypercall-subop that does no
> >memory allocation and roughly half occur when doing
> >a hypercall-subop that DOES do memory allocation.
> >With interrupts disabled, the subop that DOES
> >memory allocation still asymptotically approaches
> >10M.  The one that does NOT do memory allocation,
> >stays relatively small.
> >
> >I'm currently measuring on Xen 3.3.1 but I think I've
> >seen similar results on xen-unstable.  A single 2-vcpu
> >domain is running (in addition to domain0).
> >
> >Does any of that sound familiar?  Any smoking guns?
> >
> >_______________________________________________
> >Xen-devel mailing list
> >Xen-devel@xxxxxxxxxxxxxxxxxxx
> >http://lists.xensource.com/xen-devel
> >

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-devel