[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] RE: [Xen-devel] 10 million cycles disappearing
After looking into this a bit more, it appears to be only an artifact and a race in my measurement code (buried in macros where it was non-obvious). Thanks for the help and sorry for the noise. P.S. Average of tmem ops including page copy and compression/decompresion is in the 20K-50K cycle range. > -----Original Message----- > From: Tian, Kevin [mailto:kevin.tian@xxxxxxxxx] > Sent: Wednesday, April 08, 2009 10:47 PM > To: Dan Magenheimer; Xen-Devel (E-mail); jbeulich@xxxxxxxxxx > Subject: RE: [Xen-devel] 10 million cycles disappearing > > > In your later test with memory allocation, is it still same case that > only some samples approach 10M, or the average close to 10M? > If average cost is high, then xenoprofile could be useful here to > show hotpot to you. > > Thanks, > Kevin > > >From: Dan Magenheimer [mailto:dan.magenheimer@xxxxxxxxxx] > >Sent: 2009年4月9日 8:47 > > > >The problem still occurs on latest tip (c/s 19515). :-( > > > > > >> -----Original Message----- > >> From: Tian, Kevin [mailto:kevin.tian@xxxxxxxxx] > >> Sent: Tuesday, April 07, 2009 6:33 PM > >> To: Dan Magenheimer; Xen-Devel (E-mail); jbeulich@xxxxxxxxxx > >> Subject: RE: [Xen-devel] 10 million cycles disappearing > >> > >> > >> I remember that some previous post (from Jan?) solved some heavy > >> operation incurred in some special action of writable page > able. Not > >> recall the detail, but your description about subop with memory > >> allocation leads me to that part... > >> > >> Thanks, > >> Kevin > >> > >> >From: Dan Magenheimer > >> >Sent: 2009年4月8日 7:54 > >> > > >> >I've been seeing a possible performance problem off and on > >> >and I've spent some time tracking it but haven't made much > >> >progress and have to give up for now, so I thought I'd at > >> >least document what I know and see if it sounds familiar > >> >to anyone. > >> > > >> >The problem: Something in Xen seems to periodically take about > >> >10M cycles. I think it is an interrupt and I think it is > >> >taking a lock related to memory allocation and holding > >> >it for a LONG time (i.e. 10M cycles or close). > >> > > >> >I am measuring inside a hypercall using TSC, taking a TSC > >> >reading at entry to the hypercall code and at exit. Xen > >> >is not pre-emptive, so it can't be switching context or > >> >something, right? Nearly all of the readings are less > >> >than 100K cycles, but some samples are "huge" and > >> >usually at 9M-10M cycles. Since I am recording the max > >> >difference between the TSCs, the max "huge" grows over > >> >a long period of time, but eventually converges close > >> >to 10M (and this is a 3Ghz processor). I can see > >> >it grow using "watch". And I've NEVER seen a reading > >> >over 10M. > >> > > >> >I am able to disable interrupts and still take > >> >measurements. Roughly half of the measurements > >> >occur when doing a hypercall-subop that does no > >> >memory allocation and roughly half occur when doing > >> >a hypercall-subop that DOES do memory allocation. > >> >With interrupts disabled, the subop that DOES > >> >memory allocation still asymptotically approaches > >> >10M. The one that does NOT do memory allocation, > >> >stays relatively small. > >> > > >> >I'm currently measuring on Xen 3.3.1 but I think I've > >> >seen similar results on xen-unstable. A single 2-vcpu > >> >domain is running (in addition to domain0). > >> > > >> >Does any of that sound familiar? Any smoking guns? > >> > > >> >_______________________________________________ > >> >Xen-devel mailing list > >> >Xen-devel@xxxxxxxxxxxxxxxxxxx > >> >http://lists.xensource.com/xen-devel > >> > > > _______________________________________________ Xen-devel mailing list Xen-devel@xxxxxxxxxxxxxxxxxxx http://lists.xensource.com/xen-devel
|
![]() |
Lists.xenproject.org is hosted with RackSpace, monitoring our |