[Xen-ia64-devel] Xen/ia64 overhead sliced in half! (sort of)

["Magenheimer, Dan (HP Labs Fort Collins)" <dan.magenheimer@xxxxxx>]

In the last 24 hours, the measured overhead of Xen/ia64 was
sliced in half!  And I didn't have to change a line of code!
Well, sort of.  Here's the story.

I periodically benchmark domain0 performance against linux by
timing the compilation of linux-2.6 in a script (10-20 times).
I keep the output of these so I can compare across time.

During May and June, I did some manual tuning work, implementing
fast hyperprivops and fast reflection, to reduce the overhead
of running on Xen.  By July, the difference between Linux (compiling
linux) and domain0 (compiling linux) was down to about 4%.
In late July or early August, the overhead inexplicably went up
to about 9-10%.   However at this time, my disk had crashed and I had
to rebuild my environment and scripts; one change I made when rebuilding
was to do the performance comparison by compiling a different version
of Linux (2.6.9 instead of 2.6.7).  This shouldn't matter but
it appeared that it did.  The change in which kernel I was compiling,
however, rendered all my previous measurements useless for comparison.

During the last month or so, I've been working primarily on merging
various code and haven't had a chance to look into this discrepancy.
But I've been wondering nearly every day:  Why did the Xen overhead
suddenly jump up from 4% to 9-10%???

Over the last few days, I got Stephane Eranian's pfmon tool working
on domain0 so as to do a more thorough analysis of where Xen is
spending its time.  As it turned out that was (almost) unnecessary!
The first time I ran pfmon on Xen (compiling Linux), the results
were as expected.  Then I ran pfmon on Linux and got a surprise.
Pfmon output measures *each* processor that is running... and
when I was running Linux (compiling Linux) for the last two months,
TWO processors were being used (to compile Linux).  Compiling Linux
appears to be serialized, and it mostly is, but apparently there
is some process parallelism and the second processor is used for
a few things, which accounts for the few percentage points of
difference.  Apparently when I rebuilt my environment and scripts,
I had neglected to add "nosmp" to my elilo.conf line for booting
Linux!  Pfmon was kind enough to expose my oversight.  When I added
"nosmp", the measurement for compiling Linux on Linux went up several
percent and now can be compared directly against the "doesn't yet
support smp" Xen domain0... and the difference is about 4%.

Silly me :-}

So, overhead for domain0 on Xen/ia64 (as measured by this one benchmark
and comparing "apples and apples" -- single processor only), is in
the 4% range.  I still expect this to go lower, probably into the
2% range.  But for now functionality is higher priority than improving
performance.

Thanks much to Stephane's pfmon tool for being smarter than I am!

Dan

P.S. If you've done your own benchmarks in the past and found
a much higher difference, try again.  There was a bug in Xen
PAL calls in which "time" was moving forward according to a
fixed 900MHz clock.  Thus if you were running on a 1350MHz box,
it would appear that Xen was slowing the benchmark down by
over 50%.  However, if you also measured the passage of time
by a stopwatch vs Xen, you would have seen that Xen time was
moving 50% faster than stopwatch time.  That should all
now be fixed.  (It was actually fixed in July... thanks to
an observation by Matt Chapman... but the fix was disabled
as it caused another regression which was only recently
fixed, so the fix was just re-enabled this week.)


_______________________________________________
Xen-ia64-devel mailing list
Xen-ia64-devel@xxxxxxxxxxxxxxxxxxx
http://lists.xensource.com/xen-ia64-devel