[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] Call to write tests for osstest

On 09/08/13 13:00, Wei Liu wrote:
On Fri, Aug 09, 2013 at 01:46:42PM +0200, Sander Eikelenboom wrote:
Friday, August 9, 2013, 1:19:54 PM, you wrote:

On gio, 2013-08-08 at 19:32 +0200, Sander Eikelenboom wrote:
* Some performance testing (network, block, cpu/mem/fork/real apps benchmarks, 
some other metrics)
    - perhaps makes separate graphs of these, so one can see performance 
increase or decrease over a larger time frame, and see at around what commits 
that occurred.
    - only after all basic tests succeeded and a push was done.

We are after this too. I am looking at how to make it possible to do
something like that on top of OSSTest.
Perf benchmarking is a little bit different from regression
smoke-testing, but still I think (hope? :-)) that most of the
infrastructure can be reused.
Yes it does require to keep the rest of the circumstances the same for a longer 
period of time.
But it could be quite valuable, slow introduced and minor performance 
regressions are hard to discover.

Also, what hardware to use and how to properly schedule these kind of
"tests" is something that needs a bit more of thinking/discussion, I
Yes since you have to keep the "environment" the same, the machine should only 
run the perf tests at that time.

This worries me. It is really hard to keep the "environment" the same.
AIUI network / block etc. performance can be affected by other kernel
subsystems. Also the kernel config could also have impact on

The same is true for functional regressions -- a bug could be introduced by Xen, qemu, or the kernel. The tester already keeps track of this, as far as I know, and only moves one thing at a time.


Xen-devel mailing list



Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.