Re: [Xen-devel] Call to write tests for osstest

[Sander Eikelenboom <linux@xxxxxxxxxxxxxx>]

Thursday, August 8, 2013, 6:01:01 PM, you wrote:

> As release coordinator, and therefore de-facto
> tracker-of-bugs-and-regressions, it seems to me that one of the
> shortcomings in our current development mode is a lack of good
> regression testing for many of the less common, but still very
> important features of Xen; things like S3, driver domains, nested
> virt, and so on.

> We do have a regression-testing push gate for the xen trees, written
> by Ian Jackson, called osstest, with a lot of great functionality,
> including a test scheduler, an automatic Baysean bisector, and so on.
> Apart from more hardware, it is mainly lacking a more complete set of
> tests.

> The Xen Project team here at Citrix agreed with me, and have decided
> during the 4.4 timeframe to take on the task of adding some important
> functional tests to osstest; our list is below.  We'd love for you to
> join us.

> I will be tracking the implementation of the tests as part of the
> regular 4.4 release updates.  I encourage people to join us by
> thinking about a particular feature or bit of functionality that is
> not yet tested by osstest, which you would like to see implemented in
> the 4.4 timeframe, and responding to this e-mail, or one of the
> regular development update e-mails asking it to be added.

> There is a description of osstest, along with a link to the source code, here:

> http://blog.xen.org/index.php/2013/02/02/xen-automatic-test-system-osstest/

> === Testing coverage ===

> * Network driver domains
>  @George

> * new libxl w/ previous versions of xl
>  @IanJ

> * Host S3 suspend
>  @bguthro?

> * Default [example] XSM policy
>  @Stefano to ask Daniel D

> * Xen on ARM
>  # problem ATM: hardware
>   @ianc
>    emulator: @stefano to think about it

> * Storage driver domains
>  @roger

> * HVM pci passthrough
>  @anthony

> * Nested virt?
>  @intel (chased by George)

> * Fix SRIOV test (chase intel)
>  @ianj

> * Fix bisector to e-mail blame-worthy parties
>  @ianj

> * Fix xl shutdown
>   @ianj

:-)

> * stub domains
>   @athony

Some ideas, perhaps only with a xen version after a push (so Xen shouldn't be 
directly to blame) :

* More current kernels:
   - latest stable ?
   - linux-next ? (although perhaps only just before merge window, to prevent 
spending to much time in breaking on other random kernel stuff)
   - linus's tree ?
   - xen kernel tree's next branch

* Some performance testing (network, block, cpu/mem/fork/real apps benchmarks, 
some other metrics)
   - perhaps makes separate graphs of these, so one can see performance 
increase or decrease over a larger time frame, and see at around what commits 
that occurred.
   - only after all basic tests succeeded and a push was done.

--
Sander


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel