Re: [Xen-devel] Proposal: Feature Maturity Lifecycle

[George Dunlap <dunlapg@xxxxxxxxx>]

On Mon, Jun 15, 2015 at 1:31 PM, Ian Jackson <Ian.Jackson@xxxxxxxxxxxxx> wrote:
> George Dunlap writes ("Re: [Xen-devel] Proposal: Feature Maturity Lifecycle"):
>> So if we accept these definitions, it would would officially make
>> adding functionality to osstest a requirement for everybody
>> contributing new functionality.  I think at the moment, that may be
>> too high a barrier, unless it becomes simple and straightforward to
>> write, test, and contribute new tests to osstest.  (I think Stefano
>> was trying to do this a bit with the raisin project, but that's not
>> complete yet.)
>
> Adding tests to osstest is always going to be difficult, at least for
> some features or hardware.  It often involves a mixture of hardware
> and software, and may involve complicated setup.  Xen by its nature is
> slightly awkward to write tests for, because even for a simple test
> you need to have a whole machine set up running Xen, and for many
> features you need to teach the test suite how to set up the system to
> use them.
>
> That's not to say we shouldn't continue to make osstest easier to
> extend, and provide other ways to get tests executed (eg, provide some
> working in-tree tests in xen.git).  But we need to acknowledge that
> there is often some irreducible difficulty in the task of adding test
> cases.

Sure; for example, adding a host usb testcase would involve extending
osstest to know how to find a sensible USB device to pass through.

But many test cases wouldn't necessarily need too much more
infrastructure.  For cpupools, for instance, it would be fairly easy
to write a stand-alone script to randomly perform cpupool operations
with nothing but the ip of the dom0 of the host.

> A more practical requirement for a feature with "Supported" status is
> that _either_:
>
>  * The feature is tested automatically.
>
>  * At least once during each release freeze, the feature's
>    maintainers produce a test report (by a deadline specified by
>    the release manager).  Features with no test report get
>    downgraded from "Supported" to some lower status.

This seems a reasonable compromise.  Rather than forcing a maintainer
to engage with osstest as a condition of considering the feature
flagged Supported, we gently nudge them to automate it to save
themselves the hassle of testing it every release. :-)

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel