Re: [Xen-devel] Proposal: Feature Maturity Lifecycle

[George Dunlap <dunlapg@xxxxxxxxx>]

On Fri, Jun 12, 2015 at 4:22 PM, Lars Kurth <lars.kurth.xen@xxxxxxxxx> wrote:
> Hi all,
>
> following up from 
> http://lists.xenproject.org/archives/html/xen-devel/2015-06/msg01775.html I 
> wanted to propose the following convention related to feature classification 
> as a proposal for discussion. I tried to pretty much describe what we do now 
> and hope my understanding is correct. I did propose a new Complete state, to 
> cover for bigger new features which we may not want to mark as supported 
> straight away, for various reasons.
>
> I marked areas which with some of my thoughts or references to other 
> documents in [Note: ...] brackets
>
> Regards
> Lars
>
> ----
>
>         Feature Maturity Lifecycle
>         ==========================
>
>         Purpose and Scope
>         -----------------
>
> The purpose of this document is to define the possible states a feature.
> *Features* may be one of the following:
> * End user Feature: aka a pice of functionality that is controllable
>   by users of Xen through a command line option exposed via xl or
>   libvirt, a config file or otherwise
> * Support for a Hardware platform: e.g. x86 or ARM
> * Support for a specific Hardware platforms, e.g. Thunder X,
>   Xilinx ZynqMP, etc.
> * A set of APIs, e.g. men-event API, memory sharing APIs, etc.
>
> It is also the intent, that this document is used to set expectations
> of contributors to the Xen Project hypervisor, such that they can take
> appropriate steps to ensure that eventually a feature that they develop
> is a supported by the community.
>
> Further, it will help users of the Xen Project hypervisors understand
> how developers use the feature states. This will help users make informed
> decisions about the risk involved in using a feature.
>
>         Dependencies
>         -------------
>
> This document refers to definitions in other files and project conventions,
> in particular:
>
> 1.      Status of subsystems in the MAINTAINERS file
>         The MAINTAINERS file maps individuals against groups of files
>         in the source tree. In there context of this document, we say that
>         a feature is *maintained*, iff all components of that feature
>         are marked as one of the following
>         Supported:  Someone is actually paid to look after the code.
>         Maintained: Someone actually looks after the code.
>
> 2.      Classification of public APIs and interfaces:
>         APIs and other interfaces are declared stable by agreement
>         on the xen-devel@ mailing list. We consider a feature as
>         *stable*, iff all public APIs and interfaces a feature depends
>         on have been declared stable.
>
> 3.      Testing
>         The Xen Project runs a continuous test and integration system
>         in the project's test lab. We consider a feature *tested*, iff
>         there are test cases that are run in our test lab, that test a
>         specific feature and consistently pass. For hardware platforms we
>         consider a platform *tested*, iff appropriate hardware for said
>         platform is live and in active use in our test lab and tests
>         consistently pass against that platform.
>
>         In some cases, it may not be possible to add hardware to the
>         Xen Project test lab (e.g. for cost, space, or other reasons).
>         In such cases, it is acceptable for a community member or
>         organisation to run their own tests on behalf of the community.
>         In such cases, we consider a feature or platform *tested*
>         if said community member tests a feature/platform using their
>         own infrastructure and regularly and consistently reports results
>         to the community via xen-devel@. At a minimum, release candidates
>         should be tested.
>
> 4.      Documentation
>         The Xen Project requires that documentation for user facing
>         features and in-code API documentation (or user guides as
>         appropriate) are provided in tree. We say that a feature as
>         *documented*, if relevant documentation has been committed to
>         the tree.
>
>         State Definitions
>         -----------------
>
>         This section lists state definitions of a *Feature* in terms of
>         properties. States are listed in order of increasing number
>         of properties. Note that note all features will require to go
>         through each state: for example small and non-risky features
>         may go straight from under development to supported. It is up to
>         the development community to judge and discuss, which states
>         are necessary based on the size and risk of a feature.
>
> 1.      Preview
>         - Partially completed, with missing functionality
>         - May not be fully functional in all cases
>         - May not be tested
>         - APIs and interfaces may not be stable
>         - The developer is actively looking for user feedback
>         - Bugs and issues can be raised on xen-devel@ and will be
>           handled on a best-effort basis
>
> [Note:  the term prototype is listed in some cases in
>         http://wiki.xenproject.org/wiki/Xen_Project_Release_Features -
>         I propose to purge prototype and replace with Preview]
>
> 2.      Experimental
>         - Core functionality is fully functional
>         - However, not all functionality or platform support may be
>           present
>         - May not be tested, although there is an expectation that a plan
>           to improve testing is in place or worked on
>         - APIs and interfaces may not be stable
>         - Bugs and issues can be raised on xen-devel@ and will be
>           handled on a best-effort basis. However, there is an expectation
>           that issues related to broken core functionality are addressed.
>
> 3.      Complete
> [Note:  This is a state which has not existed in the past. It is aimed
>         at larger new features, which may only be in use or of interest
>         to a small number of contributors, or where not enough expertise
>         exists in the community to treat the feature as *Supported*. It
>         presents an opportunity for developers to prove over one or
>         two release cycles that said feature can be *supported* in future.]
>         - Intended functionality is fully implemented
>         - Feature is *maintained*
>         - Feature is *tested*
>         - Feature is *stable*
>         - Feature is *documented*
>         - Bugs and issues can be raised on xen-devel@ and will be
>           handled by the developers/maintainers behind the feature. There
>           is an expectation for the developers/maintainers to address
>           bugs and issues over a period of time.
>         - Regressions and blockers in a complete feature do *not* normally
>           block new releases. It is at the discretion of the community
>           and Release Manager to downgrade the feature.
>         - Security issues would *not* be handled by the security team.
>
> 4.      Supported
>         - Intended functionality is fully implemented
>         - Feature is *maintained*
>         - Feature is *tested*
>         - Feature is *stable*
>         - Feature is *documented*
>         - Bugs and issues can be raised on xen-devel@ and will be
>           handled by the developers/maintainers/committers within the
>           community.
>         - Regressions and blockers in a complete feature do normally
>           block new releases.
>         - Security issues would be handled by the security team.

So if we accept these definitions, it would would officially make
adding functionality to osstest a requirement for everybody
contributing new functionality.  I think at the moment, that may be
too high a barrier, unless it becomes simple and straightforward to
write, test, and contribute new tests to osstest.  (I think Stefano
was trying to do this a bit with the raisin project, but that's not
complete yet.)

Another practical issue: There are existing large bits of
functionality that are currently considered "supported" that are not
tested as part of our continuous integration testing.  PCI
pass-through is, I think, one major example.  According to this
classification, a lot of our features would have to move back to
"Experimental" until we managed to get things tested and documented.

Do we want to have another category -- perhaps "Legacy-Stable" or
something like that -- to indicate functionality which has been sort
of "grandfathered" in to the "supported" state?  That would allow us
to indicate that this functionality is actually usable, while keeping
us honest and motivating us to move things into new "stable".

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel