Re: [Xen-devel] [Notes for xen summit 2018 design session] Process changes: is the 6 monthly release Cadence too short, Security Process, ...

[Ian Jackson <ian.jackson@xxxxxxxxxx>]

Andrew Cooper writes ("Re: [Xen-devel] [Notes for xen summit 2018 design 
session] Process changes: is the 6 monthly release Cadence too short, Security 
Process, ..."):
> On 06/07/2018 09:32, Jan Beulich wrote:
> > I think every test failure warrants looking into. It is just the case that
> > after having seen a certain "uninteresting" case a number of times, I
> > for instance make further implications from that on later flight reports.
> > Maybe I shouldn't, but I also can't afford spending endless hours on
> > looking all the details of all the flights.
> 
> The results of testing should be a single bit.  Yes or No.
> 
> No means that someone needs to investigate and get it back to saying Yes.

That one bit is "did we git a push".  It's in the Subject line.

> I've said this before, but categories like "fail never pass" and "fail
> not blocking" only muddy the water and train people to get complacent at
> the results.

These messages are clearly separated from the things one needs to care
about.

IMO the real difficulties we are having are (i) the blockers which are
not bugs in the code under test; (ii) the heisenbugs; and (iii) that
bugs are being detected in expensive (slow) osstest runs which could
have been detected earlier.

As for (i), we have discussed the various categories of this and what
we are doing about them, elsewhere in this thread.  (ii) is a bit more
complicated.  (iii) is addressed to a large extent by the patchwork
proposals.

> (Also, fail never pass is a 100% waste of time running in
> the first place, and this isn't the first time I've suggested that it is
> the lowest hanging of the low hanging fruit...)

I disagree entirely.  In general, these consume negligible resources,
and never block pushes.  Allowing the existence of this category (and
also "fail pass in NNNN" means that it is possible to develop tests
for a feature in parallel with the feature and greatly reduces the
amount of sequencing between committing to various trees.

"fail not blocking" is obviously an essential category.  If a
particular thing is unreliable, it needs to be stopped from blocking
tests.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel