[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] Re: [Xen-devel] [OSSTEST PATCH] README.email: Add `Worked example of relevant regression in previous flight'
On Fri, 2015-11-27 at 15:38 +0000, Ian Jackson wrote: > Signed-off-by: Ian Jackson <Ian.Jackson@xxxxxxxxxxxxx> > CC: Jan Beulich <JBeulich@xxxxxxxx> Acked-by: Ian Campbell <ian.campbell@xxxxxxxxxx> > --- > ÂREADME.email |ÂÂÂ51 +++++++++++++++++++++++++++++++++++++++++++++++++++ > Â1 file changed, 51 insertions(+) > > diff --git a/README.email b/README.email > index 5de63dd..e14a816 100644 > --- a/README.email > +++ b/README.email > @@ -89,6 +89,9 @@ history.ÂÂHere are some examples: > ÂÂÂÂÂÂÂjustifiable because they prevent other tests from running and > ÂÂÂÂÂÂÂcan so conceal bugs.) > Â > +ÂÂÂÂÂÂSee `Worked example of relevant regression in previous flight', > +ÂÂÂÂÂÂbelow. > + > ÂÂÂÂfail in 58948 pass in 58965 > ÂÂÂÂfail in 58948 like 37628 > Â > @@ -159,3 +162,51 @@ X-Osstest-Versions-That: > ÂÂÂÂÂÂtreeÂÂÂÂÂrevision > Â > Â`This' is the version being tested, and `That' is the baseline. > + > + > + > +Worked example of relevant regression in previous flight > +-------------------------------------------------------- > + > +Suppose two test steps A and B, which normally run in that order: > +ÂÂjob test-foo > +ÂÂÂÂÂÂÂAÂÂÂ./ts-do-some-thing > +ÂÂÂÂÂÂÂBÂÂÂ./ts-do-another-thing > + > +Suppose failure of A prevents the execution of B.ÂÂ(This is the usual > +case where step A precedes step B; normally later steps in a job > +depend on the success of earlier steps, because after an earlier > +failure the testbed state is not necessarily well-defined.) > + > +Now suppose A has an intermittent bug, but B is totally broken. > + > +With our current policy on intermittent bugs[1], we would allow a push > +despite the bug in A.ÂÂBut we should not allow a push despite B: the > +100% reproducible failure of B should prevent all pushes. > + > +But the bug in B only shows up when A happens to pass.ÂÂSo the > +heisenbug compensator has to insist on seeing an actual pass of B > +(which in this hypothetical situation, will not occur). > + > +Eg, consider these flights: > + > +ÂÂ100ÂÂis now masterÂÂA pass, B passÂÂÂÂÂÂpushed > +ÂÂ200ÂÂstagingÂÂÂÂÂÂÂÂA pass, B failÂÂÂÂÂÂ`B REGR. vs 100' > +ÂÂ201ÂÂstagingÂÂÂÂÂÂÂÂA fail, B not runÂÂÂ`B fail in 200 REGR. vs 100' > + > +In flight 201, the failure of A is indeed justifiable as a heisenbug > +because it can be seen to succeed in flight 200.ÂÂIt is the problem > +with B which is actually blocking the push - but that failure is only > +visible in flight 200. > + > +If, contrary to the suppositions above, the failure of B is actually a > +heisenbug, then hopefully eventually both A and then B will happen to > +pass in the same run.ÂÂEven if that particular flight has other > +problems, a future evaluation of a test of the same version can use > +that flight's passes of A and B to justify, respectively, whatever > +failures of A and/or B that it comes across. > + > +[1] In principle we could have a different policy: to try to reject > +intermittent bugs.ÂÂBut it would require a lot of test resources > +because all tests would have to be repeated a lot, and naturally > +intermittent bugs would slip through anyway. _______________________________________________ Xen-devel mailing list Xen-devel@xxxxxxxxxxxxx http://lists.xen.org/xen-devel
|
Lists.xenproject.org is hosted with RackSpace, monitoring our |