[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [Xen-devel] [OSSTEST PATCH] README.email: Add `Worked example of relevant regression in previous flight'
Signed-off-by: Ian Jackson <Ian.Jackson@xxxxxxxxxxxxx> CC: Jan Beulich <JBeulich@xxxxxxxx> --- README.email | 51 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 51 insertions(+) diff --git a/README.email b/README.email index 5de63dd..e14a816 100644 --- a/README.email +++ b/README.email @@ -89,6 +89,9 @@ history. Here are some examples: justifiable because they prevent other tests from running and can so conceal bugs.) + See `Worked example of relevant regression in previous flight', + below. + fail in 58948 pass in 58965 fail in 58948 like 37628 @@ -159,3 +162,51 @@ X-Osstest-Versions-That: tree revision `This' is the version being tested, and `That' is the baseline. + + + +Worked example of relevant regression in previous flight +-------------------------------------------------------- + +Suppose two test steps A and B, which normally run in that order: + job test-foo + A ./ts-do-some-thing + B ./ts-do-another-thing + +Suppose failure of A prevents the execution of B. (This is the usual +case where step A precedes step B; normally later steps in a job +depend on the success of earlier steps, because after an earlier +failure the testbed state is not necessarily well-defined.) + +Now suppose A has an intermittent bug, but B is totally broken. + +With our current policy on intermittent bugs[1], we would allow a push +despite the bug in A. But we should not allow a push despite B: the +100% reproducible failure of B should prevent all pushes. + +But the bug in B only shows up when A happens to pass. So the +heisenbug compensator has to insist on seeing an actual pass of B +(which in this hypothetical situation, will not occur). + +Eg, consider these flights: + + 100 is now master A pass, B pass pushed + 200 staging A pass, B fail `B REGR. vs 100' + 201 staging A fail, B not run `B fail in 200 REGR. vs 100' + +In flight 201, the failure of A is indeed justifiable as a heisenbug +because it can be seen to succeed in flight 200. It is the problem +with B which is actually blocking the push - but that failure is only +visible in flight 200. + +If, contrary to the suppositions above, the failure of B is actually a +heisenbug, then hopefully eventually both A and then B will happen to +pass in the same run. Even if that particular flight has other +problems, a future evaluation of a test of the same version can use +that flight's passes of A and B to justify, respectively, whatever +failures of A and/or B that it comes across. + +[1] In principle we could have a different policy: to try to reject +intermittent bugs. But it would require a lot of test resources +because all tests would have to be repeated a lot, and naturally +intermittent bugs would slip through anyway. -- 1.7.10.4 _______________________________________________ Xen-devel mailing list Xen-devel@xxxxxxxxxxxxx http://lists.xen.org/xen-devel
|
Lists.xenproject.org is hosted with RackSpace, monitoring our |