[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] Re: [Xen-devel] [xen-4.6-testing test] 65112: regressions - FAIL
Jan Beulich writes ("Re: [Xen-devel] [xen-4.6-testing test] 65112: regressions - FAIL"): > On 27.11.15 at 13:02, <Ian.Jackson@xxxxxxxxxxxxx> wrote: > > The fact that we have both `guest-localmigrate' and > > `guest-localmigrate/x10' isn't ideal because it hides from the > > heisenbug compensator that these are actually the same underlying > > test. Maybe it is time now to rename `guest-localmigrate/x10' to > > `guest-localmigrate' and abolish the latter. > > Independent of that, does it make sense for a dependent test to > not be considered failing intermittently when the test it depends on > is? Suppose two test steps A and B, which normally run in that order. Suppose failure of A prevents the execution of B (this is the usual case where step A precedes step B; normally later steps in a job depend on the success of earlier steps, because after an earlier failure the testbed state is not necessarily well-defined). Now suppose A has an intermittent bug, but B is totally broken. With our current policy on intermittent bugs[1], we would allow a push despite the bug in A. But we should not allow a push despite B: the 100% reproducible failure of B should prevent all pushes. But the bug in B only shows up when A happens to pass. So the heisenbug compensator has to insist on seeing an actual pass of B (which in this hypothetical situation, will not occur). Eg, consider these flights: 100 is now master A pass, B pass pushed 200 staging A pass, B fail `B REGR. vs 100' 201 staging A fail, B not run `B fail in 200 REGR. vs 100' In flight 201, the failure of A is indeed justifiable as a heisenbug because it can be seen to succeed in flight 200. It is the problem with B which is actually blocking the push - it is merely that the failure occurred in flight 200. If, contrary to my suppositions above, the failure of B is actually a heisenbug, then hopefully eventually both A and then B will happen to pass in the same run. Even if that particular flight has other problems, a future evaluation of a test of the same version can use that flight's passes of A and B to justify, respectively, whatever failures of A and/or B that it comes across. Ian. [1] In principle we could have a different policy: to try to reject intermittent bugs. But it would require a lot of test resources because all tests would have to be repeated a lot, and naturally intermittent bugs would slip through anyway. _______________________________________________ Xen-devel mailing list Xen-devel@xxxxxxxxxxxxx http://lists.xen.org/xen-devel
|
Lists.xenproject.org is hosted with RackSpace, monitoring our |