Re: [Xen-devel] [xen-4.6-testing test] 65112: regressions - FAIL [and 1 more messages]

[Ian Jackson <Ian.Jackson@xxxxxxxxxxxxx>]

Ian Campbell writes ("Re: [Xen-devel] [xen-4.6-testing test] 65112: regressions 
- FAIL"):
> On Fri, 2015-11-27 at 13:24 +0000, Ian Jackson wrote:
> > But it might not be true that it was blocked.
> 
> Can't sg-run-job tell if it was blocked vs something else though?

(You meant sg-execute-flight.)  No, it can't.  Steps that aren't run
simply don't appear for that job, in the db steps table.

> >   Maybe the version of
> > osstest used didn't have that step at all, for example.
> 
> In which case would it still be considering the step for failures at all?
> 
> i.e. if:
> 
> flight 100 had test-foo == pass
> flight 200 had test-foo == fail (blocking)
> flight 201 had test-foo == blocked; fail in 201 vs 100
> flight 202 had no test-foo present at all
> 
> Would the decision for flight 202 really be to consider the test-foo
> results in 100, 200 and 201, and therefore block?

Only if the evaluation of flight 202 needs to use the results in 200
or 201 to justify a failure of test-bar in 202.  Then it would spot
the earlier problems with test-foo and want a justification for them.


> > Perhaps sg-report-flight could, if there are any blockages of the form
> > `fail in XXX REGR. vs YYY', add a note below the blockage section,
> > saying something like `XXX examined since needed to justify other
> > failures, see below'.
> > 
> > I'm a bit reluctant to suggest this because it is, essentially,
> > boilerplate - it would always say the same thing about any `fail in
> > XXX' - and filling reports like this with boilerplate isn't always a
> > good idea.
> 
> In general I agree, in this case it might be worth it to counteract a
> (perfectly understandable IMHO) natural tendency to only look at the
> section labelled blocking, it's basically "don't forget that this non-
> blocking stuff might actually be relevant to the blockage".

I'll see about doing this.


Ian Campbell writes ("Re: [Xen-devel] [xen-4.6-testing test] 65112: regressions 
- FAIL"):
> On Fri, 2015-11-27 at 13:44 +0000, Ian Jackson wrote:
> > In flight 201, the failure of A is indeed justifiable as a heisenbug
> > because it can be seen to succeed in flight 200.  It is the problem
> > with B which is actually blocking the push - it is merely that the
> > failure occurred in flight 200.
> 
> This example really helped clarify things for me, thanks.
> 
> I don't know if this is the sort of thing which could fit into a doc
> somewhere (maybe README.email could have some of these kinds of worked
> examples?)

We could put some of this at the bottom of README.email, sure.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel