Re: [Xen-devel] [xen-4.6-testing test] 65112: regressions - FAIL

[Ian Campbell <ian.campbell@xxxxxxxxxx>]

On Fri, 2015-11-27 at 12:02 +0000, Ian Jackson wrote:
> Ian Campbell writes ("Re: [Xen-devel] [xen-4.6-testing test] 65112:
> regressions - FAIL"):
> > On Fri, 2015-11-27 at 01:18 -0700, Jan Beulich wrote:
> > > Neither of these failed in this flight, and there's nothing else
> > > blocking
> > > the push. Why did this not result in a push then? Or in other words
> > > why do the failures in earlier flights get considered a reason not to
> > > push?
> > 
> > @Ian, README.email covers lots of these kinds of patterns, but not this
> > specific one.
> 
> See below for proposed docs patch to explain the general meaning of
> `fail in X REGR. vs. Y'.
> 
> 
> > > > Âbuild-i386 5 xen-build fail in 65062 REGR. vs. 63449
> 
> This is completely explained below, I think.
> 
> > > > Âtest-amd64-amd64-xl-qemut-stubdom-debianhvm-amd64-xsm 16 guest-
> > > > localmigrate/x10 fail in 65088 REGR. vs. 63449
> 
> As explained below, in 65112 this step did not run because the earlier
> step `guest-localmigrate' failed:
> Â http://logs.test-lab.xenproject.org/osstest/logs/65112/test-amd64-
> amd64-xl-qemut-stubdom-debianhvm-amd64-xsm/info.html

Would it be possible to arrange for "blocked" to appear somewhere in the
results for the job? e.g. "blocked fail in XXX REGR. vs. YYY". README.email
says "The results normally start with the result in this flight" and I
think this would be in keeping with that.

Otherwise I think people naturally tend to just read the "and are blocking"
section and forget to consider that non-blocking stuff further down may
have (tolerably) failed but then blocking something else which is then
blocking the push.

> The fact that we have both `guest-localmigrate' and
> `guest-localmigrate/x10' isn't ideal because it hides from the
> heisenbug compensator that these are actually the same underlying
> test.ÂÂMaybe it is time now to rename `guest-localmigrate/x10' to
> `guest-localmigrate' and abolish the latter.

I think this would be a good idea.

> From 987dd088192f9f94c59beeddc073cecaad76a24e Mon Sep 17 00:00:00 2001
> From: Ian Jackson <ian.jackson@xxxxxxxxxxxxx>
> Date: Fri, 27 Nov 2015 11:36:05 +0000
> Subject: [OSSTEST PATCH] README.email: Document `fail in 58948 REGR. vs.
> Â63449'
> 
> Signed-off-by: Ian Jackson <Ian.Jackson@xxxxxxxxxxxxx>

Acked-by: Ian Campbell <ian.campbell@xxxxxxxxxx>

> ---
> ÂREADME.email |ÂÂÂ18 ++++++++++++++++++
> Â1 file changed, 18 insertions(+)
> 
> diff --git a/README.email b/README.email
> index 992a574..40df71a 100644
> --- a/README.email
> +++ b/README.email
> @@ -71,6 +71,24 @@ history.ÂÂHere are some examples:
> ÂÂÂÂÂÂÂdetect regressions of this test.ÂÂPerhaps the test has been
> ÂÂÂÂÂÂÂrecently introduced.
> Â
> +ÂÂÂfail in 58948 REGR. vs. 63449
> +
> +ÂÂÂÂÂÂThe results processor used 58948 (another flight testing the
> +ÂÂÂÂÂÂjust-tested version) to convince itself that some other test
> +ÂÂÂÂÂÂfailure is intermittent.ÂÂLook for other references to 58948 in
> +ÂÂÂÂÂÂthe report to see which those other test failures are.
> +
> +ÂÂÂÂÂÂHowever, in 58948, there were further failures.ÂÂIn particular,
> +ÂÂÂÂÂÂthe step being reported here failed, and that failure could not
> +ÂÂÂÂÂÂin turn be justified.
> +
> +ÂÂÂÂÂÂIf this further failure is in a test job, this is usually
> +ÂÂÂÂÂÂbecause the reported step did not run at all in the most recent
> +ÂÂÂÂÂÂflight, usually because it was blocked by an earlier failure.
> +ÂÂÂÂÂÂ(Intermittent build job failures are never considered
> +ÂÂÂÂÂÂjustifiable because they prevent other tests from running and
> +ÂÂÂÂÂÂcan so conceal bugs.)
> +
> ÂÂÂÂfail in 58948 pass in 58965
> ÂÂÂÂfail in 58948 like 37628
> Â

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel