Re: [Xen-devel] [Notes for xen summit 2018 design session] Process changes: is the 6 monthly release Cadence too short, Security Process, ...

[Ian Jackson <ian.jackson@xxxxxxxxxx>]

Juergen Gross writes ("Re: [Xen-devel] [Notes for xen summit 2018 design 
session] Process changes: is the 6 monthly release Cadence too short, Security 
Process, ..."):
> On 06/07/18 16:03, Ian Jackson wrote:
> > "fail not blocking" is obviously an essential category.  If a
> > particular thing is unreliable, it needs to be stopped from blocking
> > tests.
> 
> What is the value of such a test?
> 
> Either we say the tested functionality isn't mandatory, so a failure
> should not block the release - we could just drop the test without
> losing anything.

Such a test exists in anticipation that the code will be fixed, and
start to pass.

When the test is a longstanding failure it represents a longstanding
deficiency.  Deleting the test removes any remaining pressure to fix
the deficiency.

> Or the test is wrong, so it should be either corrected or removed, but
> letting it use scarce resources is questionable.

Certainly if a whole job has been failing, for any significant period
of time, then this is indeed a waste of resources.  But that isn't
always the case.

Looking at the report from 124956, for example, the failing steps are
ones like these:

 test-armhf-armhf-libvirt-xsm 14 saverestore-support-check    fail  like 124566

   Dozens of these, and of the corresponding migrate support check.  1
   second each.  This is not a problem.  I do not intend to remove
   this.

 test-amd64-amd64-xl-qemuu-dmrestrict-amd64-dmrestrict 10 debian-hvm-install 
fail never pass

   Whole job is essentially a wipeout, so resources are indeed being
   "wasted".  But this is a new feature.  It depends on changes in
   Linux, Xen, qemu, and of course osstest.  Those pieces are on their
   way so it should start to pass soon, in at least some branches.

So those are OK I think.


Then we have this:

 test-amd64-amd64-xl-qemut-win7-amd64 17 guest-stop            fail like 124789

   200 seconds each; this is more of a problem.

 test-amd64-i386-xl-qemuu-win10-i386 10 windows-install         fail never pass

   7062 seconds each.  Four of these.

This is a much more serious problem.  Also, these tests take
inordinately long because Windows takes forever to install even when
it succeeds.  We do install tests because the churning that Windows
does when it installs do seem to tend to reveal all sorts of
bizarrenesses.

But we have a problem with triage and minding these tests.  I know
very little about Windows.  You may remember me posting to xen-devel
asking for help debugging these Windows tests.  Such help has not
really been forthcoming; certainly not in the quantity needed.

I probably should have chased this up, and set a deadline for dropping
all Windows 10 testing.  You can see why that wouldn't be popular.


Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel