Re: [Xen-devel] [Notes for xen summit 2018 design session] Process changes: is the 6 monthly release Cadence too short, Security Process, ...

[Ian Jackson <ian.jackson@xxxxxxxxxx>]

Juergen Gross writes ("Re: [Xen-devel] [Notes for xen summit 2018 design 
session] Process changes: is the 6 monthly release Cadence too short, Security 
Process, ..."):
> On 05/07/18 17:14, Ian Jackson wrote:
> > Certainly it would be a bad idea to use anything *on the test host
> > itself* as a basis for a subsequent test.  The previous test might
> > have corrupted it.
> 
> Right. Not sure, whether possible, but in an ideal environment we'd have
> an external storage system reachable by the control nodes and the test
> systems. The test systems should be able to access their test data only,
> while the control nodes would initialize the test data while the related
> test machine is still offline.

That would mean that every test would run with the test host accessing
its primary OS storage via something like iSCSI.  That would be an
awful lot of extra complexity.

> > Unless you think we should do our testing of Xen mainly with released
> > versions of Linux stable branches (in which case, given how Linux
> > stable branches are often broken, we might be long out of date), or
> > our testing of Linux only with point releases of Xen, etc.
> 
> Yes, that's what I think.
> 
> The Xen (hypervisor, tools) tests should be done with released kernels
> (either stable or the last one from upstream).
> 
> Tests of Linux Xen support should be done with released Xen versions.

The result of this is that a feature which requires support in
multiple trees could not be tested until at least one (and probably
several) of the relevant trees had been released.  Which would be
rather late to discover that it doesn't work.

> > The current approach is mostly to take the most recent
> > tested-and-working git commit from each of the inputs.  This aspect of
> > osstest generally works well, I think.
> 
> We have a bandwidth problem already. If one unstable input product is
> failing all related tests do so, too. I'd rather know which of the
> input sources is the most probable one to be blamed for a test failure.

I think you are conflating "released" with "tested".  In the current
osstest setup each test of xen-unstable#staging is done with *tested*
versions of all the other trees.

So barring heisenbugs, hardware problems, or whatever, blocking
failures will be due to changes in xen-unstable#staging.
(Host-specific failures might also slip through, but this is not very
likely.)

The problem is that we have too many "heisenbugs, hardware problems,
or whatever".

> Another aspect of using stable versions is the possibility to find even
> performance regressions automatically. Changing multiple versions
> between tests makes that impossible, as you don't know who is to blame.

osstest does not change multiple versions in that sense.  At each
stage it is testing a candidate version of some particular component,
with tested-and-passed versions of the other components.

The candidate component is what the "branch" is, in the Subject line.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel