Re: [Xen-devel] [Notes for xen summit 2018 design session] Process changes: is the 6 monthly release Cadence too short, Security Process, ...

[Lars Kurth <lars.kurth@xxxxxxxxxx>]

On 03/07/2018, 07:26, "Juergen Gross" <jgross@xxxxxxxx> wrote:
    On 02/07/18 20:03, Lars Kurth wrote:
    > We then had a discussion around why the positive benefits didn't 
materialize:
    > * Andrew and a few other believe that the model isn't broken, but that 
the issue is with how we 
    >   develop. In other words, moving to a 9 months model will *not* fix the 
underlying issues, but 
    >   merely provide an incentive not to fix them.
    
    Keeping the 6 month schedule still leads to a substantial burden for
    managing the stable releases.

That is true, which is why there was a discussion about maybe treating even/odd 
releases differently and other models were discussed (see below). But that of 
course has an impact on down streams and vendors.
    
    > * Issues highlighted were:
    >   * 2-3 months stabilizing period is too long
    
    I absolutely agree.

Could we try and establish a list for the underlying reasons. Issues with 
OSSTEST appears to be one of the reasons, but this may not be the only one.
    
    >   * Too much start/stop of development - we should branch earlier (we 
mainly do this on the last 
    >     RC now). The serial period of development has essentially become too 
short. *Everyone* in the 
    >     room agreed that fixing this, is the *most important issue*.
    
    While I'm really in favor of branching early I fear that this will even
    raise the burden on some few developers who need to backport fixes to
    the just branched off release candidate. An approach to solve this would
    be to accept a development patch only in case it is accompanied by the
    release backport.
    
    >   * Testing (aka OSSTEST) and CI automation is too fragile and we are too 
slow. Do we need another
    >     way of smoke testing which acts as a filter to OSSTEST and reduces 
the number of issues that 
    >     block everyone - also see the Gitlab discussion 
    >     
(https://lists.xenproject.org/archives/html/xen-devel/2018-07/threads.html#00126)
    >     Also, we could do testing using QEMU for most cases which are not 
hardware specific for faster 
    >     turnaround as ViryaOS is planning on doing. This is slow, but can be 
offloaded into the cloud
    >     and could be triggered before a patch hits staging. Doug indicated he 
could help get the needed 
    >     capacity for free.
    
    I really like automatic testing, but OSSTEST has its problems.

The GitLab discussion was really interesting. Looking at OSSTEST, it basically 
performs build test and integration tests on Hardware. Whereas all these are 
needed, build testing and testing of functionality that does not depend on 
hardware could be done earlier. The ghist of the GitLab discussion was to 
"move" build testing - and possibly some basic integration testing - to the 
point of submitting a patch. The basic flow is:
* Someone posts a patch to the list => this will start the GitLab machinery 
* The Gitlab machinery will do build tests (and the discussion showed that we 
should be able to do this via cross compilation or compilation on a real system 
if a service such as infosifter is used - @Matt: I can't find the company via 
Google, maybe the spelling in the minutes is wrong)
* This could eventually include a basic set of smoke tests that are system 
independent and could run under QEMU - Doug already uses a basic test where a 
xen host and/or VM is started 
* If it fails, a mail is sent in reply to the patch submission by a bot - that 
piece has been developed by Intel for QEMU and Linux and could be re-used

This would free up time by reviewers and also leads to issues found earlier. In 
other words, OSSTEST would merely re-test what had been tested earlier and 
would focus on testing on real hardware. Thus, OSSTEST failures should become 
less likely. But obviously implementing such a system, even though all the 
pieces for it exist, will take some time.

Another interesting piece of information is that OpenXT started looking at 
OSSTEST, but is now not going down this route. Maybe Rich can expand. 
    
    A major source of the pain seems to be the hardware: About half of all
    cases where I looked into the test reports to find the reason for a
    failing test flight were related to hardware failures. Not sure how to
    solve that.

This is worrying and I would like to get Ian Jackson's viewpoint on it. 
    
    Another potential problems showed up last week: OSSTEST is using the
    Debian servers for doing the basic installation. A change there (e.g.
    a new point release) will block tests. I'd prefer to have a local cache
    of the last known good set of *.deb files to be used especially for the
    branched Xen versions. This would rule out remote problems for releases.

This is again something which we should definitely look at.
    
    >   * Testing for end-users is too hard: we should not require them to 
build Xen - again with GitLab 
    >     as per previous discussion we could deliver unsigned RPMs and Debug 
Builds for testing. If we 
    >     were able to do this, this should help 
    >  
    > Then there was a discussion on how we could make more downstreams 
coalesce around the same Xen 
    > releases. This discussion was driven by Doug G, but we didn't come to a 
conclusion. If we were 
    > able to do this, we could introduce a model similarly to Debian, were 
downstreams could pull 
    > together and fund a longer support period for a Xen version which the 
majority of downstreams 
    > use, without impacting the stable release maintainer. 
    >  
    > Doug mentioned to me afterwards, that there is a Xen Slack channel for 
distros which is very 
    > active, where we could test this idea.
    >  
    > Another approach would be to follow a model as the Wayland and X11 
communities do, which have 
    > different support periods for odd and even releases. They use the 
odd-unstable/even-stable style 
    > of versioning. This may be worth investigating further.
    
    I believe just leaving out every second release (i.e. doing one release
    per year) is a rough equivalent for that with less overhead.

Well, yes and no. A longer development cycle leads to less discipline and thus 
longer stabilization periods, at least according to theory. I think for now, we 
should park this and collect more information on why the stabilisation period 
is so long and on OSSTEST and improving CI capability. All these activities 
benefit the release process whatever the length.
    
    > Lars noted that I may make sense to arrange a community call to make more 
progress on this discussion.
    
    Yes, please. I'd like to know how to do the next release. :-)

Until we decide on a different way forward, the default assumption would be 
that we stick with what we have. I am wondering, whether we should use 30 
minutes in the upcoming community call (next week) to discuss the Release cycle 
(in particular for the next release). A lot of progress was made on most of the 
blocked feature work at the summit, so I think covering the release model 
should be OK.

Regards
Lars


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel