Re: [Xen-devel] [Notes for xen summit 2018 design session] Process changes: is the 6 monthly release Cadence too short, Security Process, ...

[Wei Liu <wei.liu2@xxxxxxxxxx>]

On Thu, Jul 05, 2018 at 09:28:07AM +0100, George Dunlap wrote:
> 
> 
> > On Jul 5, 2018, at 8:53 AM, Wei Liu <wei.liu2@xxxxxxxxxx> wrote:
> > 
> > On Wed, Jul 04, 2018 at 03:26:16PM +0000, George Dunlap wrote:
> >> 
> >> 
> >>> On Jul 3, 2018, at 11:07 AM, Roger Pau Monné <roger.pau@xxxxxxxxxx> wrote:
> >>> 
> >>> On Mon, Jul 02, 2018 at 06:03:39PM +0000, Lars Kurth wrote:
> >>>> We then had a discussion around why the positive benefits didn't 
> >>>> materialize:
> >>>> * Andrew and a few other believe that the model isn't broken, but that 
> >>>> the issue is with how we 
> >>>>  develop. In other words, moving to a 9 months model will *not* fix the 
> >>>> underlying issues, but 
> >>>>  merely provide an incentive not to fix them.
> >>>> * Issues highlighted were:
> >>>>  * 2-3 months stabilizing period is too long
> >>> 
> >>> I think one of the goals with the 6 month release cycle was to shrink
> >>> the stabilizing period, but it didn't turn that way, and the
> >>> stabilizing period is quite similar with a 6 or a 9 month release
> >>> cycle.
> >> 
> >> Right, and I think this was something that wasn’t quite captured in Lars’ 
> >> summary.
> >> 
> >> Everyone agreed:
> >> 1. The expectation was that a shorter release cycle would lead to shorter 
> >> stabilization periods
> >> 2. This has not turned out to be the case, which means
> >> 3 At the moment, our “time doing development” to “time fixing bugs for a 
> >> release” ratio is far too low.
> >> 
> >> One option to fix #3 is to go back to a 9-month cycle (or even a
> >> 12-month cycle), which would increase the “development” part of the
> >> equation.
> > 
> > You get more changes in, you also get more bugs.  Assuming bugs are
> > introduced at a constant rate in relation to changes, moving back to 9
> > months won't help.
> 
> Er, sorry — are you saying you think that the stabilization period
> we’ve had for 6-month releases *has* been shorter than the one we’ve
> had for 9-month releases?  Or are you saying they’re the same length,
> but due to some other reason (such as, we’ve been finding more bugs),
> and so if we go back to 9 months it will now be *even longer* than it
> was before?
> 

The stabilisation period for 6 months is actually shorter or of same
length proportional to development window -- barring any unpredictable
security issues.

> Bugs are found outside of the “stabilization” window; so going from 6
> to 9 months won’t result in 50% more bugs to find during the
> stabilization window — many of those will have been found during the
> normal course of development.

Right. Bugs are found outside of that stabilisation window, but bugs are
also introduced outside of the same window.  "Stabilisation" in Xen's
sense means no more new features. If we keep committing new features, we
get new bugs.

To be clear, I'm not against moving back to 9 months, but I'm not buying
the argument that moving to 9 months changes the pattern of
stabilisation period. We will still get bugs proportional to development
window.

> 
> > 
> > At least in my experience, a majority of time during the freeze is spent
> > on *waiting*. Waiting for osstest to turn around, waiting for security
> > issues to become public. Moving to 9 months won't change those factors.
> 

> Agreed, but rather than spending 3 months developing and 3 months
> stabilizing (as it has nearly been), we’d be spending 6 months
> developing and 3 months stabilizing.  

Well 6 months means 4 months development + 2 months stabilisation.
There has never been 3+3. So it is the same as 6+3 in terms of ratio.

> 
> Obviously 7 months developing and 2 months stabilizing or 8 months
> developing and 1 month stabilizing would be even better, so doing the
> testing / automation improvements discussed would definitely be
> worthwhile.  But can we get all those improvements done in a
> reasonable amount of time?
> 
> Personally I much prefer the idea of doing 6-month releases.  But at
> the moment it seems clear to me that 1) it's causing a lot of extra
> overhead, and 2) we can’t fix the root causes in any reasonable time
> frame.  Given that the extra overhead will *also* distract us from
> fixing root causes, I think it makes sense to consider moving back to
> 9 months in the short term, and reconsidering once we’ve sorted out
> all our automation / testing issues.

If we move back to 9 months. I think we should try 7+2.

> 
> > 
> >> 
> >> So a fair amount of the discussion was about what it would look like,
> >> and what it would take, to make it such that almost any push from
> >> osstest (or whatever testing infrasctructure we went with) could
> >> reasonably be released, and would have a very low expectation of
> >> having extraneous bugs.
> > 
> > I would also like to advocate changing the mentality a bit. The current
> > mentality is that "we want to be reasonably sure there is low
> > expectation of bugs before we can release". Why not change to "we
> > release when we're sure there is definitely improvement in the tree
> > compared to last release”?
> 

> I’m pretty sure most people consuming our releases care much more
> about bugs than about new features.  If *we* aren’t doing the testing
> to find some point that is reasonably bug-free, then who will?
> Fedora, Centos, Debian certainly don’t have the resources for that,
> which means suddenly those become not viable platforms for  normal
> users anymore.  Citrix, SuSE, and Oracle have to go back to
> duplicating their own testing and having massive patchqueues on every
> release.  And where does that leave less well-funded projects, like
> QubesOS and OpenXT?
> 
> I think our bug-finding standard is about right.  It doesn’t need to
> be as high as for a commercial offering, but it certainly needs to be
> good enough for an average motivated user.

"Improvement" doesn't mean new features only. Why not release when a
major security issue is fixed? Isn't that what downstream wants?

Wei.

> 
>  -George

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel