Re: [Xen-devel] [Notes for xen summit 2018 design session] Process changes: is the 6 monthly release Cadence too short, Security Process, ...

[George Dunlap <George.Dunlap@xxxxxxxxxx>]

> On Jul 5, 2018, at 8:53 AM, Wei Liu <wei.liu2@xxxxxxxxxx> wrote:
> 
> On Wed, Jul 04, 2018 at 03:26:16PM +0000, George Dunlap wrote:
>> 
>> 
>>> On Jul 3, 2018, at 11:07 AM, Roger Pau Monné <roger.pau@xxxxxxxxxx> wrote:
>>> 
>>> On Mon, Jul 02, 2018 at 06:03:39PM +0000, Lars Kurth wrote:
>>>> We then had a discussion around why the positive benefits didn't 
>>>> materialize:
>>>> * Andrew and a few other believe that the model isn't broken, but that the 
>>>> issue is with how we 
>>>>  develop. In other words, moving to a 9 months model will *not* fix the 
>>>> underlying issues, but 
>>>>  merely provide an incentive not to fix them.
>>>> * Issues highlighted were:
>>>>  * 2-3 months stabilizing period is too long
>>> 
>>> I think one of the goals with the 6 month release cycle was to shrink
>>> the stabilizing period, but it didn't turn that way, and the
>>> stabilizing period is quite similar with a 6 or a 9 month release
>>> cycle.
>> 
>> Right, and I think this was something that wasn’t quite captured in Lars’ 
>> summary.
>> 
>> Everyone agreed:
>> 1. The expectation was that a shorter release cycle would lead to shorter 
>> stabilization periods
>> 2. This has not turned out to be the case, which means
>> 3 At the moment, our “time doing development” to “time fixing bugs for a 
>> release” ratio is far too low.
>> 
>> One option to fix #3 is to go back to a 9-month cycle (or even a
>> 12-month cycle), which would increase the “development” part of the
>> equation.
> 
> You get more changes in, you also get more bugs.  Assuming bugs are
> introduced at a constant rate in relation to changes, moving back to 9
> months won't help.

Er, sorry — are you saying you think that the stabilization period we’ve had 
for 6-month releases *has* been shorter than the one we’ve had for 9-month 
releases?  Or are you saying they’re the same length, but due to some other 
reason (such as, we’ve been finding more bugs), and so if we go back to 9 
months it will now be *even longer* than it was before?

Bugs are found outside of the “stabilization” window; so going from 6 to 9 
months won’t result in 50% more bugs to find during the stabilization window — 
many of those will have been found during the normal course of development.

> 
> At least in my experience, a majority of time during the freeze is spent
> on *waiting*. Waiting for osstest to turn around, waiting for security
> issues to become public. Moving to 9 months won't change those factors.

Agreed, but rather than spending 3 months developing and 3 months stabilizing 
(as it has nearly been), we’d be spending 6 months developing and 3 months 
stabilizing.  

Obviously 7 months developing and 2 months stabilizing or 8 months developing 
and 1 month stabilizing would be even better, so doing the testing / automation 
improvements discussed would definitely be worthwhile.  But can we get all 
those improvements done in a reasonable amount of time?

Personally I much prefer the idea of doing 6-month releases.  But at the moment 
it seems clear to me that 1) it's causing a lot of extra overhead, and 2) we 
can’t fix the root causes in any reasonable time frame.  Given that the extra 
overhead will *also* distract us from fixing root causes, I think it makes 
sense to consider moving back to 9 months in the short term, and reconsidering 
once we’ve sorted out all our automation / testing issues.

> 
>> 
>> So a fair amount of the discussion was about what it would look like,
>> and what it would take, to make it such that almost any push from
>> osstest (or whatever testing infrasctructure we went with) could
>> reasonably be released, and would have a very low expectation of
>> having extraneous bugs.
> 
> I would also like to advocate changing the mentality a bit. The current
> mentality is that "we want to be reasonably sure there is low
> expectation of bugs before we can release". Why not change to "we
> release when we're sure there is definitely improvement in the tree
> compared to last release”?

I’m pretty sure most people consuming our releases care much more about bugs 
than about new features.  If *we* aren’t doing the testing to find some point 
that is reasonably bug-free, then who will?  Fedora, Centos, Debian certainly 
don’t have the resources for that, which means suddenly those become not viable 
platforms for  normal users anymore.  Citrix, SuSE, and Oracle have to go back 
to duplicating their own testing and having massive patchqueues on every 
release.  And where does that leave less well-funded projects, like QubesOS and 
OpenXT?

I think our bug-finding standard is about right.  It doesn’t need to be as high 
as for a commercial offering, but it certainly needs to be good enough for an 
average motivated user.

 -George
_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel