Re: [Xen-devel] [Notes for xen summit 2018 design session] Process changes: is the 6 monthly release Cadence too short, Security Process, ...

[Wei Liu <wei.liu2@xxxxxxxxxx>]

On Wed, Jul 04, 2018 at 03:26:16PM +0000, George Dunlap wrote:
> 
> 
> > On Jul 3, 2018, at 11:07 AM, Roger Pau Monné <roger.pau@xxxxxxxxxx> wrote:
> > 
> > On Mon, Jul 02, 2018 at 06:03:39PM +0000, Lars Kurth wrote:
> >> We then had a discussion around why the positive benefits didn't 
> >> materialize:
> >> * Andrew and a few other believe that the model isn't broken, but that the 
> >> issue is with how we 
> >>   develop. In other words, moving to a 9 months model will *not* fix the 
> >> underlying issues, but 
> >>   merely provide an incentive not to fix them.
> >> * Issues highlighted were:
> >>   * 2-3 months stabilizing period is too long
> > 
> > I think one of the goals with the 6 month release cycle was to shrink
> > the stabilizing period, but it didn't turn that way, and the
> > stabilizing period is quite similar with a 6 or a 9 month release
> > cycle.
> 
> Right, and I think this was something that wasn’t quite captured in Lars’ 
> summary.
> 
> Everyone agreed:
> 1. The expectation was that a shorter release cycle would lead to shorter 
> stabilization periods
> 2. This has not turned out to be the case, which means
> 3 At the moment, our “time doing development” to “time fixing bugs for a 
> release” ratio is far too low.
> 
> One option to fix #3 is to go back to a 9-month cycle (or even a
> 12-month cycle), which would increase the “development” part of the
> equation.

You get more changes in, you also get more bugs.  Assuming bugs are
introduced at a constant rate in relation to changes, moving back to 9
months won't help.

At least in my experience, a majority of time during the freeze is spent
on *waiting*. Waiting for osstest to turn around, waiting for security
issues to become public. Moving to 9 months won't change those factors.

A typical bug would need five working days (one week) to fix.

1. Someone report or osstest reports a bug. (Day 1)
2. Someone analyses it and writes a patch. (Day 2)
3. Someone reviews it. (Day 2 or 3).
4. Someone commits it. (Day 3 or 4).
5. Osstest produces test results (Day 3 to 5).

For a simple bug, we might finish 1-4 in one day. But we still need to
allow for at least two days to get a push.

In reality, a number of factors actually prolong getting things fixed
(in the sense that patches are pushed to master): 1. bug fixes are
incomplete; 2. hardware issues in test system; 3. other random hiccups.
Should any of these happens, another 2 to 3 days is required to get
patches pushed.

Osstest is really resource intense and heavy weight. We need to think of
a way to  reduce its turnaround time.  Or we can introduce other
auxiliary test systems to reduce its burden.

> 
> But Doug was advocating trying instead to attack the “time fixing
> bugs” part of the equation.  He said he was a big fan “continuous
> delivery” — of being *always* ready to release.  And I think there’s a
> fair amount of agreement that one of the reasons it takes so long to
> stabilize is that our testing isn’t reliably catching bugs for
> whatever reason.

I also think CD is a good idea.

Is there anything technical that stops us doing this? I don't think so.
We just need more automation. Maintainers push tags and tarballs are
automatically produced. Currently the process involves a lot of manual
work. I don't think that's a good use of people's time frankly.

> 
> So a fair amount of the discussion was about what it would look like,
> and what it would take, to make it such that almost any push from
> osstest (or whatever testing infrasctructure we went with) could
> reasonably be released, and would have a very low expectation of
> having extraneous bugs.

I would also like to advocate changing the mentality a bit. The current
mentality is that "we want to be reasonably sure there is low
expectation of bugs before we can release". Why not change to "we
release when we're sure there is definitely improvement in the tree
compared to last release"?

Wei.

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel