Re: [Xen-devel] [Notes for xen summit 2018 design session] Process changes: is the 6 monthly release Cadence too short, Security Process, ...

[Juergen Gross <jgross@xxxxxxxx>]

On 05/07/18 13:16, Ian Jackson wrote:
> Juergen Gross writes ("Re: [Xen-devel] [Notes for xen summit 2018 design 
> session] Process changes: is the 6 monthly release Cadence too short, 
> Security Process, ..."):
>> We didn't look at the sporadic failing tests thoroughly enough. The
>> hypercall buffer failure has been there for ages, a newer kernel just
>> made it more probable. This would have saved us some weeks.
> 
> In general, as a community, we are very bad at this kind of thing.

We should nominate someone taking care of that. This could be e.g.
the Release Manager (who will have to do that during the stabilization
period anyway). I'd be happy for other people helping out, obviously.

> In my experience, the development community is not really interested
> in fixing bugs which aren't directly in their way.
> 
> You can observe this easily in the way that regression in Linux,
> spotted by osstest, are handled.  Linux 4.9 has been broken for 43
> days.  Linux mainline is broken too.

Just sent the patch to stable repairing that issue.

Unfortunately I didn't spot the problem when sending the backports
of the patches for repairing the recent problems on AMD hardware: I
had specified kernel parameters in my tests avoiding the latest issues.

It took longer than I have hoped to find some time looking into the
problem due to ongoing security work and the spent time for release
related stuff.

> We do not have a team of people reading these test reports, and
> chasing developers to fix them.  I certainly do not have time to do
> this triage.  On trees where osstest failures do not block
> development, things go unfixed for weeks, sometimes months.

Maybe we should find an owner for each tree who will get the reports
directly and who is responsible for reaching out to the developers?
As said above I think the Release Manager is a possible owner of the
xen-unstable test tree.

> And overall my gut feeling is that tests which fail intermittently are
> usually blamed (even if this is not stated explicitly) on problems
> with osstest or with our test infrastructure.  It is easy for
> developers to think this because if they wait, the test will get
> "lucky", and pass, and so there will be a push and the developers can
> carry on.

Yes.

> I have a vague plan to sit down and think about how osstest's
> results analysers could respond better to intermittent failures.  The
> If I can, I would like intermittent failures to block pushes.  That
> would at least help address the problem of heisenbugs (which are often
> actually quite serious issues) not beint taken seriously.

+1

> I would love to hear suggestions for how to get people to actually fix
> test failures in trees not maintained by the Xen Project and therefore
> not gated by osstest.

In case nobody stands up to do it this will be quite difficult. One
option could be to drop the failing feature from Xen in case it isn't
an absolutely mandatory one. In case somebody really wants to keep that
feature he would have to act in order to repair it.


Juergen

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel