Re: [Xen-devel] automation: Creating a patchwork instance to improve pre-commit build testing

[George Dunlap <george.dunlap@xxxxxxxxxx>]

On 07/24/2018 12:23 PM, Lars Kurth wrote:
> 
> On 24/07/2018, 11:50, "Julien Grall" <julien.grall@xxxxxxx> wrote:
> 
>     Hi Lars,
>     
>     On 24/07/18 11:33, Lars Kurth wrote:
>     > 
>     > On 24/07/2018, 11:19, "Wei Liu" <wei.liu2@xxxxxxxxxx> wrote:
>     >      On Tue, Jul 24, 2018 at 04:04:05AM -0600, Jan Beulich wrote:
>     >      > I'm afraid my personal bar for any such automation is pretty
>     >      > high: There must not ever be any negative effect from such an
>     >      > addition. Positive effects would of course be very welcome. I
>     >      > realize this is an unrealistic goal, but it should at least come
>     >      > close (perhaps after some initial learning phase). But this 
> implies
>     >      > that at least in theory it is possible to come close in the first
>     >      > place, which I can't take for given with the information I've 
> been
>     >      > provided so far.
>     >      
>     >      Then I'm afraid the only suggestion I get for you at the moment is 
> to
>     >      add a filter to dump those emails to /dev/null -- you already 
> realised
>     >      that's an unrealistic goal (at least at the beginning).
>     >      
>     >      Wei.
>     >      
>     > First of all, there should only be mail (aka spam) if there was a 
> failure.
>     
>     This seems a little strange to only send e-mail on failure. How do you 
>     differentiate between the bot has successfully tested that series and 
>     the series is still in queue then?
>     
> Yes, that would be a trade-off to minimize "spam"
> 
> It seems to me there are a number of options we have and thus some decisions
> that need to be made.
>  
> 1: Do we trigger a CI cycle for *every* patch?

In a world with infinite resources, yes, because we want to detect
broken bisections.  My guess is that this would be too
resource-intensive for the real world.

No matter what, I'd prefer only one email per series; Definitely *don't*
want a success email for every patch.

> 2: Do we have an opt-in or op-out (e.g. through a tag, a specific CC, etc.) 
> for patches

Opt-out.

> 3: Do we report results back to xen-devel or to a separate list
> Looking at Linux 0 day, they report failures to a separate list - see 
> https://lists.01.org/pipermail/kbuild-all/2018-July/thread.html
> They also only seem to report failures
> 
> I am not quite sure what QEMU does. But I can't see any bot messages on their 
> list archives
[snip]
> 5: Do we report back on success or only on failure?
> See question by Julien

I'd start with having the bot respond to 00/NN exactly once, both on
success and failure.

> 4: Who else, besides the author should get a mail
> The patch submitters should definitely get a mail, the question is whether 
> people on the CC list should also get one

I think the bot should reply-to-all.  Maybe we can add an opt-out to our
website, so that the bot won't reply to you if you don't want it to.

> 6: What exactly do we report back
> Aka what is in the actual mail

A link to the git branch it created (if the patch applied), or a snippet
of the rejection message if it didn't.

Success / failure, with a link to a page containing the various tests
run, so people can see which one failed and investigate the failures.

 -George


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel