Re: [Xen-devel] osstest "short fast" tests of xen-unstable proposal

["Jan Beulich" <JBeulich@xxxxxxxx>]

>>> On 30.06.15 at 19:46, <Ian.Jackson@xxxxxxxxxxxxx> wrote:
> osstest's test coverage is improving, which means that the tests keep
> taking longer.  Even with ongoing expansion of the test facility, the
> overall time between a bad push and getting results can be quite long
> - currently perhaps 12-36h depending on circumstances.
> 
> It would be better if we could get quicker notification of build
> breakages and other kinds of very basic breakage.
> 
> I propose:
> 
> 
> Invent a new `xen-unstable-smoke' flight (in osstest terminology, a
> `branch').
> 
> This would be a push gate for xen.git.  Its input would be
> xen.git#staging.  Its output would be a new branch, xen.git#smoked.
> The existing `xen-unstable' flights would take xen.git#smoked as
> input.  So there would be a two-stage push gate,
> 
>  staging -[xen-unstable-smoke test]-> smoked -[xen-unstable test]-> master
> 
> This would apply to xen-unstable only, not to stable braches, nor to
> any other codebase (eg, qemu or Linux).  It would aim to run every 2h.
> 
> Arrangements would be made to reuse the outputs of most recent builds
> of qemuu and the currently favoured Linux branch.[5]
> 
> If any test failed, the flight would be automatically aborted and
> report immediately [6].  Unlike most flights, tests would not be
> `sticky' to failing hosts, nor (when succeding) prefer hosts which
> they hadn't run on for a while [6], so they would take the first
> available machine.
> 
> 
> The new flights would contain only:
>                                                  x86    ARM  [1]
> 
>   build-amd64                                    800s
>   build-armhf                                           800s
> 
>   test-amd64-amd64-xl                           2300s
>   test-armhf-armhf-xl                                  2500s
> 
>   test-amd64-amd64-xl-qemuu-debianhvm-i386 [2]  3200s
> 
>                                                 -----  -----
>                                                 6300s  3300s
> 
> 
> Total machine resources for each run would be about 2h of x86 capacity
> and 1h of ARM capacity, with the infrastructure as currently
> configured.  We currently have 19 x86 machines live out of a planned
> 24, and 5 ARM machines out of a planned 8 [3].
> 
> The ARM machines are much less heavily used because they have a much
> smaller variety of tests (no HVM, no Windows, no RHEL, etc).
> 
> So thinking about x86, doing a test like this about every 2h would
> take ~5% of our current capacity, which would be well worthwhile.
> Doing it much more often would be tricky because of resource
> allocation delays: we have to wait for another test to finish, and we
> may need to reinstall a build box.[4]
> 
> 
> Comments welcome.

Yes, please.

Thanks, Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel