Re: [Xen-devel] [Notes for xen summit 2018 design session] Process changes: is the 6 monthly release Cadence too short, Security Process, ...

[Juergen Gross <jgross@xxxxxxxx>]

On 05/07/18 13:51, Andrew Cooper wrote:
> On 05/07/18 12:18, George Dunlap wrote:
>>
>>>>    Another potential problems showed up last week: OSSTEST is using the
>>>>    Debian servers for doing the basic installation. A change there (e.g.
>>>>    a new point release) will block tests. I'd prefer to have a local cache
>>>>    of the last known good set of *.deb files to be used especially for the
>>>>    branched Xen versions. This would rule out remote problems for releases.
>>>>
>>>> This is again something which we should definitely look at.
>>> This was bad luck.  This kind of update happens about 3-4 times a
>>> year.  It does break everything, leading to a delay of a day or two,
>>> but the fix is straightforward.
>>>
>>> Obviously this is not ideal but the solutions are nontrivial.  It is
>>> not really possible to "have a local cache of the last known good set
>>> of *.deb files" without knowing what that subset should be; that would
>>> require an edifice to track what is used, or some manual configuration
>>> which would probably break.  Alternatively we could run a complete
>>> mirror but that is a *lot* of space and bandwidth, most of which would
>>> be unused.
>>>
>>> I think the right approach is probably to switch from using d-i for
>>> host installs, to something like FAI.  That would be faster as well.
>>> However that amouns to reengineering the way osstest does host
>>> installs; it would also leave us maintaining an additional way to do
>>> host installs, since we would still want to be able to *test* d-i
>>> operation as a guest.
>> What I think would be ideal is a way to take ‘snapshots’ of different states 
>> of setup for various hosts and revert to them.  There’s absolutely no reason 
>> to do a full install of a host every osstest run, when that install happens 
>> 1) before we even install Xen, and 2) should be nearly identical each time.  
>> We should be able to install a host, take a snapshot of the “clean” install, 
>> then do the build prep, take a snapshot of that, and then simply revert to 
>> one or both of those (assuming build requirements haven’t changed in the 
>> mean time) whenever necessary.  Re-generating these snapshots once per week 
>> per host should be plenty, and sounds like it would massively improve the 
>> current throughput.
>>
>> I’d like to propose the idea also that we try to find a more efficient way 
>> of testing guest functionality than doing a guest install.  I understand 
>> it’s a natural way to test a reasonable range of functionality, but 
>> particularly for Windows guests, my impression is that it’s very slow; there 
>> must be a way to make a test that would have similar coverage but be able to 
>> be completed with a pre-installed snapshot, in only a few minutes.
> 
> We've had similar discussions in XenServer. That idea is superficially
> attractive but actually makes things worse, because it now means that
> filesystem clone/snapshot is now in the mix of things which can go wrong.
> 
> Particularly with OSSTest testing mainline kernels, rather than distro
> stable kernels, the chances of finding filesystem bugs grows
> substantially, and the complexity of diagnosing an issue is outside of
> our area of expertise.

But are really all tests required to use a freshly installed mainline
kernel? Can't we e.g. do a guest install test once a new kernel is
released and clone the resulting guest disk image for other tests
(after shutting down the guest, of course)?

Same applies to the host: the base system (without the to be tested
component like qemu, xen, or whatever) could be installed just by
cloning a disk/partition/logical volume.

Each image would run through the stages new->staging->stable:

- Each time a component is released an image is based on (e.g. a new
  mainline kernel) a new image is created by installing it. In case this
  succeeds, the image is moved to the staging area.
- The images in the staging area are tested using known stable
  components in order to test the image, not the test-components. In
  case of all tests succeeded the image is moved to the stable area.
- The stable images are used to test components from staging. In case of
  success the related components can be pushed to stable/master.


Juergen

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-devel