[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Xen-devel] [PATCH OSSTEST v2 2/5] Handle osstest's own local push gate in non-master production instances



On Wed, 2015-04-15 at 11:04 +0100, Ian Campbell wrote:
> If the option is set then osstest branch flights proceed by merging,
> the referenced branch into the local pretest branch and testing the
> result.

This turned out to not be quite right in practice. Since pretest is not
updated, in the absence of anything actually being changed on each
iteration, git would create a new merge, identical to the last time
around but for the date, test that and push the result to production.
Creating lots of little branches branches in the history. Not what we
want at all.

After some discussion IRL and drawing some pictures on a whiteboard here
is a replacement version.

I'm about to kick off an adhoc test on the Cambridge VM, but thought I'd
best send this out while it was fresh in our heads...

Ian.

From 0ad809cacd931d68d92dd9dd0f8a94f7cb01c94d Mon Sep 17 00:00:00 2001
From: Ian Campbell <ian.campbell@xxxxxxxxxx>
Date: Wed, 1 Apr 2015 13:50:50 +0100
Subject: [PATCH] Handle osstest's own local push gate in non-master production
 instances

We want to arrange that the master XenProject instance continues to
test its own pretest branch while any downstream instances will pickup
changes from the master instance's production (i.e. tested) branch,
which is published at git://xenbits.xen.org/osstest.git#master. We
want to also be able to use local pretest for local changes (which may
or may not get merged back upstream).

Add a new configuration option OsstestUpstream which by default is
"git://xenbits.xen.org/osstest.git master" and which is cleared to
nothing on the master instance via production-config.

If the option is not set then the existing behaviour is unchanged.

If the option is set then osstest branch flights will still prefer to
test the local pretest branch, but if nothing is pending there then it
will proceed by merging the upstream branch into the local production
branch and testing the result.

This merge must be done:
 - in a clone not in the main testing.git in order to avoid inserting
   merge conflict markers into the active set of scripts.
 - in a non-bare repo because git merge requires it.

$repos/osstest is a bare repo which we want to keep that way because
using repo_tree_rev_fetch_git to fetch the remote branch is
convenient.

So we use $repos/osstest-merge as a temporary merge repo and reclone
from the active local repo each time.

All of this happens in ap-fetch-version.

As part of this arrange that the result is always left in the ap-fetch
branch of the for-osstest.git repo (even for existing cases) and the
sha1 is produced as output. Resetting to that revision is handled by
cr-daily-branch.

If the merge fails then manual intervention (i.e. a manual merge and
push to the _local_ pretest) will be required. Likewise if local
pretest and local production have diverged manual intervention will be
required.

In ap-push we stop pushing to xenbits#master except for the master
instance if an upstream is defined. At some point it might be useful
to add a configuration option for where to push to but I don't have
that requirement right now.

ap-fetch-version-old requires no changes.

Signed-off-by: Ian Campbell <ian.campbell@xxxxxxxxxx>
---
v3:
 - Only merge from upstream if there is nothing pending locally.
 - Always update ap-fetch.

v2:
 - Arrange for $OSSTEST_USE_HEAD=y to take precendence
 - drop LOCALREV (which was wrong anyway) in favour of inline
   branchname
 - Rename OSSTEST_REVISION_MERGE as revision_merge to avoid implying
   it can be set and will be honoured.
 - Git in Debian Squeeze lacks -C and --no-edit, adjust accordingly.
---
 Osstest.pm        |  2 ++
 ap-common         |  2 ++
 ap-fetch-version  | 73 ++++++++++++++++++++++++++++++++++++++++++++++++++++---
 ap-push           |  8 +++++-
 production-config |  2 ++
 5 files changed, 82 insertions(+), 5 deletions(-)

diff --git a/Osstest.pm b/Osstest.pm
index 7989129..7f72bc6 100644
--- a/Osstest.pm
+++ b/Osstest.pm
@@ -102,6 +102,8 @@ sub readglobalconfig () {
     return if $readglobalconfig_done;
     $readglobalconfig_done=1;
 
+    $c{OsstestUpstream} = 'git://xenbits.xen.org/osstest.git master';
+
     $c{HostProp_DhcpWatchMethod} = 'leases dhcp3 infra:5556';
     $c{AuthorizedKeysFiles} = '';
     $c{AuthorizedKeysAppend} = '';
diff --git a/ap-common b/ap-common
index 64749e3..de7bb1d 100644
--- a/ap-common
+++ b/ap-common
@@ -21,6 +21,8 @@
 
 : ${XENBITS:=osstest@xxxxxxxxxxxxxxx}
 
+: ${TREEBRANCH_OSSTEST_UPSTREAM=`getconfig OsstestUpstream`}
+
 : ${TREE_XEN:=git://xenbits.xen.org/xen.git}
 : ${PUSH_TREE_XEN:=$XENBITS:/home/xen/git/xen.git}
 
diff --git a/ap-fetch-version b/ap-fetch-version
index 33aaf00..41486b3 100755
--- a/ap-fetch-version
+++ b/ap-fetch-version
@@ -90,12 +90,77 @@ ovmf)
                $TREE_OVMF_UPSTREAM master $LOCALREV_OVMF
        ;;
 osstest)
-       if [ "x$OSSTEST_USE_HEAD" != "xy" ] ; then
-           git fetch $HOME/testing.git pretest:ap-fetch >&2
-            git rev-parse ap-fetch^0
+        if [ "x$OSSTEST_USE_HEAD" = "xy" ] ; then
+           git update-ref -m "Arranging to test HEAD" \
+               ap-fetch HEAD^0
+       elif [ x"$TREEBRANCH_OSSTEST_UPSTREAM" != x ] ; then
+           # Deciding what to test here is somewhat complex because
+           # we would prefer to avoid octopus merges and to try and
+           # keep the change history as clean as reasonably possible.
+           #
+           # Therefore we prefer to test anything which is in local
+           # pretest before testing anything which is in
+           # upstream. Whether or not there is anything in local
+           # pretest is determined by using "git merge base"
+
+           git fetch $HOME/testing.git +production:production
+           git fetch $HOME/testing.git +pretest:pretest
+
+           local_production=$(git rev-parse production^0)
+           local_pretest=$(git rev-parse pretest^0)
+
+           merge_base=$(git merge-base $local_production $local_pretest)
+
+           echo >&2 "Considering merge from upstream"
+           echo >&2 "Local production: $local_production"
+           echo >&2 "Local pretest:    $local_pretest"
+           echo >&2 "Merge base:       $merge_base"
+
+           if [ x$merge_base = x$local_pretest ] ; then
+               # local production is a ff of local pretest
+               # (i.e. there is nothing local to test). So merge from
+               # upstream master into production and test that.
+
+               revision_merge=`repo_tree_rev_fetch_git osstest \
+                   $TREEBRANCH_OSSTEST_UPSTREAM daily-cron.$branch`
+               echo >&2 "Testing $TREEBRANCH_OSSTEST_UPSTREAM = 
$revision_merge"
+
+               rm -rf $repos/osstest-merge >&2
+               git clone -b production $HOME/testing.git $repos/osstest-merge 
>&2
+
+               (
+                   cd $repos/osstest-merge
+                   git fetch $repos/osstest daily-cron.$branch:ap-merge >&2
+                   git merge \
+                       -m "Automerge of $TREEBRANCH_OSSTEST_UPSTREAM into 
production" \
+                       ap-merge >&2
+               )
+
+               # Fetch the result of the above merge into local tree
+               # ready for use. cr-daily-branch handles reseting the
+               # working tree to this version
+               git fetch $repos/osstest-merge +production:ap-fetch
+           elif [ x$merge_base = x$local_production ] ; then
+               # there are commits in pretest which we should test in
+               # an attempt to progress production to this point.
+               echo >&2 "Testing local pretest"
+               git update-ref -m "Arranging to test local pretest" \
+                   ap-fetch $local_pretest
+           else
+               # It's too hard for to deal with automatically. This
+               # might happen if a push to pretest races with a
+               # successful flight which merges from upstream. We
+               # expect pushes to pretest to be rare in practice.
+               echo >&2 "************************************************"
+               echo >&2 "production and pretest have diverged."
+               echo >&2 "need manual fixup/rebase and push to pretest."
+               echo >&2 "************************************************"
+           fi
        else
-           git rev-parse HEAD^0
+           git fetch $HOME/testing.git pretest:ap-fetch >&2
        fi
+
+       git rev-parse ap-fetch^0
         ;;
 *)
        echo >&2 "branch $branch ?"
diff --git a/ap-push b/ap-push
index c141464..5967b42 100755
--- a/ap-push
+++ b/ap-push
@@ -99,7 +99,13 @@ ovmf)
        ;;
 osstest)
        git push $HOME/testing.git $revision:production
-       git push $XENBITS:/home/xen/git/osstest.git $revision:master
+       if [ x"$TREEBRANCH_OSSTEST_UPSTREAM" != x ] ; then
+           # could push to instance specific location, but
+           # certainly not to master instance's xenbits repo!
+           :
+       else
+           git push $XENBITS:/home/xen/git/osstest.git $revision:master
+       fi
        ;;
 *)
        echo >&2 "branch $branch ?"
diff --git a/production-config b/production-config
index 8f20c27..84a3bb7 100644
--- a/production-config
+++ b/production-config
@@ -15,6 +15,8 @@
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
+# This is the master instance, therefore it has no upstream
+OsstestUpstream ''
 
 JobDB Executive
 HostDB Executive
-- 
2.1.4




_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel


 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.