Xen project Mailing List

Re: [Xen-devel] [PATCH OSSTEST v2 2/5] Handle osstest's own local push gate in non-master production instances

From: Ian Campbell <ian.campbell@xxxxxxxxxx>

Date: Mon, 20 Apr 2015 14:14:24 +0100

Delivery-date: Mon, 20 Apr 2015 13:14:48 +0000

List-id: Xen developer discussion <xen-devel.lists.xen.org>

On Wed, 2015-04-15 at 11:04 +0100, Ian Campbell wrote: > If the option is set then osstest branch flights proceed by merging, > the referenced branch into the local pretest branch and testing the > result. This turned out to not be quite right in practice. Since pretest is not updated, in the absence of anything actually being changed on each iteration, git would create a new merge, identical to the last time around but for the date, test that and push the result to production. Creating lots of little branches branches in the history. Not what we want at all. After some discussion IRL and drawing some pictures on a whiteboard here is a replacement version. I'm about to kick off an adhoc test on the Cambridge VM, but thought I'd best send this out while it was fresh in our heads... Ian. From 0ad809cacd931d68d92dd9dd0f8a94f7cb01c94d Mon Sep 17 00:00:00 2001 From: Ian Campbell <ian.campbell@xxxxxxxxxx> Date: Wed, 1 Apr 2015 13:50:50 +0100 Subject: [PATCH] Handle osstest's own local push gate in non-master production instances We want to arrange that the master XenProject instance continues to test its own pretest branch while any downstream instances will pickup changes from the master instance's production (i.e. tested) branch, which is published at git://xenbits.xen.org/osstest.git#master. We want to also be able to use local pretest for local changes (which may or may not get merged back upstream). Add a new configuration option OsstestUpstream which by default is "git://xenbits.xen.org/osstest.git master" and which is cleared to nothing on the master instance via production-config. If the option is not set then the existing behaviour is unchanged. If the option is set then osstest branch flights will still prefer to test the local pretest branch, but if nothing is pending there then it will proceed by merging the upstream branch into the local production branch and testing the result. This merge must be done: - in a clone not in the main testing.git in order to avoid inserting merge conflict markers into the active set of scripts. - in a non-bare repo because git merge requires it. $repos/osstest is a bare repo which we want to keep that way because using repo_tree_rev_fetch_git to fetch the remote branch is convenient. So we use $repos/osstest-merge as a temporary merge repo and reclone from the active local repo each time. All of this happens in ap-fetch-version. As part of this arrange that the result is always left in the ap-fetch branch of the for-osstest.git repo (even for existing cases) and the sha1 is produced as output. Resetting to that revision is handled by cr-daily-branch. If the merge fails then manual intervention (i.e. a manual merge and push to the _local_ pretest) will be required. Likewise if local pretest and local production have diverged manual intervention will be required. In ap-push we stop pushing to xenbits#master except for the master instance if an upstream is defined. At some point it might be useful to add a configuration option for where to push to but I don't have that requirement right now. ap-fetch-version-old requires no changes. Signed-off-by: Ian Campbell <ian.campbell@xxxxxxxxxx> --- v3: - Only merge from upstream if there is nothing pending locally. - Always update ap-fetch. v2: - Arrange for $OSSTEST_USE_HEAD=y to take precendence - drop LOCALREV (which was wrong anyway) in favour of inline branchname - Rename OSSTEST_REVISION_MERGE as revision_merge to avoid implying it can be set and will be honoured. - Git in Debian Squeeze lacks -C and --no-edit, adjust accordingly. --- Osstest.pm | 2 ++ ap-common | 2 ++ ap-fetch-version | 73 ++++++++++++++++++++++++++++++++++++++++++++++++++++--- ap-push | 8 +++++- production-config | 2 ++ 5 files changed, 82 insertions(+), 5 deletions(-) diff --git a/Osstest.pm b/Osstest.pm index 7989129..7f72bc6 100644 --- a/Osstest.pm +++ b/Osstest.pm @@ -102,6 +102,8 @@ sub readglobalconfig () { return if $readglobalconfig_done; $readglobalconfig_done=1; + $c{OsstestUpstream} = 'git://xenbits.xen.org/osstest.git master'; + $c{HostProp_DhcpWatchMethod} = 'leases dhcp3 infra:5556'; $c{AuthorizedKeysFiles} = ''; $c{AuthorizedKeysAppend} = ''; diff --git a/ap-common b/ap-common index 64749e3..de7bb1d 100644 --- a/ap-common +++ b/ap-common @@ -21,6 +21,8 @@ : ${XENBITS:=osstest@xxxxxxxxxxxxxxx} +: ${TREEBRANCH_OSSTEST_UPSTREAM=`getconfig OsstestUpstream`} + : ${TREE_XEN:=git://xenbits.xen.org/xen.git} : ${PUSH_TREE_XEN:=$XENBITS:/home/xen/git/xen.git} diff --git a/ap-fetch-version b/ap-fetch-version index 33aaf00..41486b3 100755 --- a/ap-fetch-version +++ b/ap-fetch-version @@ -90,12 +90,77 @@ ovmf) $TREE_OVMF_UPSTREAM master $LOCALREV_OVMF ;; osstest) - if [ "x$OSSTEST_USE_HEAD" != "xy" ] ; then - git fetch $HOME/testing.git pretest:ap-fetch >&2 - git rev-parse ap-fetch^0 + if [ "x$OSSTEST_USE_HEAD" = "xy" ] ; then + git update-ref -m "Arranging to test HEAD" \ + ap-fetch HEAD^0 + elif [ x"$TREEBRANCH_OSSTEST_UPSTREAM" != x ] ; then + # Deciding what to test here is somewhat complex because + # we would prefer to avoid octopus merges and to try and + # keep the change history as clean as reasonably possible. + # + # Therefore we prefer to test anything which is in local + # pretest before testing anything which is in + # upstream. Whether or not there is anything in local + # pretest is determined by using "git merge base" + + git fetch $HOME/testing.git +production:production + git fetch $HOME/testing.git +pretest:pretest + + local_production=$(git rev-parse production^0) + local_pretest=$(git rev-parse pretest^0) + + merge_base=$(git merge-base $local_production $local_pretest) + + echo >&2 "Considering merge from upstream" + echo >&2 "Local production: $local_production" + echo >&2 "Local pretest: $local_pretest" + echo >&2 "Merge base: $merge_base" + + if [ x$merge_base = x$local_pretest ] ; then + # local production is a ff of local pretest + # (i.e. there is nothing local to test). So merge from + # upstream master into production and test that. + + revision_merge=`repo_tree_rev_fetch_git osstest \ + $TREEBRANCH_OSSTEST_UPSTREAM daily-cron.$branch` + echo >&2 "Testing $TREEBRANCH_OSSTEST_UPSTREAM = $revision_merge" + + rm -rf $repos/osstest-merge >&2 + git clone -b production $HOME/testing.git $repos/osstest-merge >&2 + + ( + cd $repos/osstest-merge + git fetch $repos/osstest daily-cron.$branch:ap-merge >&2 + git merge \ + -m "Automerge of $TREEBRANCH_OSSTEST_UPSTREAM into production" \ + ap-merge >&2 + ) + + # Fetch the result of the above merge into local tree + # ready for use. cr-daily-branch handles reseting the + # working tree to this version + git fetch $repos/osstest-merge +production:ap-fetch + elif [ x$merge_base = x$local_production ] ; then + # there are commits in pretest which we should test in + # an attempt to progress production to this point. + echo >&2 "Testing local pretest" + git update-ref -m "Arranging to test local pretest" \ + ap-fetch $local_pretest + else + # It's too hard for to deal with automatically. This + # might happen if a push to pretest races with a + # successful flight which merges from upstream. We + # expect pushes to pretest to be rare in practice. + echo >&2 "************************************************" + echo >&2 "production and pretest have diverged." + echo >&2 "need manual fixup/rebase and push to pretest." + echo >&2 "************************************************" + fi else - git rev-parse HEAD^0 + git fetch $HOME/testing.git pretest:ap-fetch >&2 fi + + git rev-parse ap-fetch^0 ;; *) echo >&2 "branch $branch ?" diff --git a/ap-push b/ap-push index c141464..5967b42 100755 --- a/ap-push +++ b/ap-push @@ -99,7 +99,13 @@ ovmf) ;; osstest) git push $HOME/testing.git $revision:production - git push $XENBITS:/home/xen/git/osstest.git $revision:master + if [ x"$TREEBRANCH_OSSTEST_UPSTREAM" != x ] ; then + # could push to instance specific location, but + # certainly not to master instance's xenbits repo! + : + else + git push $XENBITS:/home/xen/git/osstest.git $revision:master + fi ;; *) echo >&2 "branch $branch ?" diff --git a/production-config b/production-config index 8f20c27..84a3bb7 100644 --- a/production-config +++ b/production-config @@ -15,6 +15,8 @@ # You should have received a copy of the GNU Affero General Public License # along with this program. If not, see <http://www.gnu.org/licenses/>. +# This is the master instance, therefore it has no upstream +OsstestUpstream '' JobDB Executive HostDB Executive -- 2.1.4 _______________________________________________ Xen-devel mailing list Xen-devel@xxxxxxxxxxxxx http://lists.xen.org/xen-devel

©2013 Xen Project, A Linux Foundation Collaborative Project. All Rights Reserved.
Linux Foundation is a registered trademark of The Linux Foundation.
Xen Project is a trademark of The Linux Foundation.