[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [Xen-devel] [PATCH osstest v4] Add libvirt build and test jobs.
I have run this through a "cr-daily-branch --real" and the push gate machinery seem functional. The test cases run through to starting a guest which fails because virsh support has not been implemented yet. The libvirt.git initscript is horribly redhat specific, and the debian package one is large and complex, so ts-libvirt-build inserts its own simple initscript. I have observed the autogen.sh phase taking 34 minutes on army (arm build cluster) hence the surprisingly large timeout of an hour. The build took around 20 minutes on the same system, but I used a timeout of an hour anyway to be safe. The standalone wrapper is updated to handle path_libvirt in set-paths. libvirt.git's autogen.sh involves initialising/cloning a git submodule, which doesn't use osstest's git cache and introduces unpredicability into what is built. Therefore ap-* are taught about a new libvirt-gnulib tree (prefixed libvirt incase we end up with other users of gnulib in the future). However this doesn't use the usual push gate mechanism, since this was deemed to be overkill and just always uses the tip of gnulib. The results are recorded though so the bisector can still operate over this tree in the event of breakage. Signed-off-by: Ian Campbell <ian.campbell@xxxxxxxxxx> Acked-by: Ian Jackson <ian.jackson@xxxxxxxxxxxxx> Cc: Jim Fehlig <jfehlig@xxxxxxxx> Cc: Dario Faggioli <dario.faggioli@xxxxxxxxxx> --- v4: Use "xen-tested-master" as the push gate output. Install extra libraries needed by the binary onto the target. Install build dependencies in ts-xen-build-prep instead of ts-libvirt-build. Although apt-get invocations are now locked I still have concerns: http://lists.xen.org/archives/html/xen-devel/2014-04/msg01951.html Clone gnulib separately. Avoid running stuff as root during build, no need Correct ap-fetch-version-old (actually use the gate, not the main libvirt tree) Increase timeouts for arm build. Ran through cr-daily-branch v3: Disable uml, lxc and vbox backends at build time Create test job on armhf as well as x86 Don't run non-libvirt tests in libvirt flights. v2: Add a test job, which will clearly fail. Make ts-xen-install DTRT though --- Osstest/TestSupport.pm | 8 ++- ap-common | 10 +++ ap-fetch-version | 8 +++ ap-fetch-version-old | 9 +++ ap-print-url | 6 ++ ap-push | 8 +++ cr-daily-branch | 12 ++++ cri-common | 1 + make-flight | 7 +++ mfi-common | 17 +++++- sg-run-job | 5 ++ standalone | 2 +- ts-libvirt-build | 162 +++++++++++++++++++++++++++++++++++++++++++++++++ ts-xen-build-prep | 3 + ts-xen-install | 7 ++- 15 files changed, 261 insertions(+), 4 deletions(-) create mode 100755 ts-libvirt-build diff --git a/Osstest/TestSupport.pm b/Osstest/TestSupport.pm index 65afc5b..99b313e 100644 --- a/Osstest/TestSupport.pm +++ b/Osstest/TestSupport.pm @@ -1776,7 +1776,13 @@ our %toolstacks= Command => 'xl', CfgPathVar => 'cfgpath', RestoreNeedsConfig => 1, - } + }, + 'libvirt' => { + NewDaemons => [qw(libvirtd)], + Dom0MemFixed => 1, + Command => 'virsh', + ExtraPackages => [qw(libnl1 libavahi-client3)], + }, ); sub toolstack () { diff --git a/ap-common b/ap-common index cb0cfff..998df34 100644 --- a/ap-common +++ b/ap-common @@ -33,6 +33,14 @@ : ${TREEVCS_LINUX:=git} +: ${TREE_LIBVIRT:=git://libvirt.org/libvirt.git} +: ${PUSH_TREE_LIBVIRT:=$XENBITS:/home/xen/git/libvirt.git} +: ${BASE_TREE_LIBVIRT:=git://xenbits.xen.org/libvirt.git} +: ${TREEVCS_LIBVIRT:=git} + +: ${TREE_LIBVIRT_GNULIB:=git://git.sv.gnu.org/gnulib.git} +: ${TREEVCS_LIBVIRT_GNULIB:=git} + : ${TREE_LINUXFIRMWARE:=git://xenbits.xen.org/osstest/linux-firmware.git} : ${PUSH_TREE_LINUXFIRMWARE:=$XENBITS:/home/osstest/ext/linux-firmware.git} : ${UPSTREAM_TREE_LINUXFIRMWARE:=$GIT_KERNEL_ORG/pub/scm/linux/kernel/git/firmware/linux-firmware.git} @@ -56,6 +64,8 @@ fi : ${LOCALREV_XEN:=daily-cron.$branch} : ${LOCALREV_LINUX:=daily-cron.$branch} +: ${LOCALREV_LIBVIRT:=daily-cron.$branch} +: ${LOCALREV_LIBVIRT_GNULIB:=daily-cron.$branch} : ${TREEBASE_LINUX_XCP:=http://hg.uk.xensource.com/carbon/trunk/linux-2.6.27} diff --git a/ap-fetch-version b/ap-fetch-version index aaf884d..4cbe84d 100755 --- a/ap-fetch-version +++ b/ap-fetch-version @@ -69,6 +69,14 @@ linuxfirmware) repo_tree_rev_fetch_git linuxfirmware \ $UPSTREAM_TREE_LINUXFIRMWARE master daily-cron.$branch ;; +libvirt) + repo_tree_rev_fetch_git libvirt \ + $TREE_LIBVIRT master $LOCALREV_LIBVIRT + ;; +libvirt-gnulib) + repo_tree_rev_fetch_git libvirt-gnulib \ + $TREE_LIBVIRT_GNULIB master $LOCALREV_LIBVIRT_GNULIB + ;; osstest) if [ "x$OSSTEST_USE_HEAD" != "xy" ] ; then git fetch $HOME/testing.git pretest:ap-fetch >&2 diff --git a/ap-fetch-version-old b/ap-fetch-version-old index 21ff786..661d1c0 100755 --- a/ap-fetch-version-old +++ b/ap-fetch-version-old @@ -27,6 +27,7 @@ select_xenbranch : ${BASE_TAG_LINUX2639:=tested/2.6.39.x} : ${BASE_LOCALREV_LINUX:=daily-cron.$branch.old} +: ${BASE_LOCALREV_LIBVIRT:=daily-cron.$branch.old} : ${BASE_TREE_QEMU_UPSTREAM:=${TREE_QEMU_UPSTREAM/\/staging\//\/}} @@ -73,6 +74,14 @@ linuxfirmware) repo_tree_rev_fetch_git linuxfirmware \ $TREE_LINUXFIRMWARE master daily-cron-old.$branch ;; +libvirt) + repo_tree_rev_fetch_git libvirt \ + $BASE_TREE_LIBVIRT xen-tested-master $BASE_LOCALREV_LIBVIRT + ;; +libvirt-gnulib) + # No push gate, same as ap-fetch-version + ./ap-fetch-version $branch + ;; osstest) if [ "x$OSSTEST_USE_HEAD" != "xy" ] ; then git fetch -f $HOME/testing.git incoming:ap-fetch diff --git a/ap-print-url b/ap-print-url index f2812d0..f05e0d1 100755 --- a/ap-print-url +++ b/ap-print-url @@ -46,6 +46,12 @@ linux-2.6.39) linuxfirmware) echo $UPSTREAM_TREE_LINUXFIRMWARE ;; +libvirt) + echo $TREE_LIBVIRT + ;; +libvirt-gnulib) + echo $TREE_LIBVIRT_GNULIB + ;; osstest) echo none:; ;; diff --git a/ap-push b/ap-push index dc0a166..c3359fc 100755 --- a/ap-push +++ b/ap-push @@ -32,6 +32,7 @@ select_xenbranch TREE_LINUX=$PUSH_TREE_LINUX TREE_QEMU_UPSTREAM=$XENBITS:/home/xen/git/qemu-upstream-${xenbranch#xen-}.git TREE_XEN=$PUSH_TREE_XEN +TREE_LIBVIRT=$PUSH_TREE_LIBVIRT if info_linux_tree "$branch"; then cd $repos/linux @@ -72,6 +73,13 @@ linux-xen-*) cd $repos/linux git push $TREE_LINUX $revision:refs/heads/$branch ;; +libvirt) + cd $repos/libvirt + git push $TREE_LIBVIRT $revision:xen-tested-master + ;; +libvirt-gnulib) + # No gate + exit 1 osstest) git push $HOME/testing.git $revision:incoming git push $XENBITS:/home/xen/git/osstest.git $revision:master diff --git a/cr-daily-branch b/cr-daily-branch index 02fef15..2f8a394 100755 --- a/cr-daily-branch +++ b/cr-daily-branch @@ -133,6 +133,14 @@ if [ "x$REVISION_LINUX_ARM" = x ]; then export REVISION_LINUX_ARM fi fi +if [ "x$REVISION_LIBVIRT" = x ]; then + determine_version REVISION_LIBVIRT libvirt + export REVISION_LIBVIRT +fi +if [ "x$REVISION_LIBVIRT_GNULIB" = x ]; then + determine_version REVISION_LIBVIRT_GNULIB libvirt-gnulib + export REVISION_LIBVIRT_GNULIB +fi if [ "x$REVISION_LINUXFIRMWARE" = x ]; then determine_version REVISION_LINUXFIRMWARE linuxfirmware export REVISION_LINUXFIRMWARE @@ -165,6 +173,10 @@ linuxfirmware) realtree=linuxfirmware NEW_REVISION=$REVISION_LINUXFIRMWARE ;; +libvirt) + realtree=libvirt + NEW_REVISION=$REVISION_LIBVIRT + ;; *) NEW_REVISION='' wantpush=false diff --git a/cri-common b/cri-common index 497d4e3..f472927 100644 --- a/cri-common +++ b/cri-common @@ -42,6 +42,7 @@ select_xenbranch () { linux) tree=linux; xenbranch=xen-unstable ;; linux-*) tree=linux; xenbranch=xen-unstable ;; linuxfirmware) tree=linuxfirmware; xenbranch=xen-unstable ;; + libvirt) tree=libvirt; xenbranch=xen-unstable ;; osstest) tree=osstest; xenbranch=xen-unstable ;; esac if [ "x$tree" = xlinux ]; then diff --git a/make-flight b/make-flight index 92b813a..aadf58b 100755 --- a/make-flight +++ b/make-flight @@ -64,6 +64,9 @@ job_create_test_filter_callback () { return 1;; esac ;; + libvirt) + if [ x$toolstack != xlibvirt ] ; then return 1; fi + ;; *) case "$job" in *-qemuu-*) @@ -238,6 +241,10 @@ test_matrix_do_one () { $xenarch $dom0arch \ $debian_runvars all_hostflags=$most_hostflags + job_create_test test-$xenarch$kern-$dom0arch-libvirt test-debian libvirt \ + $xenarch $dom0arch \ + $debian_runvars all_hostflags=$most_hostflags + # No further arm tests at the moment if [ $dom0arch = armhf ]; then return diff --git a/mfi-common b/mfi-common index 598aa02..503a1fc 100644 --- a/mfi-common +++ b/mfi-common @@ -162,6 +162,18 @@ create_build_jobs () { tree_linuxfirmware=$TREE_LINUXFIRMWARE \ revision_linuxfirmware=$REVISION_LINUXFIRMWARE + ./cs-job-create $flight build-$arch-libvirt build-libvirt \ + arch=$arch \ + tree_xen=$TREE_XEN \ + $RUNVARS $BUILD_RUNVARS $BUILD_LIBVIRT_RUNVARS $arch_runvars \ + $suite_runvars \ + host_hostflags=$build_hostflags \ + buildjob=${bfi}build-$arch \ + tree_libvirt=$TREE_LIBVIRT revision_libvirt=$REVISION_LIBVIRT\ + ${TREEVCS_LIBVIRT:+treevcs_libvirt=}${TREEVCS_LIBVIRT} \ + tree_libvirt_gnulib=$TREE_LIBVIRT_GNULIB revision_libvirt_gnulib=$REVISION_LIBVIRT_GNULIB\ + ${TREEVCS_LIBVIRT_GNULIB:+treevcs_libvirt_gnulib=}${TREEVCS_LIBVIRT_GNULIB} + case "$arch" in armhf) continue;; # don't do any other kernel builds esac @@ -193,6 +205,7 @@ job_create_test () { xenbuildjob="${bfi}build-$xenarch" buildjob="${bfi}build-$dom0arch" + tsbuildjob= case "$xenbranch:$toolstack" in xen-3.*-testing:*) ;; @@ -203,11 +216,13 @@ job_create_test () { *:xend) xenbuildjob="$xenbuildjob-xend" buildjob="${bfi}build-$dom0arch-xend" ;; + *:libvirt) tsbuildjob="libvirtbuildjob=$buildjob-libvirt" + ;; esac ./cs-job-create $flight $job $recipe toolstack=$toolstack \ $RUNVARS $TEST_RUNVARS $most_runvars \ - xenbuildjob=$xenbuildjob buildjob=$buildjob "$@" + xenbuildjob=$xenbuildjob buildjob=$buildjob $tsbuildjob "$@" } # Iterate over xenarch, dom0arch and kernel calling test_matrix_do_one diff --git a/sg-run-job b/sg-run-job index d336fbd..3d44331 100755 --- a/sg-run-job +++ b/sg-run-job @@ -312,6 +312,7 @@ proc test-guest-nomigr {g} { proc need-hosts/build {} { return BUILD } proc need-hosts/build-kern {} { return BUILD } +proc need-hosts/build-libvirt {} { return BUILD } proc run-job/build {} { run-ts . = ts-xen-build @@ -321,6 +322,10 @@ proc run-job/build-kern {} { run-ts . = ts-kernel-build } +proc run-job/build-libvirt {} { + run-ts . = ts-libvirt-build +} + proc prepare-build-host {} { global jobinfo run-ts broken = ts-hosts-allocate + host diff --git a/standalone b/standalone index 8d9d88a..6b08995 100755 --- a/standalone +++ b/standalone @@ -171,7 +171,7 @@ case $op in job=$1; shift - for d in '' xen kern ; do + for d in '' xen kern libvirt ; do runvar="path_${d}dist" path="build/${d}dist.tar.gz" if [ -f "logs/$flight/$job/$path" ] ; then diff --git a/ts-libvirt-build b/ts-libvirt-build new file mode 100755 index 0000000..c87dcba --- /dev/null +++ b/ts-libvirt-build @@ -0,0 +1,162 @@ +#!/usr/bin/perl +# This is part of "osstest", an automated testing framework for Xen. +# Copyright (C) 2009-2014 Citrix Inc. +# +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU Affero General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU Affero General Public License for more details. +# +# You should have received a copy of the GNU Affero General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. + +use strict qw(vars); +use DBI; +use Osstest; +use Osstest::TestSupport; + +tsreadconfig(); + +our ($whhost) = @ARGV; +$whhost ||= 'host'; +our $ho= selecthost($whhost); + +my $leaf= "build.$flight.$job"; +my $homedir = get_host_property($ho, 'homedir', '/home/osstest'); +my $builddir= "$homedir/$leaf"; +my $xendist= "$builddir/xendist"; + +sub libvirtd_init (); + +my $makeflags= get_host_property($ho, 'build make flags', '-j4'); + +sub checkout () { + target_cmd($ho, "rm -rf $builddir && mkdir $builddir && mkdir $xendist", 600); + + my $path = get_stashed("path_dist", $r{"buildjob"}); + my $distcopy= "$builddir/dist.tar.gz"; + target_putfile($ho, 300, $path, $distcopy); + target_cmd($ho, "tar -C $xendist -hzxf $distcopy", 300); + + build_clone($ho, 'libvirt', $builddir, 'libvirt'); + build_clone($ho, 'libvirt_gnulib', $builddir, 'libvirt-gnulib'); +} + +sub config() { + # Uses $GNULIB_SRCDIR because ./autogen.sh doesn't propagate + # --gnulib-srcdir to ./bootstap. + target_cmd_build($ho, 3600, $builddir, <<END); + cd libvirt + CFLAGS="-I$xendist/usr/local/include/" \\ + LDFLAGS="-L$xendist/usr/local/lib/ -Wl,-rpath-link=$xendist/usr/local/lib/" \\ + GNULIB_SRCDIR=$builddir/libvirt-gnulib \\ + ./autogen.sh --no-git \\ + --with-libxl --without-xen --without-xenapi --without-selinux \\ + --without-lxc --without-vbox --without-uml \\ + --sysconfdir=/etc --localstatedir=/var #/ +END +} + +sub build() { + target_cmd_build($ho, 3600, $builddir, <<END); + cd libvirt + (make $makeflags 2>&1 && touch ../build-ok-stamp) |tee ../log + test -f ../build-ok-stamp #/ + echo ok. +END +} + +sub install() { + target_cmd_build($ho, 300, $builddir, <<END); + mkdir -p dist + cd libvirt + make $makeflags install DESTDIR=$builddir/dist + mkdir -p $builddir/dist/etc/init.d +END + target_putfilecontents_stash($ho, 60, + libvirtd_init(), + $builddir.'/dist/etc/init.d/libvirtd'); + target_cmd_build($ho, 60, $builddir, <<END); + chmod +x $builddir/dist/etc/init.d/libvirtd +END +} + +sub collectversion_gnulib () { + store_revision($ho, 'libvirt_gnulib', "$builddir/libvirt-gnulib", 1); +} + +checkout(); +config(); +build(); +install(); +collectversion_gnulib(); +built_stash($ho, $builddir, 'dist', 'libvirtdist'); + +sub libvirtd_init () { + return <<'END'; +#! /bin/sh +# +# Init script for libvirtd +# +# Loosely based on Debian libvirt-bin initscript: +# (c) 2007 Guido Guenther <agx@xxxxxxxxxxx> +# based on the skeletons that comes with dh_make +# +### BEGIN INIT INFO +# Provides: libvirt-bin libvirtd +# Required-Start: $network $local_fs $remote_fs $syslog +# Required-Stop: $local_fs $remote_fs $syslog +# Should-Start: avahi-daemon cgconfig +# Should-Stop: avahi-daemon cgconfig +# Default-Start: 2 3 4 5 +# Default-Stop: 0 1 6 +# Short-Description: libvirt management daemon +### END INIT INFO + +PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin +export PATH +DAEMON=/usr/local/sbin/libvirtd +NAME=libvirtd +DESC="libvirt management daemon" + +test -x $DAEMON || exit 0 +. /lib/lsb/init-functions + +PIDFILE=/var/run/$NAME.pid + +case "$1" in + start) + [ "$VERBOSE" != no ] && log_daemon_msg "Starting $DESC" "$NAME" + start-stop-daemon --start --pidfile $PIDFILE \ + --exec $DAEMON -- -d $libvirtd_opts + [ "$VERBOSE" != no ] && log_end_msg $? + ;; + stop) + [ "$VERBOSE" != no ] && log_daemon_msg "Stopping $DESC" "$NAME" + start-stop-daemon --stop --pidfile $PIDFILE \ + --exec $DAEMON + [ "$VERBOSE" != no ] && log_end_msg $? + ;; + restart) + stop + start + ;; + reload|force-reload) + start-stop-daemon --stop --signal 1 --quiet --pidfile \ + /var/run/$NAME.pid --exec $DAEMON + ;; + *) + N=/etc/init.d/libvirtd + echo "Usage: $N {start|stop|restart|reload|force-reload}" >&2 + exit 1 + ;; +esac + +exit 0 +END +} diff --git a/ts-xen-build-prep b/ts-xen-build-prep index 528d0a4..a4e166e 100755 --- a/ts-xen-build-prep +++ b/ts-xen-build-prep @@ -172,6 +172,9 @@ sub prep () { libx11-dev git-core uuid-dev gettext gawk libsdl-dev libyajl-dev libaio-dev libpixman-1-dev libglib2.0-dev pkg-config + autoconf automake libtool xsltproc + libxml2-utils libxml2-dev libnl-dev + libdevmapper-dev w3c-dtd-xhtml ccache)); target_cmd_root($ho, "chmod -R a+r /usr/share/git-core/templates"); diff --git a/ts-xen-install b/ts-xen-install index 4a810e4..ad0254c 100755 --- a/ts-xen-install +++ b/ts-xen-install @@ -60,10 +60,15 @@ sub packages () { if ($r{arch} eq 'i386') { target_install_packages($ho, 'libc6-xen'); } + target_install_packages($ho, @{toolstack()->{ExtraPackages}}) + if toolstack()->{ExtraPackages}; } sub extract () { - foreach my $part ('', 'kern', 'xen') { + my @parts = ('', 'kern', 'xen'); + push @parts, 'libvirt' if $r{toolstack} eq "libvirt"; + + foreach my $part (@parts) { target_extract_jobdistpath($ho, $part, "path_${part}dist", $r{"${part}buildjob"}, \%distpath); } -- 1.9.0 _______________________________________________ Xen-devel mailing list Xen-devel@xxxxxxxxxxxxx http://lists.xen.org/xen-devel
|
Lists.xenproject.org is hosted with RackSpace, monitoring our |