[Xen-devel] [PATCH OSSTEST v3 2/2] Add a weekly coverity flight

[Ian Campbell <ian.campbell@xxxxxxxxxx>]

This primarily consists of ts-coverity-{build,upload} and
make-coverity-flight which constructs the sole job.

The branch is named "xen-unstable-coverity" which matches various xen*
in the cr-* scripts. Places which needed special treatement are
handled by matching xen-*-coverity, which leaves the possibility of
xen-4.7-testing-coverity etc in the future, but note that care would
be needed so that coverity's tracking of new vs existing issues would
likely be confused by uploading different branches without
differentiating somehow (I don't know how this is supposed to work).

The most recently scanned revision is pushed to a new
coverity-scanned/master branch in the usual xen.git, tests are run on
the master branch.

I initially thoughts that $c{CoverityEmail} would need to be an actual
account registered with scan, however a manual experiment using
email=security@xxxxxxx was accepted by the service. An "analysis
complete" message was sent to security@ while individual results mails
were sent to each member of the coverity project who was configured to
receive them. I think this is what we want. The "analysis complete"
mail contained no sensitive data, but also no real information other
than "success" (or presumably "failure" if that were to be the case).
I think going to security@ is probably OK.

The upload URL defaults to a dummy local URL, which will fail (it
would be possible in principal to put a stunt CGI there though). When
run with "cr-daily-branch --real" (i.e. in full on production mode)
then this is set instead to the value of CoverityUploadUrl from the
config (production-config etc). This means that adhoc and play runs
still exercise all the code (but the curl will fail) while --real runs
upload to a site-configurable location. (Note that the URL includes
the coverity project name, which would likely differ for different
instances).

I have run this via cr-daily-branch --real on the production infra
and it did upload as expected (flight 80516). Since
master==coverity-tested/master at this point it came out as a baseline
test which didn't attempt ap-push, which I would have expected to fail
anyway since it was running as my user in the colo which cannot push
to osstest@xenbits.

In my experiments the curl command took ~35 minutes to complete (rate
in the 100-200k range). Not sure if this is a problem, but use curl
--max-time passing it an hour to bound things. Note that curl is run
on the controller (via system_checked).  timeout etc.

Note that the token must be supplied with </path/to/token and not
@/path/to/token. The latter appears to the server as a file upload
rather than a text field in a form which doesn't work. In early
attempts I thought that the trailing \n in /path/to/token might be an
issue and hence wrote a big comment. However having discovered < vs @
I am no longer 100% sure that is the case, but I left the comment
anyway since I can observe on the wire that the \n is included in the
upload (but each test takes ~35 mins and there is a ratelimit on the
server side too).

A final niggle is that the descripton field in the web ui ends up as:
    80516:\ git://xenbits.xen.org/xen.git\ 
9937763265d9597e5f2439249b16d995842cdf0
(i.e. spaces are \ escaped). I've confirmed with curl --trace-ascii
the the uploaded data is not escaped (this is from an earlier attempt
which did not include the flight number):

009a: Content-Disposition: form-data; name="description"
00ce:
00d0: git://xenbits.xen.org/xen.git 9937763265d9597e5f2439249b16d99584
0110: 2cdf0f

Due to the limitations on the numbers of uploads I've not experimented
with possible fixes yet (e.g. URL escaping the upload). Worst case we
either live with it or adjust the syntax to avoid the problematic
characters.

Signed-off-by: Ian Campbell <ian.campbell@xxxxxxxxxx>
Cc: Andrew Cooper <andrew.cooper3@xxxxxxxxxx>
---

v2:
 - Split move of collect_xen_built_versions() into separate patch
 - Implemented support for coverity_upload = true (but don't yet set
   it)
 - Add host_hostflags to the job so it can actually run somewhere.
 - Call tsreadconfig() before referencing $r{coverity_upload} so that
   $r is actually populated.
 - use token=</path/to/token not token=@/path/to/token in curl
   arguments.
 - CoverityEmail == security@xxxxxxx seems to work fine.
 - Add $flight to description.

v3:
 - Use curl --max-time.
 - rename branch xen-unstable-coverity, allows switch from
   REVISION_COVERITY to REVISION_XEN which is more natureal as well as
   reducing the number of special cases needed. Also allows for the
   possibility xen-$other-coverity in the future if we desire (and
   there is scan bandwidth)
 - Control upload URL via daily-cron-setting-real and provide a dummy
   default, drop the "upload" runvar. This means that patch #3
   "make-coverity-flight: set coverity_upload=true" is no dropped.
 - Split into build and upload tests (but only a single job), needed
   to stash "built_version_xen" and "path_coverity" in runvars.
 - Don't install curl in ts-xen-build-prep, curl has been run on the
   controller for several revisions now (due to the need for
   the $c{CoverityToken} file).
 - Use "curl --fail" so 404 etc turn into actual failures

Deployment notes:
 - Put cov-analysis-linux64-7.7.0.4.tar.gz in the Images
   directory. DONE in COLO
 - Populate $HOME/.xen-osstest/coverity-secret with the token
   DONE in COLO
 - Push baseline to coverity-tested/master DONE
---
 ap-fetch-version         |  3 ++
 ap-fetch-version-old     |  3 ++
 ap-push                  |  4 +++
 cr-daily-branch          |  1 +
 cri-common               |  2 ++
 crontab                  |  1 +
 daily-cron-settings-real |  2 ++
 make-coverity-flight     | 56 +++++++++++++++++++++++++++++++++
 production-config        |  7 +++++
 sg-run-job               |  6 ++++
 ts-coverity-build        | 81 ++++++++++++++++++++++++++++++++++++++++++++++++
 ts-coverity-upload       | 54 ++++++++++++++++++++++++++++++++
 12 files changed, 220 insertions(+)
 create mode 100755 make-coverity-flight
 create mode 100755 ts-coverity-build
 create mode 100755 ts-coverity-upload

diff --git a/ap-fetch-version b/ap-fetch-version
index a7b658b..03ee754 100755
--- a/ap-fetch-version
+++ b/ap-fetch-version
@@ -53,6 +53,9 @@ xen-4.*-testing)
        repo_tree_rev_fetch_git xen \
                $TREE_XEN staging-$branchcore $LOCALREV_XEN
        ;;
+xen-unstable-coverity)
+       repo_tree_rev_fetch_git xen $TREE_XEN master $LOCALREV_XEN
+       ;;
 qemu-mainline)
        repo_tree_rev_fetch_git $branch \
                $TREE_QEMU_MAINLINE master $LOCALREV_QEMU_UPSTREAM
diff --git a/ap-fetch-version-old b/ap-fetch-version-old
index e2c6b3b..920b105 100755
--- a/ap-fetch-version-old
+++ b/ap-fetch-version-old
@@ -59,6 +59,9 @@ xen-4.*-testing)
        repo_tree_rev_fetch_git xen \
                $TREE_XEN stable-$branchcore $LOCALREV_XEN
        ;;
+xen-unstable-coverity)
+       repo_tree_rev_fetch_git xen $TREE_XEN coverity-tested/master 
$LOCALREV_XEN
+       ;;
 qemu-mainline)
         repo_tree_rev_fetch_git $branch \
                $BASE_TREE_QEMU_UPSTREAM upstream-tested $LOCALREV_QEMU_UPSTREAM
diff --git a/ap-push b/ap-push
index 8def652..eeff4a9 100755
--- a/ap-push
+++ b/ap-push
@@ -68,6 +68,10 @@ xen-*-testing)
        xenversion=${xenversion#xen-}
        git push $TREE_XEN $revision:refs/heads/stable-$xenversion
        ;;
+xen-unstable-coverity)
+       cd $repos/xen
+       git push $TREE_XEN $revision:refs/heads/coverity-tested/master
+       ;;
 qemu-mainline)
        cd $repos/qemu-mainline
        git push $TREE_QEMU_UPSTREAM $revision:refs/heads/upstream-tested
diff --git a/cr-daily-branch b/cr-daily-branch
index 364238c..8b7c789 100755
--- a/cr-daily-branch
+++ b/cr-daily-branch
@@ -259,6 +259,7 @@ fi
 
 case $branch in
 distros-*) makeflight=./make-distros-flight ;;
+xen-*-coverity)makeflight=./make-coverity-flight ;;
 *)         makeflight=./make-flight ;;
 esac
 
diff --git a/cri-common b/cri-common
index 6dfe8df..cdee48d 100644
--- a/cri-common
+++ b/cri-common
@@ -67,6 +67,8 @@ select_prevxenbranch () {
 select_xenbranch () {
        case "$branch" in
        xen-unstable-smoke)     tree=xen;       xenbranch=$branch; 
qemuubranch=qemu-upstream-unstable;;
+       xen-*-coverity)         tree=xen;       xenbranch=$branch; 
qemuubranch=qemu-upstream-${branch#xen-}
+                                                                  
qemuubranch=${qemuubranch%-coverity};;
        xen-*)                  tree=xen;       xenbranch=$branch ;;
        qemu-mainline)          tree=qemuu;     xenbranch=xen-unstable; 
qemuubranch=qemu-mainline;;
         qemu-upstream-*)    tree=qemuu; 
xenbranch=xen-${branch#qemu-upstream-};;
diff --git a/crontab b/crontab
index 09b8d14..f83e494 100755
--- a/crontab
+++ b/crontab
@@ -8,6 +8,7 @@ MAILTO=ian.jackson@xxxxxxxxxx,ian.campbell@xxxxxxxxxxxxx
 0              *       * * *           cd testing.git && 
BRANCHES=xen-unstable-smoke   ./cr-for-branches branches -q "./cr-daily-branch 
--real"
 4-59/30                *       * * *           cd testing.git &&               
                ./cr-for-branches branches -q "./cr-daily-branch --real"
 18             9       * * 1,3,5       cd testing.git && BRANCHES=linux-next   
        ./cr-for-branches branches -w "./cr-daily-branch --real"
+18             9       * * 7           cd testing.git && 
BRANCHES=xen-unstable-coverity ./cr-for-branches branches -w "./cr-daily-branch 
--real"
 18             4       * * *           cd testing.git && BRANCHES='linux-linus 
linux-mingo-tip-master linux-3.0 libvirt rumpuserxen' ./cr-for-branches 
branches -w "./cr-daily-branch --real"
 6-59/15        *       * * *           cd testing.git && 
EXTRA_BRANCHES='linux-linus linux-3.0 rumpuserxen libvirt' ./cr-for-branches 
bisects -w "./cr-try-bisect --real"
 #8-59/5                *       * * *           cd bisects/adhoc.git && 
with-lock-ex -q data-tree-lock bash -c "./cr-try-bisect-adhoc; exit $?"
diff --git a/daily-cron-settings-real b/daily-cron-settings-real
index b5032af..f1b4fbc 100644
--- a/daily-cron-settings-real
+++ b/daily-cron-settings-real
@@ -1,2 +1,4 @@
 : ${OSSTEST_HTML_SUBDIR:=results${OSSTEST_HTML_SUFFIX}}
 : ${OSSTEST_PUSH:=true}
+: ${OSSTEST_COVERITY_URL:=`getconfig CoverityUploadUrl`}
+export OSSTEST_COVERITY_URL
diff --git a/make-coverity-flight b/make-coverity-flight
new file mode 100755
index 0000000..9cba9b7
--- /dev/null
+++ b/make-coverity-flight
@@ -0,0 +1,56 @@
+#!/bin/bash
+
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2015 Citrix Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+
+set -e -o posix
+
+branch=$1
+xenbranch=$2
+blessing=$3
+buildflight=$4
+
+flight=`./cs-flight-create $blessing $branch`
+
+. ./cri-common
+. ./ap-common
+. ./mfi-common
+
+# daily-cron-settings-real will have set this to $c{CoverityUploadUrl}
+# in real runs. .
+: ${OSSTEST_COVERITY_URL:=http://localhost/xen-osstest/stunt-coverity}
+
+defsuite=`getconfig DebianSuite`
+
+arch=amd64
+suite=$defsuite
+
+build_hostflags=share-build-$suite-$arch,arch-$arch,suite-$suite,purpose-build
+
+./cs-job-create $flight coverity-$arch coverity \
+       arch=$arch host_hostflags=$build_hostflags \
+       tree_xen=$TREE_XEN \
+       revision_xen=$REVISION_XEN \
+       coverity_submit_url=${OSSTEST_COVERITY_URL}
+
+echo $flight
+
+# Local variables:
+# mode: sh
+# sh-basic-offset: 2
+# indent-tabs-mode: nil
+# End:
diff --git a/production-config b/production-config
index f2f0584..410893e 100644
--- a/production-config
+++ b/production-config
@@ -100,6 +100,13 @@ TftpGrubVersion XXXX-XX-XX
 XenUsePath /usr/groups/xencore/systems/bin/xenuse
 XenUseUser osstest
 
+# Results might include potential vulnerabilities.
+CoverityEmail security@xxxxxxx
+# This is only read from daily-cron-settings-real, everything else
+# gets the default/dummy path
+CoverityUploadUrl https://scan.coverity.com/builds?project=XenProject
+CoverityTools cov-analysis-linux64-7.7.0.4.tar.gz
+
 # We use the IP address because Citrix can't manage reliable nameservice
 #DebianMirrorHost debian.uk.xensource.com
 #DebianMirrorHost 10.80.16.196
diff --git a/sg-run-job b/sg-run-job
index 20ebb64..3e0f966 100755
--- a/sg-run-job
+++ b/sg-run-job
@@ -445,6 +445,12 @@ proc prepare-build-host {} {
     run-ts . host-build-prep ts-xen-build-prep
 }
 
+proc need-hosts/coverity {} { return BUILD }
+proc run-job/coverity {} {
+    run-ts . = ts-coverity-build + host
+    run-ts . = ts-coverity-upload + host
+}
+
 #---------- main program ----------
 
 jobdb::set-flight
diff --git a/ts-coverity-build b/ts-coverity-build
new file mode 100755
index 0000000..1d8bd0c
--- /dev/null
+++ b/ts-coverity-build
@@ -0,0 +1,81 @@
+#!/usr/bin/perl -w
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2015 Citrix Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+use strict qw(vars);
+use DBI;
+use Osstest;
+use File::Path;
+use POSIX;
+use Osstest::TestSupport;
+use Osstest::BuildSupport;
+
+tsreadconfig();
+selectbuildhost(\@ARGV);
+# remaining arguments are passed as targets to "make"
+builddirsprops();
+
+sub checkout () {
+    prepbuilddirs();
+
+    build_clone($ho, 'xen', $builddir, 'xen');
+}
+
+sub covtools () {
+    target_putfile($ho, 100, "$c{Images}/$c{CoverityTools}", 
"$builddir/covtools.tar.gz");
+    target_cmd($ho, <<END, 100);
+set -xe
+c=$builddir/covtools
+mkdir -p \$c
+cd \$c
+tar --strip-components=1 -xaf $builddir/covtools.tar.gz
+END
+}
+
+sub build () {
+    my $make = "make $makeflags";
+
+    # Pre build things we don't want coverity to scan, but which are
+    # normally built by some other command.
+    target_cmd_build($ho, 1000, $builddir, <<END);
+cd $builddir/xen
+./configure
+$make -C tools/firmware/etherboot all
+$make mini-os-dir
+END
+
+    # Now the stuff we want coverity to look at. The resulting tarball
+    # must have "cov-int/" as a root directory.
+    target_cmd_build($ho, 9000, $builddir, <<END);
+cd $builddir/xen
+export PATH=$builddir/covtools/bin:\$PATH
+cov-build --dir $builddir/coverity/cov-int $make -C extras/mini-os/
+cov-build --dir $builddir/coverity/cov-int $make xen tools
+END
+
+    built_stash($ho, $builddir, 'coverity', 'coverity');
+
+    my $xen_version = target_cmd_output($ho, <<END, 30);
+    cd $builddir/xen
+    make xenversion
+END
+    store_runvar("built_version_xen", $xen_version);
+}
+
+checkout();
+covtools();
+build();
+collect_xen_built_versions();
diff --git a/ts-coverity-upload b/ts-coverity-upload
new file mode 100755
index 0000000..485a6a4
--- /dev/null
+++ b/ts-coverity-upload
@@ -0,0 +1,54 @@
+#!/usr/bin/perl -w
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2015 Citrix Inc.
+#
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+#
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+use strict qw(vars);
+use DBI;
+use Osstest;
+use File::Path;
+use POSIX;
+use Osstest::TestSupport;
+use Osstest::BuildSupport;
+
+tsreadconfig();
+selectbuildhost(\@ARGV);
+
+# This must contain exactly and only the token, for example there must
+# be no trailing "\n", otherwise it is included in the literal token,
+# which is then invalid.
+my $tokenfile = "$ENV{HOME}/.xen-osstest/coverity-secret";
+
+sub upload() {
+    my $dist = get_stashed("path_coverity", $job);
+    
+    my @form_args;
+    push @form_args, "token=\<$tokenfile";
+    push @form_args, "email=$c{CoverityEmail}";
+    push @form_args, "file=\@$dist";
+    push @form_args, "version=$r{built_version_xen}";
+    push @form_args, "description=$flight: $r{tree_xen} 
$r{built_revision_xen}";
+
+    my @args = map { ("--form", $_) } @form_args;
+
+    push @args, qw(--max-time 3600);
+    push @args, qw(--fail); # turn 404 etc into a failure.
+    push @args, $r{coverity_submit_url};
+
+    logm("curl args: ".(join " ", map { qq("$_") } @args));
+    system_checked("curl", @args);
+}
+
+upload();
-- 
2.6.1


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel