[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Xen-devel] [PATCH v3 6/7] osstest: add script for building custom mfsBSD images



In order to test FreeBSD HEAD we need to build a mfsBSD installer based on
the version that we want to test. This patch adds support for building such
mfsBSD image, and modifies the previously introduced ts-freebsd-host-install
script in order to use those custom mfsBSD images.

In order to build the images sources from two different repositories are
needed. The first ones of course are the FreeBSD sources which can be
fetched from the official github mirror at:

https://github.com/freebsd/freebsd.git

And the mfsBSD sources which can be downloaded from:

https://github.com/royger/mfsbsd.git

I'm using a custom fork because there's one patch needed for mfsBSD which
has not yet been merged into the master repository, once that's done we can
switch to:

https://github.com/mmatuska/mfsbsd.git

Anyway, we would need to have a local copy of them in order to have a push
gate like we do for Linux, Qemu or SeaBIOS.

The workflow of this would be:

1. Setup a FreeBSD host using a RELEASE image (10.1 right now).
2. Fetch upstream FreeBSD sources and compile a fresh FreeBSD world +
kernel.
3. Generate a mfsBSD image using the output from the FreeBSD build process.
4. Upload the mfsBSD image into the TFTP boot directory, the nomenclature
is: freebsd/mfsbsd-{git-hash}-{arch}.img.
5. Upload the install sets generated by the build process to an http server
accesible from the test hosts, the following hierarchy is used:
freebsd/{arch}/{git-hash}/
6. Use the newly created mfsBSD image in order to boot and the uploaded
install sets with zfsinstall.

Signed-off-by: Roger Pau Monnà <roger.pau@xxxxxxxxxx>
Cc: Ian Jackson <Ian.Jackson@xxxxxxxxxxxxx>
Cc: Ian Campbell <ian.campbell@xxxxxxxxxx>
---
 ts-freebsd-create-mfsbsd | 145 +++++++++++++++++++++++++++++++++++++++++++++++
 ts-freebsd-host-install  |  54 +-----------------
 2 files changed, 148 insertions(+), 51 deletions(-)
 create mode 100755 ts-freebsd-create-mfsbsd

diff --git a/ts-freebsd-create-mfsbsd b/ts-freebsd-create-mfsbsd
new file mode 100755
index 0000000..7c40520
--- /dev/null
+++ b/ts-freebsd-create-mfsbsd
@@ -0,0 +1,145 @@
+#!/usr/bin/perl -w
+# This is part of "osstest", an automated testing framework for Xen.
+# Copyright (C) 2009-2014 Citrix Inc.
+# 
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU Affero General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+# 
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU Affero General Public License for more details.
+# 
+# You should have received a copy of the GNU Affero General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+use strict qw(vars);
+use DBI;
+use POSIX;
+
+use Osstest;
+use Osstest::TestSupport;
+use Osstest::Logtailer;
+
+tsreadconfig();
+
+our ($whhost) = grep /host=/, @ARGV;
+$whhost ||= 'host';
+our $ho= selecthost($whhost);
+exit 0 if $ho->{Flags}{'no-reinstall'};
+exit 0 if $ho->{SharedReady};
+
+our %timeout= qw(ReadPreseed  350
+                 Sshd        2400);
+
+# TODO: this should be runvars
+our ($version) = grep /version=/, @ARGV;
+$version =~ s/version=//;
+logm("Generating mfsBSD image version: $version");
+
+our $freebsd_git = "https://github.com/freebsd/freebsd.git";;
+our $mfsbsd_git = "https://github.com/royger/mfsbsd.git";;
+
+our $public_html = "/home/rogerpau/public_html/";
+our $http_folder = "http://cbg-cloud-vm33.xs.citrite.net/~rogerpau/";;
+
+sub pkg_install {
+
+    logm("Installing git");
+    target_cmd_root($ho, <<END, 1000);
+        set -e
+        export ASSUME_ALWAYS_YES="YES"
+        pkg install -y git
+END
+}
+
+sub fetch_source {
+    my ($git_url, $id, $dest) = @_;
+
+    logm("Fetching FreeBSD sources from $git_url");
+    # TODO: this needs to be done using build_clone
+    target_cmd_root($ho, <<END, 4800);
+        chflags -R noschg $dest/*
+        rm -rf $dest
+        git clone $git_url $dest
+        cd $dest
+        git checkout $id
+END
+}
+
+sub build_freebsd {
+    my ($dest) = @_;
+
+    logm("Building FreeBSD wold");
+    target_cmd_root($ho, <<END, 4800);
+        set -e
+        cd $dest
+        make -j12 buildworld
+END
+
+    logm("Building FreeBSD kernel");
+    target_cmd_root($ho, <<END, 2400);
+        set -e
+        cd $dest
+        make -j12 buildkernel
+END
+
+    logm("Packing release files");
+    target_cmd_root($ho, <<END, 2400);
+        set -e
+        cd $dest/release
+        make ftp
+END
+}
+
+sub build_mfsbsd {
+    my ($freebsd_dest, $mfsbsd_dest, $id) = @_;
+
+    target_cmd_root($ho, <<END, 900);
+        set -e
+        cd $mfsbsd_dest
+        printf "%s" "-Dh" > conf/boot.config
+        cp conf/loader.conf.sample conf/loader.conf
+        cp conf/ttys.sample conf/ttys
+        sed -i .bak 's/std.[^"]*/std.$c{Baud}/' conf/ttys
+        cat <<ENDB >> conf/loader.conf
+mfsbsd.rootpw="root"
+boot_multicons="YES"
+boot_serial="YES"
+comconsole_speed="$c{Baud}"
+console="comconsole,vidconsole"
+boot_verbose="YES"
+ENDB
+END
+
+    target_cmd_root($ho, <<END, 2400);
+        cd $mfsbsd_dest
+        make BASE=$freebsd_dest/release/ftp RELEASE=$id PKG_STATIC=`which 
pkg-static`
+END
+}
+
+sub copy_sets {
+    my ($freebsd_dir, $mfsbsd_dir, $id) = @_;
+
+    target_getfile_root($ho, 2400, "$mfsbsd_dir/mfsbsd-$id-$r{arch}.img",
+        "$ho->{Tftp}{Path}/freebsd/mfsbsd-$id-$r{arch}.img");
+
+    ensuredir("$public_html/freebsd/$r{arch}/$id/");
+    target_getfile_root($ho, 2400, "$freebsd_dir/release/ftp/kernel.txz",
+        "$public_html/freebsd/$r{arch}/$id/kernel.txz");
+    target_getfile_root($ho, 2400, "$freebsd_dir/release/ftp/base.txz",
+        "$public_html/freebsd/$r{arch}/$id/base.txz");
+    target_getfile_root($ho, 2400, "$freebsd_dir/release/ftp/MANIFEST",
+        "$public_html/freebsd/$r{arch}/$id/MANIFEST");
+
+}
+
+pkg_install();
+
+fetch_source($freebsd_git, $version, "/root/freebsd");
+build_freebsd("/root/freebsd");
+fetch_source($mfsbsd_git, "HEAD", "/root/mfsbsd");
+build_mfsbsd("/root/freebsd", "/root/mfsbsd", $version);
+copy_sets("/root/freebsd", "/root/mfsbsd", $version);
diff --git a/ts-freebsd-host-install b/ts-freebsd-host-install
index 8ff9be5..4241e3d 100755
--- a/ts-freebsd-host-install
+++ b/ts-freebsd-host-install
@@ -39,40 +39,7 @@ our ($version) = grep /version=/, @ARGV;
 $version =~ s/version=//;
 logm("Installing FreeBSD version: $version");
 
-# List of FreeBSD mirrors, obtained from:
-# http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/mirrors-ftp.html
-# 13/08/2014
-our @ftp_hosts = qw(ftp.freebsd.org ftp1.freebsd.org ftp2.freebsd.org
-                    ftp3.freebsd.org ftp4.freebsd.org ftp5.freebsd.org
-                    ftp6.freebsd.org ftp7.freebsd.org ftp10.freebsd.org
-                    ftp11.freebsd.org ftp13.freebsd.org ftp14.freebsd.org
-                    ftp1.am.freebsd.org ftp.au.freebsd.org ftp2.au.freebsd.org
-                    ftp3.au.freebsd.org ftp.at.freebsd.org ftp2.br.freebsd.org
-                    ftp3.br.freebsd.org ftp4.br.freebsd.org ftp.ca.freebsd.org
-                    ftp2.ca.freebsd.org ftp.cn.freebsd.org ftp.cz.freebsd.org
-                    ftp2.cz.freebsd.org ftp.dk.freebsd.org ftp.ee.freebsd.org
-                    ftp.fi.freebsd.org ftp.fr.freebsd.org ftp1.fr.freebsd.org
-                    ftp3.fr.freebsd.org ftp5.fr.freebsd.org ftp6.fr.freebsd.org
-                    ftp7.fr.freebsd.org ftp8.fr.freebsd.org ftp.de.freebsd.org
-                    ftp1.de.freebsd.org ftp2.de.freebsd.org ftp4.de.freebsd.org
-                    ftp5.de.freebsd.org ftp7.de.freebsd.org ftp8.de.freebsd.org
-                    ftp.gr.freebsd.org ftp2.gr.freebsd.org ftp.hk.freebsd.org
-                    ftp3.ie.freebsd.org ftp.jp.freebsd.org ftp2.jp.freebsd.org
-                    ftp3.jp.freebsd.org ftp4.jp.freebsd.org ftp5.jp.freebsd.org
-                    ftp6.jp.freebsd.org ftp7.jp.freebsd.org ftp8.jp.freebsd.org
-                    ftp9.jp.freebsd.org ftp.kr.freebsd.org ftp2.kr.freebsd.org
-                    ftp.nl.freebsd.org ftp2.nl.freebsd.org ftp.nz.freebsd.org
-                    ftp.no.freebsd.org ftp.pl.freebsd.org ftp.ru.freebsd.org
-                    ftp2.ru.freebsd.org ftp4.ru.freebsd.org ftp5.ru.freebsd.org
-                    ftp6.ru.freebsd.org ftp.si.freebsd.org ftp.za.freebsd.org
-                    ftp2.za.freebsd.org ftp4.za.freebsd.org ftp.es.freebsd.org
-                    ftp3.es.freebsd.org ftp.se.freebsd.org ftp2.se.freebsd.org
-                    ftp.ch.freebsd.org ftp.tw.freebsd.org ftp2.tw.freebsd.org
-                    ftp4.tw.freebsd.org ftp5.tw.freebsd.org ftp.ua.freebsd.org
-                    ftp6.ua.freebsd.org ftp7.ua.freebsd.org ftp.uk.freebsd.org
-                    ftp2.uk.freebsd.org ftp3.uk.freebsd.org ftp4.uk.freebsd.org
-                    ftp5.uk.freebsd.org ftp1.us.freebsd.org ftp2.us.freebsd.org
-                    ftp3.us.freebsd.org ftp4.us.freebsd.org 
ftp5.us.freebsd.org);
+our $http_host = "http://cbg-cloud-vm33.xs.citrite.net/~rogerpau/freebsd/";;
 
 # TODO: this has to be set on a per-host basis.
 # It should probably come from $ho?
@@ -125,26 +92,11 @@ END
     }
     length($disk) or die "Unable to find a valid disk, exiting";
 
-    logm("Searching for a mirror");
-    foreach my $test_ftp (@ftp_hosts) {
-        logm("Probing mirror $test_ftp");
-        # Test from the target system.
-        my $output = target_cmd_output_root($ho,
-            'nc -zw 10 '.$test_ftp.' 21 >/dev/null 2>&1 && echo "yes" || echo 
"no"',
-            200);
-        if ($output eq "yes") {
-            logm("Found a valid mirror: $test_ftp");
-            $ftp = $test_ftp;
-            last;
-        }
-    }
-    length($ftp) or die "Unable to find a valid ftp mirror, exiting";
-
     logm('Install of base system using ZFS on root');
     target_cmd_root($ho, <<END,2400);
             set -e
             gpart destroy -F $disk || true
-            zfsinstall -d $disk -u 
ftp://$ftp/pub/FreeBSD/releases/$r{arch}/$version -s 4g
+            zfsinstall -d $disk -u $http_host/$r{arch}/$version -s 4g
 END
 
     logm('Setting up ssh and keys for the installed system');
@@ -218,7 +170,7 @@ END
 }
 
 sub setup_pxeboot_firstboot() {
-    my $mfs= 
'freebsd/'.$version.'/'.$r{arch}.'/'.'mfsbsd-'.$version.'-'.$r{arch}.'.img';
+    my $mfs= "freebsd/mfsbsd-$version-$r{arch}.img";
 
     logm('Using mfsBSD image: ' . $mfs);
 
-- 
1.9.3 (Apple Git-50)


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel

 


Rackspace

Lists.xenproject.org is hosted with RackSpace, monitoring our
servers 24x7x365 and backed by RackSpace's Fanatical Support®.