Re: [Xen-devel] [PATCH 03/11] Introduce cirros tests

[Stefano Stabellini <sstabellini@xxxxxxxxxx>]

On Sun, 19 Mar 2017, Géza Gémes wrote:
> Add support for using cirros images in raisin tests
> 
> Signed-off-by: Géza Gémes <geza.gemes@xxxxxxxxx>
> ---
>  configs/config-cirros |  44 ++++++++++++++++++++++
>  defconfig             |   2 +
>  lib/common-tests.sh   | 102 
> ++++++++++++++++++++++++++++++++++++++++++++++++++
>  3 files changed, 148 insertions(+)
>  create mode 100644 configs/config-cirros
> 
> diff --git a/configs/config-cirros b/configs/config-cirros
> new file mode 100644
> index 0000000..fa2823e
> --- /dev/null
> +++ b/configs/config-cirros

The files under configs are meant to contain the git urls and software
versions of each component to build. It is not the right place for all
these cirros related variables.

Instead, I would move config-cirros to a new top level directory, maybe
I would call it "tests-configs".



> @@ -0,0 +1,44 @@
> +CIRROS_BASE_URL="https://download.cirros-cloud.net/";
> +CIRROS_VERSION="0.3.5"
> +
> +source `pwd`/lib/common-functions.sh
> +get_arch
> +case $RAISIN_ARCH in
> +    x86_64)
> +        CIRROS_ARCH=x86_64
> +        ;;
> +    x86_32)
> +        CIRROS_ARCH=i386
> +        ;;
> +    *)
> +        echo $PREPEND cirros tests only valid on x86, 32 or 64 bit
> +        exit 1
> +esac
> +
> +CIRROS_KERNEL_FILE=cirros-${CIRROS_VERSION}-${CIRROS_ARCH}-kernel
> +CIRROS_INITRD_FILE=cirros-${CIRROS_VERSION}-${CIRROS_ARCH}-initramfs
> +CIRROS_ROOTFS_FILE=cirros-${CIRROS_VERSION}-${CIRROS_ARCH}-rootfs.img
> +CIRROS_DISK_FILE=cirros-${CIRROS_VERSION}-${CIRROS_ARCH}-disk.img
> +CIRROS_KERNEL_URL=${CIRROS_BASE_URL}/${CIRROS_VERSION}/${CIRROS_KERNEL_FILE}
> +CIRROS_INITRD_URL=${CIRROS_BASE_URL}/${CIRROS_VERSION}/${CIRROS_INITRD_FILE}
> +CIRROS_ROOTFS_URL=${CIRROS_BASE_URL}/${CIRROS_VERSION}/${CIRROS_ROOTFS_FILE}.gz
> +CIRROS_DISK_URL=${CIRROS_BASE_URL}/${CIRROS_VERSION}/${CIRROS_DISK_FILE}
> +
> +CIRROS_GRUB_CFG="(xen/xvda,msdos1)/boot/grub/grub.cfg"
> +

I like to keep the config files as simple as possible, with just
parameters and variable assignments. Ideally, we would have one
config-cirros file with only varibles, like configs/config-master for
example. Everything else should be elsewhere. I would even go as far as
removing the get_arch call above, writing two fully static cirros config
files, one for x86_64 and one for x86_32.


> +set +e
> +QEMU_IMG=`which qemu-img`
> +set -e
> +if [[ -z "$QEMU_IMG" ]]
> +then
> +    QEMU_IMG="/usr/lib/xen/bin/qemu-img"
> +fi
> +
> +set +e
> +PVGRUB=`which grub-${CIRROS_ARCH}-xen`
> +set -e
> +if [[ -z "$PVGRUB" ]]
> +then
> +    PVGRUB="/usr/lib/xen/boot/grub-${CIRROS_ARCH}-xen"
> +fi

I would move the detection of QEMU_IMG and PVGRUB to functions in
lib/common-functions.sh.


> diff --git a/defconfig b/defconfig
> index f8ef398..111554e 100644
> --- a/defconfig
> +++ b/defconfig
> @@ -32,3 +32,5 @@ GIT_TRANSPORT="git"
>  ## All tests: busybox-pv busybox-hvm
>  ## ENABLED_TESTS is the list of test run by raise test
>  ENABLED_TESTS="busybox-pv busybox-hvm"
> +
> +. configs/config-cirros
> diff --git a/lib/common-tests.sh b/lib/common-tests.sh
> index d346af4..79815ce 100644
> --- a/lib/common-tests.sh
> +++ b/lib/common-tests.sh
> @@ -178,3 +178,105 @@ function get_host_initrd() {
>          exit 1
>      fi
>  }
> +
> +function cirros_network_init() {
> +    rootdir=$1
> +    # Configure static ip
> +    $SUDO sed -i -e 's/iface eth0 inet dhcp/iface eth0 inet static/' 
> ${rootdir}/etc/network/interfaces
> +    $SUDO sed -i -e '/iface eth0 inet static/a\    address 169.254.0.2' 
> ${rootdir}/etc/network/interfaces
> +    $SUDO sed -i -e '/address/a\    network 169.254.0.0' 
> ${rootdir}/etc/network/interfaces
> +    $SUDO sed -i -e '/network/a\    broadcast 169.254.0.255' 
> ${rootdir}/etc/network/interfaces
> +    $SUDO sed -i -e '/broadcast/a\    netmask 255.255.255.0' 
> ${rootdir}/etc/network/interfaces

I think that it would be more future-proof to just generate and
overwrite ${rootdir}/etc/network/interfaces with our own. Also in
general we don't use ${} for variables.


> +    # Disable cloud-init
> +    $SUDO rm -f ${rootdir}/etc/rc3.d/S*cirros*ds*
> +    $SUDO rm -f ${rootdir}/etc/rc3.d/S*-cirros-userdata
> +}
> +
> +function get_cirros_kernel() {
> +    bootdir=$1
> +    basename `find $bootdir -name vmlinuz* 2>/dev/null | head -1`
> +}
> +
> +function get_cirros_initrd() {
> +    bootdir=$1
> +    basename `find $bootdir -name initrd* 2>/dev/null | head -1`
> +}
> +
> +function cirros_grub_cfg() {
> +    rootdir=$1
> +    grubcfg="`echo $CIRROS_GRUB_CFG | cut -d ')' -f 2`"
> +    grubdir=`dirname $grubcfg`
> +    bootdir=`dirname $grubdir`
> +    tmpgrubcfg=`mktemp`
> +    cat > $tmpgrubcfg <<EOF
> +root='(xen/xvda,msdos1)'
> +insmod xzio
> +insmod gzio
> +insmod btrfs
> +insmod ext2
> +set timeout=1
> +set default=0
> +menuentry Cirros {
> +    linux `echo $bootdir`/`get_cirros_kernel ${rootdir}/${bootdir}` 
> root=/dev/xvda1 ro
> +    initrd `echo $bootdir`/`get_cirros_initrd ${rootdir}/${bootdir}`
> +}
> +EOF
> +    $SUDO mv $tmpgrubcfg ${rootdir}/${grubcfg}
> +}
> +
> +function set_up_cirros_tests() {
> +    verbose_echo "Setting up environment for cirros tests"
> +    tmpdir=`mktemp -d`

Reading later patches, I noticed that you rely on the variable being
called "tmpdir". If you need such as a variable please give a more
meaningful name.


> +    wget -q $CIRROS_KERNEL_URL -P $tmpdir
> +    wget -q $CIRROS_INITRD_URL -P $tmpdir
> +    wget -q $CIRROS_ROOTFS_URL -P $tmpdir
> +    wget -q $CIRROS_DISK_URL -P $tmpdir

Shouldn't we check whether they have been already downloaded? So that,
if you run 'raise tests' twice, the second time you don't need to wait?
In fact, I would download them to a "downloads" directory, rather than a
tmpdir.

Today, each busybox test setups its own disk and loop device. If
possible I would do the same for the cirros test. We can reuse already
downloaded images to save time and bandwidth. Of course, we can share
the same common init functions in all cirros tests. For example, in this
function we setup the disk for both pv and hvm tests. I would create two
separate functions, one for each.


> +    gunzip ${tmpdir}/${CIRROS_ROOTFS_FILE}.gz
> +    mv ${tmpdir}/${CIRROS_DISK_FILE} ${tmpdir}/${CIRROS_DISK_FILE}.qcow2
> +    $QEMU_IMG convert -f qcow2 -O raw ${tmpdir}/${CIRROS_DISK_FILE}.qcow2 
> ${tmpdir}/${CIRROS_DISK_FILE}
> +    CIRROS_ROOTFS_LOOP=`create_loop ${tmpdir}/${CIRROS_ROOTFS_FILE}`
> +    CIRROS_DISK_LOOP_P0=`$SUDO $BASEDIR/scripts/lopartsetup 
> ${tmpdir}/${CIRROS_DISK_FILE} | head -1 | cut -d ":" -f 1`
> +    CIRROS_ROOTFS_MNTPT=`mktemp -d`
> +    CIRROS_DISK_MNTPT=`mktemp -d`

local variables should be lower case and declared "local"


> +    $SUDO mount $CIRROS_ROOTFS_LOOP $CIRROS_ROOTFS_MNTPT
> +    $SUDO mount $CIRROS_DISK_LOOP_P0 $CIRROS_DISK_MNTPT
> +    cirros_network_init $CIRROS_ROOTFS_MNTPT
> +    cirros_network_init $CIRROS_DISK_MNTPT
> +    cirros_grub_cfg $CIRROS_DISK_MNTPT
> +    $SUDO umount $CIRROS_ROOTFS_MNTPT
> +    $SUDO umount $CIRROS_DISK_MNTPT
> +    $SUDO rmdir $CIRROS_ROOTFS_MNTPT
> +    $SUDO rmdir $CIRROS_DISK_MNTPT
> +    $SUDO losetup -d $CIRROS_ROOTFS_LOOP
> +    $SUDO losetup -d $CIRROS_DISK_LOOP_P0
> +}
> +
> +function tear_down_cirros_tests() {
> +    tmpdir=$1
> +    if [[ `$SUDO xl vm-list | grep -e "raisin-test" | wc -l` -gt 0 ]]
> +    then
> +        $SUDO xl destroy "raisin-test"
> +    fi
> +    number_of_cirros_tests=0
> +    for test in $TESTS
> +    do
> +        if [[ "`echo $test | cut -d '-' -f 1`" == "cirros" ]]
> +        then
> +            number_of_cirros_tests=$((number_of_cirros_tests+1))
> +        fi
> +    done
> +    number_of_run_cirros_tests=0
> +    for test in $TESTS
> +    do
> +        if [[ -f ${tmpdir}/${test}.cfg ]]
> +        then
> +            number_of_run_cirros_tests=$((number_of_run_cirros_tests+1))
> +        fi
> +    done
> +    if [[ $number_of_cirros_tests == $number_of_run_cirros_tests ]]
> +    then
> +        verbose_echo "Deleting environment of cirros tests"
> +        $SUDO rm -rf $tmpdir
> +    fi
> +}

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel