Re: [PATCH 06/12] RFC: automation: Add linux stubdom build and smoke test

[Marek Marczykowski-Górecki <marmarek@xxxxxxxxxxxxxxxxxxxxxx>]

On Fri, May 17, 2024 at 05:40:52PM -0700, Stefano Stabellini wrote:
> On Thu, 16 May 2024, Marek Marczykowski-Górecki wrote:
> > Add minimal linux-stubdom smoke test. It starts a simple HVM with
> > linux-stubdom. The actual stubdom implementation is taken from Qubes OS
> > and then stripped off Qubes-specific code. In particular, the remaining
> > code does _not_ support:
> >  - direct kernel boot (implemented by relaying on specific guest disk
> >    laying in Qubes OS)
> >  - graphical console (used Qubes GUI agent injected into
> >    stubdomain's qemu)
> >  - audio input/output (used Qubes audio agent inside stubdomain)
> >  - USB passthrough (used qrexec <-> usbip proxy inside stubdomain)
> >  - setting up DHCP server (assumes guest addressing used in Qubes OS)
> > 
> > For this smoke test, the relevant part is missing direct kernel boot, as
> > that's used in other smoke tests. Solve this by preparing disk image
> > with proper bootloader (grub) installed. Since the test script is
> > running on arm64 to control x86_64 box, it cannot (easily) install grub
> > directly. For this reason, prepare bootsector as part of the Xen build
> > (which runs on x86_64) and then prepend do the disk image during the
> > test (and adjust partitions table afterwards).
> 
> I am not an expert on this, but do you think it would be possible to use
> network boot and tftp instead of grub on emulated disk? That would not
> require us to build neither /grub-core.img nor build_domU_disk().

Honestly, I don't know. I guess I'd need at least dnsmasq in dom0, and
also iPXE for the domU (if not built already?). I can try for this test.
But a later test (the PCI one) connects a network card and dom0 can't
really setup own DHCP on that network. Additionally combining this with
vif network for PXE might be confusing down the road.

> I am trying to avoid grub-core.img and disk.img because I think direct
> kernel boot or network boot are easier to maintain and more similar to
> the other tests. If you see the ARM tests, they all use tftp boot.

The ARM ones boot as dom0less, where there is only one boot mode for the
system to start in. Here, we have two: xen+dom0 (which already
does network boot), and then domU started by dom0. The latter would
need either a separate DHCP server on a separate network (vif interface
in dom0 should be fine), or some other way to separate dom0/domU boot
mode.

That said, the stubdom used in Qubes does support direct kernel boot. It
is removed from this version, because it relies on specific disk layout
(it reserves xvdd for this purpose). But I do want to bring this
capability to the upstream version too at some point.

> > Signed-off-by: Marek Marczykowski-Górecki <marmarek@xxxxxxxxxxxxxxxxxxxxxx>
> > ---
> > The test is implemented using hardware runner, because some of the
> > further tests will require it (for example PCI passthrough with
> > stubdomain). But if there is strong desire to have stubdomain tested
> > inside qemu tests (to be included in patchew runs), it is probably an
> > option for this basic smoke test.
> 
> Thanks for this amazing work. This is a great start, we can see how to
> create more tests after merging this one.
> 
> 
> > For now I'm keeping stubdomain code (build and glue scripts) in separate
> > repository on my github account. This is far from ideal. What would be
> > preferred option? New repository on xenbits? Or add directly into
> > xen.git (stubdom directory)? Honestly, I'd rather avoid the latter, as
> > from packager point of view those are mostly separate beings (similar to
> > qemu, where many use distribution-provide one instead of the one bundled
> > with Xen) and it's convenient to not need to rebuild stubdomain on every
> > hypervisor change (like a security patch).
> 
> My suggestion is to create repositories under gitlab.com/xen-project

gitlab.com/xen-project/stubdom-dm-linux ?
Initially I can create the repository under people/marmarek/.

Is there any preference regarding git history? I see two options:
1. Preserve the current history, where there is a lot of qubes-specific
work and on top a bunch of commits making it not qubes-specific (this is
what is there now).
2. Start with fresh history and reference original repository (and the
commit id) in the initial commit message.

> > Another topic is QEMU version inside stubdomain. It needs to be a
> > separate build due to vastly different configure options, so I cannot
> > reuse the qemu binary built for dom0 (or distribution-provided one if
> > Xen is configured to use it). But also, at this moment qemu for
> > stubdomain needs few extra patches that are not upstream yet.
> > What should be the proper solution here (after upstreaming all the
> > patches)?
> 
> It is fine to use a different QEMU version. For now, I would suggest to
> keep the QEMU patches as patches to be applied, in the new repositories
> under gitlab.com/xen-project

Ok, makes sense.

> > Generally, I try to add tests early, even though there is still some
> > work to do for proper stubdomain integration into upstream Xen, so any
> > cleanups and future changes (like the CDROM libxl patches by Jason
> > Andryuk) can be made with more confidence and reduce risk of
> > regressions.
> > 
> > The patch is RFC only because of the stubdom repository location.
> > ---
> >  automation/build/alpine/3.19-arm64v8.dockerfile   |  2 +-
> >  automation/build/alpine/3.19.dockerfile           |  9 ++-
> >  automation/gitlab-ci/build.yaml                   |  3 +-
> >  automation/gitlab-ci/test.yaml                    |  8 +-
> >  automation/scripts/build                          | 12 ++-
> >  automation/scripts/qubes-x86-64.sh                | 87 +++++++++++++++-
> >  automation/tests-artifacts/alpine/3.19.dockerfile |  6 +-
> >  7 files changed, 123 insertions(+), 4 deletions(-)
> > 
> > diff --git a/automation/build/alpine/3.19-arm64v8.dockerfile 
> > b/automation/build/alpine/3.19-arm64v8.dockerfile
> > index 158cf465a9ff..12810f87ecc6 100644
> > --- a/automation/build/alpine/3.19-arm64v8.dockerfile
> > +++ b/automation/build/alpine/3.19-arm64v8.dockerfile
> > @@ -47,3 +47,5 @@ RUN apk --no-cache add \
> >    # qubes test deps
> >    openssh-client \
> >    fakeroot \
> > +  sfdisk \
> > +  e2fsprogs \
> > diff --git a/automation/build/alpine/3.19.dockerfile 
> > b/automation/build/alpine/3.19.dockerfile
> > index 0be6d7c85fe7..108284613987 100644
> > --- a/automation/build/alpine/3.19.dockerfile
> > +++ b/automation/build/alpine/3.19.dockerfile
> > @@ -49,3 +49,12 @@ RUN apk --no-cache add \
> >    pixman-dev \
> >    # livepatch-tools deps
> >    elfutils-dev \
> > +  # stubdom deps
> > +  dracut-core \
> > +  quilt \
> > +  gnupg \
> > +  libseccomp-dev \
> > +  glib-static \
> > +  gmp-dev \
> > +  mpc1-dev \
> > +  mpfr-dev \
> > diff --git a/automation/gitlab-ci/build.yaml 
> > b/automation/gitlab-ci/build.yaml
> > index b186289bbd82..783a0687ba34 100644
> > --- a/automation/gitlab-ci/build.yaml
> > +++ b/automation/gitlab-ci/build.yaml
> > @@ -323,9 +323,11 @@ alpine-3.19-rootfs-export:
> >    image: registry.gitlab.com/xen-project/xen/tests-artifacts/alpine:3.19
> >    script:
> >      - mkdir binaries && cp /initrd.tar.gz binaries/initrd.tar.gz
> > +    - cp /grub-core.img binaries/grub-core.img
> >    artifacts:
> >      paths:
> >        - binaries/initrd.tar.gz
> > +      - binaries/grub-core.img
> >    tags:
> >      - x86_64
> >  
> > @@ -353,6 +355,7 @@ alpine-3.19-gcc-debug:
> >    extends: .gcc-x86-64-build-debug
> >    variables:
> >      CONTAINER: alpine:3.19
> > +    STUBDOM_LINUX: y
> >  
> >  debian-stretch-gcc-debug:
> >    extends: .gcc-x86-64-build-debug
> > diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
> > index f62d426a8d34..80d10eb7f476 100644
> > --- a/automation/gitlab-ci/test.yaml
> > +++ b/automation/gitlab-ci/test.yaml
> > @@ -199,6 +199,14 @@ adl-pci-hvm-x86-64-gcc-debug:
> >      - *x86-64-test-needs
> >      - alpine-3.19-gcc-debug
> >  
> > +adl-stubdom-hvm-x86-64-gcc-debug:
> > +  extends: .adl-x86-64
> > +  script:
> > +    - ./automation/scripts/qubes-x86-64.sh stubdom-hvm 2>&1 | tee 
> > ${LOGFILE}
> > +  needs:
> > +    - *x86-64-test-needs
> > +    - alpine-3.19-gcc-debug
> > +
> >  zen3p-smoke-x86-64-gcc-debug:
> >    extends: .zen3p-x86-64
> >    script:
> > diff --git a/automation/scripts/build b/automation/scripts/build
> > index b3c71fb6fb60..f7c85b07a8d0 100755
> > --- a/automation/scripts/build
> > +++ b/automation/scripts/build
> > @@ -102,4 +102,16 @@ else
> >      # build Xen
> >      cp -r dist binaries/
> >      if [[ -f xen/xen ]] ; then cp xen/xen binaries/xen; fi
> > +
> > +    if [[ "${STUBDOM_LINUX}" == "y" ]]; then
> > +        git clone https://github.com/marmarek/qubes-vmm-xen-stubdom-linux 
> > -b for-upstream2
> > +        make -j$(nproc) -C qubes-vmm-xen-stubdom-linux get-sources
> > +        CFLAGS="-I${PWD}/dist/install/usr/local/include" \
> > +            LDFLAGS="-L${PWD}/dist/install/usr/local/lib" \
> > +            LD_LIBRARY_PATH="${PWD}/dist/install/usr/local/lib" \
> > +            PATH="${PWD}/dist/install/usr/local/bin:$PATH" \
> > +            make -j$(nproc) -C qubes-vmm-xen-stubdom-linux -f 
> > Makefile.stubdom all
> > +        cp qubes-vmm-xen-stubdom-linux/build/linux/arch/x86/boot/bzImage 
> > binaries/dist/install/usr/local/lib/xen/boot/qemu-stubdom-linux-kernel
> > +        cp qubes-vmm-xen-stubdom-linux/build/rootfs/stubdom-linux-rootfs 
> > binaries/dist/install/usr/local/lib/xen/boot/qemu-stubdom-linux-rootfs
> > +    fi
> 
> I think qemu-stubdom-linux-kernel and qemu-stubdom-linux-rootfs should
> be built as test-artifacts if they are not strictly dependent on the Xen
> version.

This is almost true - QEMU needs to be built for the specific Xen version,
since it still uses some of the unstable ABIs... In practice, it might
be possible to have test-artifact for each Xen version separately, but
it will be PITA about the time of bumping the interface version (like
XEN_DOMCTL_INTERFACE_VERSION).

> Otherwise, please create a separate new build job for linux stubdoms (do
> not reuse alpine-3.19-gcc-debug). Also I would prefer if these build
> instructions were on a separate file, separate from "build".

Ok.

> >  fi
> > diff --git a/automation/scripts/qubes-x86-64.sh 
> > b/automation/scripts/qubes-x86-64.sh
> > index 77cb0d45815d..fc73403dbadf 100755
> > --- a/automation/scripts/qubes-x86-64.sh
> > +++ b/automation/scripts/qubes-x86-64.sh
> > @@ -18,6 +18,37 @@ memory = 512
> >  vif = [ "bridge=xenbr0", ]
> >  disk = [ ]
> >  '
> > +domU_disk_path=
> > +
> > +### helper functions
> > +
> > +build_domU_disk() {
> > +    local kernel="$1"
> > +    local initrd="$2"
> > +    local rootfs="$3"
> > +    local output="$4"
> > +    local grubcfg="$rootfs/boot/grub2/grub.cfg"
> > +    local kernel_cmdline="root=/dev/xvda1 console=hvc0 earlyprintk=xen"
> > +
> > +    mkdir -p "$rootfs/boot/grub2"
> > +    cp "$kernel" "$rootfs/boot/vmlinuz"
> > +    echo "linux /boot/vmlinuz $kernel_cmdline" >> "$grubcfg"
> > +    if [ -n "$initrd" ]; then
> > +        cp "$initrd" "$rootfs/boot/initrd.img"
> > +        echo "initrd /boot/initrd.img" >> "$grubcfg"
> > +    fi
> > +    echo "boot" >> "$grubcfg"
> > +    size=$(du -sm "$rootfs")
> > +    size=${size%%  *}
> > +    # add 5M margin
> > +    size=$(( size + 5 ))
> > +    mke2fs -d "$rootfs" "$output.part1" ${size}m
> > +    cat "$rootfs/usr/lib/grub/i386-pc/boot_hybrid.img" 
> > binaries/grub-core.img > "$output"
> > +    # align for the partition 1 start (2048 sectors)
> > +    truncate -s $((2048 * 512)) "$output"
> > +    cat "$output.part1" >> "$output"
> > +    echo ",,linux,*" | sfdisk "$output"
> > +}
> >  
> >  ### test: smoke test & smoke test PVH
> >  if [ -z "${test_variant}" ] || [ "${test_variant}" = "dom0pvh" ]; then
> > @@ -116,6 +147,41 @@ until grep -q \"^domU Welcome to Alpine Linux\" 
> > /var/log/xen/console/guest-domU.
> >      sleep 1
> >  done
> >  "
> > +
> > +### test: stubdom-hvm
> > +elif [ "${test_variant}" = "stubdom-hvm" ]; then
> > +    passed="ping test passed"
> > +
> > +    domU_config='
> > +type = "hvm"
> > +name = "domU"
> > +memory = 512
> > +vif = [ "bridge=xenbr0", ]
> > +disk = [ "/srv/disk.img,format=raw,vdev=xvda" ]
> > +device_model_version = "qemu-xen"
> > +device_model_stubdomain_override = 1
> > +on_reboot = "destroy"
> > +# libxl configures vkb backend to be dom0 instead of the stubdomain, defer
> > +# changing that until there is consensus what to do about VGA output (VNC)
> > +vkb_device = 0
> > +'
> > +    domU_check="
> > +ifconfig eth0 192.168.0.2
> > +until ping -c 10 192.168.0.1; do
> > +    sleep 1
> > +done
> > +echo \"${passed}\"
> > +"
> > +    dom0_check="
> > +set +x
> > +until grep -q \"${passed}\" /var/log/xen/console/guest-domU.log; do
> > +    sleep 1
> > +done
> > +set -x
> > +echo \"${passed}\"
> > +"
> > +
> > +    domU_disk_path=/srv/disk.img
> >  fi
> >  
> >  # DomU
> > @@ -137,8 +203,17 @@ ${domU_check}
> >  chmod +x etc/local.d/xen.start
> >  echo "rc_verbose=yes" >> etc/rc.conf
> >  sed -i -e 's/^Welcome/domU \0/' etc/issue
> > -find . | fakeroot -i ../fakeroot-save cpio -H newc -o | gzip > 
> > ../binaries/domU-rootfs.cpio.gz
> >  cd ..
> > +if [ -n "$domU_disk_path" ]; then
> > +    build_domU_disk \
> > +        "binaries/bzImage" \
> > +        "" \
> > +        "rootfs" \
> > +        "binaries/disk.img"
> > +else
> > +    (cd rootfs; find . | fakeroot -i ../fakeroot-save cpio -H newc -o | 
> > gzip > ../binaries/domU-rootfs.cpio.gz)
> > +fi
> > +
> >  rm -rf rootfs
> >  
> >  # DOM0 rootfs
> > @@ -152,6 +227,9 @@ mkdir srv
> >  mkdir sys
> >  rm var/run
> >  cp -ar ../binaries/dist/install/* .
> > +if [ -n "$domU_disk_path" ]; then
> > +    cp ../binaries/disk.img "./$domU_disk_path"
> > +fi
> >  
> >  echo "#!/bin/bash
> >  
> > @@ -164,8 +242,9 @@ ifconfig eth0 up
> >  ifconfig xenbr0 up
> >  ifconfig xenbr0 192.168.0.1
> >  
> > -# get domU console content into test log
> > +# get domU (and possibly its stubdom) console content into test log
> >  tail -F /var/log/xen/console/guest-domU.log 2>/dev/null | sed -e 
> > \"s/^/(domU) /\" &
> > +tail -F /var/log/xen/console/guest-domU-dm.log 2>/dev/null | sed -e 
> > \"s/^/(domU-dm) /\" &
> >  tail -F /var/log/xen/qemu-dm-domU.log 2>/dev/null | sed -e \"s/^/(qemu-dm) 
> > /\" &
> >  xl -vvv create /etc/xen/domU.cfg
> >  ${dom0_check}
> > @@ -178,7 +257,9 @@ echo "XENCONSOLED_TRACE=all" >> etc/default/xencommons
> >  echo "QEMU_XEN=/bin/false" >> etc/default/xencommons
> >  mkdir -p var/log/xen/console
> >  cp ../binaries/bzImage boot/vmlinuz
> > -cp ../binaries/domU-rootfs.cpio.gz boot/initrd-domU
> > +if [ -r ../binaries/domU-rootfs.cpio.gz ]; then
> > +    cp ../binaries/domU-rootfs.cpio.gz boot/initrd-domU
> > +fi
> >  find . | fakeroot -i ../fakeroot-save cpio -H newc -o | gzip > 
> > ../binaries/dom0-rootfs.cpio.gz
> >  cd ..
> >  
> > diff --git a/automation/tests-artifacts/alpine/3.19.dockerfile 
> > b/automation/tests-artifacts/alpine/3.19.dockerfile
> > index 6d665daedfa4..cfb2cb30fb30 100644
> > --- a/automation/tests-artifacts/alpine/3.19.dockerfile
> > +++ b/automation/tests-artifacts/alpine/3.19.dockerfile
> > @@ -35,6 +35,8 @@ RUN \
> >    apk add pciutils && \
> >    apk add libelf && \
> >    apk add libdw && \
> > +  apk add grub-bios && \
> > +  apk add libseccomp && \
> >    \
> >    # Xen
> >    cd / && \
> > @@ -64,4 +66,6 @@ RUN \
> >    \
> >    # Create rootfs
> >    cd / && \
> > -  tar cvzf /initrd.tar.gz bin dev etc home init lib mnt opt root sbin usr 
> > var
> > +  tar cvzf /initrd.tar.gz bin dev etc home init lib mnt opt root sbin usr 
> > var && \
> > +  # Prepare boot sector for HVM disk
> > +  grub-mkimage -o /grub-core.img -O i386-pc -p '(hd0,msdos1)/boot/grub2' 
> > boot part_msdos ext2 linux biosdisk configfile normal
> > -- 
> > git-series 0.9.1
> > 


-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

Attachment: signature.asc
Description: PGP signature