Xen project Mailing List

[PATCH v3 5/7] CI: avoid repacking initrd as part of the test job

From: Marek Marczykowski-Górecki <marmarek@xxxxxxxxxxxxxxxxxxxxxx>

Date: Fri, 11 Apr 2025 22:32:17 +0200

Cc: Andrew Cooper <andrew.cooper3@xxxxxxxxxx>, Marek Marczykowski-Górecki <marmarek@xxxxxxxxxxxxxxxxxxxxxx>, Doug Goldstein <cardoe@xxxxxxxxxx>, Stefano Stabellini <sstabellini@xxxxxxxxxx>

Delivery-date: Fri, 11 Apr 2025 20:33:37 +0000

Feedback-id: i1568416f:Fastmail

List-id: Xen developer discussion <xen-devel.lists.xenproject.org>

The prerequisite for this is building rootfs.cpio.gz, not only initrd.tar.gz. That's a change in the test-artifacts repository in the "Rootfs generation" series from Andrew. Having that, do not repack the whole initrd, but only pack modified files and rely on Linux handling of concatenated archives. This allows packing just test-related files (which includes the whole toolstack), instead of the whole initrd. DomU initrd handling is a bit more complicated thing. It's sent to the target host as part of the dom0 initrd. For now include prepared domU initrd as a whole in dom0's rootfs "overlay", which means compressing it (again) each job (this takes 3s). This can be later improved by splitting domU initrd into two parts (base + overlay) and concatenate them in target dom0 already. For this to be useful, test-artifacts would need to provide pre-made double-cpio version too (rootfs.cpio.gz wrapper in yet another cpio.gz as boot/domU-initrd). Since full initrd is not unpacked now when preparing domU (and dom0) rootfs, a couple of minor changes are needed to how they are prepared. This includes creating whole etc/issue file, instead of modifying existing one, and a need to create a couple directories. Finally, move adding "rc_verbose=yes" to /etc/rc.conf to initrd building. Signed-off-by: Marek Marczykowski-Górecki <marmarek@xxxxxxxxxxxxxxxxxxxxxx> --- Changes in v3: - adjust for Andrew's version of cpio rootfs artifacts - don't require double-cpio archive for domU initrd, instead add it via dom0 initrd as plain cpio to avoid re-compressing the same thing New in v2 Jobs xilinx-* are untested, as I don't have necessary access. --- automation/gitlab-ci/test.yaml | 6 +++-- automation/scripts/qemu-alpine-x86_64.sh | 9 +++----- automation/scripts/qemu-smoke-dom0-arm64.sh | 7 ++---- automation/scripts/qemu-smoke-dom0less-arm64.sh | 7 ++---- automation/scripts/qubes-x86-64.sh | 18 +++++++--------- automation/scripts/xilinx-smoke-dom0-x86_64.sh | 15 ++++++------- automation/scripts/xilinx-smoke-dom0less-arm64.sh | 14 +++++------- 7 files changed, 35 insertions(+), 41 deletions(-) diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml index 59a2de28c864..3e02bf0e4d21 100644 --- a/automation/gitlab-ci/test.yaml +++ b/automation/gitlab-ci/test.yaml @@ -11,7 +11,9 @@ - project: xen-project/hardware/test-artifacts job: linux-6.6.86-arm64 ref: master - - alpine-3.18-arm64-rootfs-export + - project: xen-project/hardware/test-artifacts + job: alpine-3.18-arm64-rootfs + ref: master - qemu-system-aarch64-6.0.0-arm64-export .arm32-test-needs: &arm32-test-needs @@ -22,7 +24,7 @@ job: linux-6.6.56-x86_64 ref: master - project: xen-project/hardware/test-artifacts - job: x86_64-rootfs-alpine-3.18 + job: alpine-3.18-x86_64-rootfs ref: master .qemu-arm64: diff --git a/automation/scripts/qemu-alpine-x86_64.sh b/automation/scripts/qemu-alpine-x86_64.sh index 89bdb4df7dac..6e0a67b4018c 100755 --- a/automation/scripts/qemu-alpine-x86_64.sh +++ b/automation/scripts/qemu-alpine-x86_64.sh @@ -28,16 +28,15 @@ cd initrd find . | cpio --create --format='newc' | gzip > ../initrd.cpio.gz cd .. -# initrd.tar.gz is Dom0 rootfs +# Dom0 rootfs - overlay on top of rootfs.cpio.gz mkdir -p rootfs cd rootfs -tar xvzf ../initrd.tar.gz mkdir proc mkdir run mkdir srv mkdir sys -rm var/run cp -ar ../dist/install/* . +mkdir -p root etc/local.d mv ../initrd.cpio.gz ./root cp ../bzImage ./root echo "name=\"test\" @@ -60,9 +59,9 @@ xl -vvv create -c /root/test.cfg " > etc/local.d/xen.start chmod +x etc/local.d/xen.start -echo "rc_verbose=yes" >> etc/rc.conf # rebuild Dom0 rootfs -find . |cpio -H newc -o|gzip > ../xen-rootfs.cpio.gz +cp ../rootfs.cpio.gz ../xen-rootfs.cpio.gz +find . |cpio -H newc -o|gzip >> ../xen-rootfs.cpio.gz cd ../.. cat >> binaries/pxelinux.0 << EOF diff --git a/automation/scripts/qemu-smoke-dom0-arm64.sh b/automation/scripts/qemu-smoke-dom0-arm64.sh index 51d037b0003e..86047ccd9e7b 100755 --- a/automation/scripts/qemu-smoke-dom0-arm64.sh +++ b/automation/scripts/qemu-smoke-dom0-arm64.sh @@ -29,13 +29,12 @@ cd .. mkdir -p rootfs cd rootfs -tar xvzf ../initrd.tar.gz mkdir proc mkdir run mkdir srv mkdir sys -rm var/run cp -ar ../dist/install/* . +mkdir -p etc/local.d root mv ../initrd.cpio.gz ./root cp ../Image ./root echo "name=\"test\" @@ -56,8 +55,8 @@ xl -vvv create -c /root/test.cfg " > etc/local.d/xen.start chmod +x etc/local.d/xen.start -echo "rc_verbose=yes" >> etc/rc.conf -find . |cpio -H newc -o|gzip > ../xen-rootfs.cpio.gz +cp ../rootfs.cpio.gz ../xen-rootfs.cpio.gz +find . |cpio -H newc -o|gzip >> ../xen-rootfs.cpio.gz cd ../.. # XXX QEMU looks for "efi-virtio.rom" even if it is unneeded diff --git a/automation/scripts/qemu-smoke-dom0less-arm64.sh b/automation/scripts/qemu-smoke-dom0less-arm64.sh index f72d20936181..b095a5008e30 100755 --- a/automation/scripts/qemu-smoke-dom0less-arm64.sh +++ b/automation/scripts/qemu-smoke-dom0less-arm64.sh @@ -117,13 +117,12 @@ cd .. # DOM0 rootfs mkdir -p rootfs cd rootfs -tar xzf ../binaries/initrd.tar.gz mkdir proc mkdir run mkdir srv mkdir sys -rm var/run cp -ar ../binaries/dist/install/* . +mkdir -p etc/local.d echo "#!/bin/bash @@ -142,8 +141,8 @@ xl network-attach 1 type=vif ${dom0_check} " > etc/local.d/xen.start chmod +x etc/local.d/xen.start -echo "rc_verbose=yes" >> etc/rc.conf -find . | cpio -H newc -o | gzip > ../binaries/dom0-rootfs.cpio.gz +cp ../binaries/rootfs.cpio.gz ../binaries/dom0-rootfs.cpio.gz +find . | cpio -H newc -o | gzip >> ../binaries/dom0-rootfs.cpio.gz cd .. # ImageBuilder diff --git a/automation/scripts/qubes-x86-64.sh b/automation/scripts/qubes-x86-64.sh index dd88a1398f2b..99c2ece304f0 100755 --- a/automation/scripts/qubes-x86-64.sh +++ b/automation/scripts/qubes-x86-64.sh @@ -167,13 +167,11 @@ if [ -n "$domU_check" ]; then # DomU mkdir -p rootfs cd rootfs - # fakeroot is needed to preserve device nodes in rootless podman container - fakeroot -s ../fakeroot-save tar xzf ../binaries/initrd.tar.gz mkdir proc mkdir run mkdir srv mkdir sys - rm var/run + mkdir -p etc/local.d echo "#!/bin/sh echo 8 > /proc/sys/kernel/printk @@ -181,26 +179,25 @@ echo 8 > /proc/sys/kernel/printk ${domU_check} " > etc/local.d/xen.start chmod +x etc/local.d/xen.start - echo "rc_verbose=yes" >> etc/rc.conf echo "domU Welcome to Alpine Linux 3.18 Kernel \r on an \m (\l) " > etc/issue - find . | fakeroot -i ../fakeroot-save cpio -H newc -o | gzip > ../binaries/domU-rootfs.cpio.gz + cp ../binaries/rootfs.cpio.gz ../binaries/domU-rootfs.cpio.gz + find . | cpio -H newc -o | gzip >> ../binaries/domU-rootfs.cpio.gz cd .. rm -rf rootfs fi -# DOM0 rootfs +# DOM0 rootfs - this will be an overlay over alpine's initrd mkdir -p rootfs cd rootfs -fakeroot -s ../fakeroot-save tar xzf ../binaries/initrd.tar.gz mkdir boot mkdir proc mkdir run mkdir srv mkdir sys -rm var/run +mkdir -p etc/local.d cp -ar ../binaries/dist/install/* . cp -ar ../binaries/tests . cp -a ../automation/scripts/run-tools-tests tests/ @@ -237,7 +234,6 @@ fi chmod +x etc/local.d/xen.start echo "$domU_config" > etc/xen/domU.cfg -echo "rc_verbose=yes" >> etc/rc.conf echo "XENCONSOLED_TRACE=all" >> etc/default/xencommons echo "QEMU_XEN=/bin/false" >> etc/default/xencommons mkdir -p var/log/xen/console @@ -245,7 +241,9 @@ cp ../binaries/bzImage boot/vmlinuz if [ -n "$domU_check" ]; then cp ../binaries/domU-rootfs.cpio.gz boot/initrd-domU fi -find . | fakeroot -i ../fakeroot-save cpio -H newc -o | gzip > ../binaries/dom0-rootfs.cpio.gz +# take base initrd and append test-specific files +cp ../binaries/rootfs.cpio.gz ../binaries/dom0-rootfs.cpio.gz +find . | cpio -H newc -o | gzip >> ../binaries/dom0-rootfs.cpio.gz cd .. diff --git a/automation/scripts/xilinx-smoke-dom0-x86_64.sh b/automation/scripts/xilinx-smoke-dom0-x86_64.sh index 4db249530823..52a0e05e3f1b 100755 --- a/automation/scripts/xilinx-smoke-dom0-x86_64.sh +++ b/automation/scripts/xilinx-smoke-dom0-x86_64.sh @@ -80,6 +80,7 @@ argo-exec -l -p 28333 -- /bin/echo " copy_dom0_files () { + mkdir -p root usr/local/lib cp "${WORKDIR}/binaries/xen-argo.ko" "root/" cp -ar "${WORKDIR}/binaries/lib/"* "usr/local/lib/" cp "${WORKDIR}/binaries/argo-exec" "usr/local/bin/" @@ -92,9 +93,8 @@ fi # Set up domU rootfs. mkdir -p rootfs cd rootfs -tar xzf ../binaries/initrd.tar.gz mkdir proc run srv sys -rm var/run +mkdir -p etc/local.d echo "#!/bin/sh set -x export LD_LIBRARY_PATH=/usr/local/lib @@ -102,23 +102,22 @@ PATH=/usr/local/bin:/usr/local/sbin:\$PATH ${DOMU_CMD} " > etc/local.d/xen.start chmod +x etc/local.d/xen.start -echo "rc_verbose=yes" >> etc/rc.conf echo "domU Welcome to Alpine Linux 3.18 Kernel \r on an \m (\l) " > etc/issue copy_domU_files -find . | cpio -H newc -o | gzip > ../binaries/domU-rootfs.cpio.gz +cp ../binaries/initrd.cpio.gz ../binaries/domU-rootfs.cpio.gz +find . | cpio -H newc -o | gzip >> ../binaries/domU-rootfs.cpio.gz cd .. rm -rf rootfs # Set up dom0 rootfs. mkdir -p rootfs cd rootfs -tar xzf ../binaries/initrd.tar.gz mkdir boot proc run srv sys -rm var/run cp -ar ../binaries/dist/install/* . +mkdir -p etc/local.d echo "#!/bin/bash set -x export LD_LIBRARY_PATH=/usr/local/lib @@ -127,14 +126,14 @@ ${DOM0_CMD} " > etc/local.d/xen.start chmod +x etc/local.d/xen.start echo "${DOMU_CFG}${DOMU_CFG_EXTRA}" > etc/xen/domU.cfg -echo "rc_verbose=yes" >> etc/rc.conf echo "XENCONSOLED_TRACE=all" >> etc/default/xencommons echo "QEMU_XEN=/bin/false" >> etc/default/xencommons mkdir -p var/log/xen/console cp ../binaries/bzImage boot/vmlinuz cp ../binaries/domU-rootfs.cpio.gz boot/initrd-domU copy_dom0_files -find . | cpio -H newc -o | gzip > ../binaries/dom0-rootfs.cpio.gz +cp ../binaries/rootfs.cpio.gz ../binaries/dom0-rootfs.cpio.gz +find . | cpio -H newc -o | gzip >> ../binaries/dom0-rootfs.cpio.gz cd .. # Load software into TFTP server directory. diff --git a/automation/scripts/xilinx-smoke-dom0less-arm64.sh b/automation/scripts/xilinx-smoke-dom0less-arm64.sh index 3e1fcf6bf93c..97190d0cbb9c 100755 --- a/automation/scripts/xilinx-smoke-dom0less-arm64.sh +++ b/automation/scripts/xilinx-smoke-dom0less-arm64.sh @@ -41,33 +41,31 @@ fi # DomU mkdir -p rootfs cd rootfs -tar xzf ../binaries/initrd.tar.gz mkdir proc mkdir run mkdir srv mkdir sys -rm var/run +mkdir -p etc/local.d echo "#!/bin/sh ${domU_check} /bin/sh" > etc/local.d/xen.start chmod +x etc/local.d/xen.start -echo "rc_verbose=yes" >> etc/rc.conf -find . | cpio -H newc -o | gzip > ../binaries/domU-rootfs.cpio.gz +cp ../binaries/rootfs.cpio.gz ../binaries/domU-rootfs.cpio.gz +find . | cpio -H newc -o | gzip >> ../binaries/domU-rootfs.cpio.gz cd .. rm -rf rootfs # DOM0 rootfs mkdir -p rootfs cd rootfs -tar xzf ../binaries/initrd.tar.gz mkdir proc mkdir run mkdir srv mkdir sys -rm var/run cp -ar ../binaries/dist/install/* . +mkdir -p etc/local.d echo "#!/bin/bash export LD_LIBRARY_PATH=/usr/local/lib @@ -78,8 +76,8 @@ bash /etc/init.d/xencommons start ${dom0_check} " > etc/local.d/xen.start chmod +x etc/local.d/xen.start -echo "rc_verbose=yes" >> etc/rc.conf -find . | cpio -H newc -o | gzip > ../binaries/dom0-rootfs.cpio.gz +cp ../binaries/rootfs.cpio.gz ../binaries/dom0-rootfs.cpio.gz +find . | cpio -H newc -o | gzip >> ../binaries/dom0-rootfs.cpio.gz cd .. -- git-series 0.9.1

©2013 Xen Project, A Linux Foundation Collaborative Project. All Rights Reserved.
Linux Foundation is a registered trademark of The Linux Foundation.
Xen Project is a trademark of The Linux Foundation.