Re: [PATCH v2] releases: use newer compression methods for tarballs

[Andrew Cooper <andrew.cooper3@xxxxxxxxxx>]

On 25/08/2025 2:54 pm, Jan Beulich wrote:
> Other projects have long switched to xz and/or lzip.
>
> Tidy things some as well: With the removal of qemu from the tarball,
> intermediately extracting the tarball again has become wasteful. Drop
> that. Invoke compressors using asynchronous lists, to reduce overall
> latency. Drop the -v option from the (previously implicit) gzip
> invocation.
>
> Signed-off-by: Jan Beulich <jbeulich@xxxxxxxx>
> ---
> v2: Don't expand intermediate uncompressed tarball. Don't check for
>     commands' availablity. Don't request statistics. Use async lists.
>
> --- a/docs/process/release-technician-checklist.txt
> +++ b/docs/process/release-technician-checklist.txt
> @@ -119,7 +119,7 @@ RELEASE TARBALL
>         make src-tarball           # uses git-describe (best for RCs)
>          # ^find some way to add git-cache-proxy to this (done in 
> ~iwj/.gitconfig)
>         mkdir /volatile/iwj/website-thing/xen.org/oss-xen/release/$v
> -       mv dist/xen-$v.tar.gz 
> /volatile/iwj/website-thing/xen.org/oss-xen/release/$v/.
> +       mv dist/xen-$v.tar.[glx]z 
> /volatile/iwj/website-thing/xen.org/oss-xen/release/$v/.
>  
>         # website-thing/xen.org is cvs -d 
> mail.xenproject.org:/home/downloads-cvs/cvs-repos co xen.org
>       cd /volatile/iwj/website-thing/xen.org
> @@ -139,9 +139,12 @@ RELEASE TARBALL
>       cvs add -kb oss-xen/release/$v/
>  
>          cd oss-xen/release/$v
> -        gpg --digest-algo=SHA256 --detach-sign -u 'xen tree' xen-$v.tar.gz
> -     cvs add -kb xen-$v.tar.gz
> -        cvs add -kb xen-$v.tar.gz.sig
> +        for t in xen-$v.tar.[glx]z
> +        do
> +            gpg --digest-algo=SHA256 --detach-sign -u 'xen tree' $t
> +            cvs add -kb $t
> +            cvs add -kb $t.sig
> +        done
>          cd ../../..
>  
>       cvs ci -m $v
> @@ -152,6 +155,10 @@ RELEASE TARBALL
>       # should show something like
>       #   U oss-xen/release/4.8.0-rc2/xen-4.8.0-rc2.tar.gz
>       #   U oss-xen/release/4.8.0-rc2/xen-4.8.0-rc2.tar.gz.sig
> +     #   U oss-xen/release/4.8.0-rc2/xen-4.8.0-rc2.tar.lz
> +     #   U oss-xen/release/4.8.0-rc2/xen-4.8.0-rc2.tar.lz.sig
> +     #   U oss-xen/release/4.8.0-rc2/xen-4.8.0-rc2.tar.xz
> +     #   U oss-xen/release/4.8.0-rc2/xen-4.8.0-rc2.tar.xz.sig
>  
>  After a .0 release, update XEN_EXTRAVERSION again (to .1-pre, see above).
>  
> --- a/docs/process/xen-release-management.pandoc
> +++ b/docs/process/xen-release-management.pandoc
> @@ -274,10 +274,10 @@ Xen X.Y rcZ is tagged. You can check tha
>  https://xenbits.xen.org/git-http/xen.git X.Y.0-rcZ
>  
>  For your convenience there is also a tarball at:
> -https://downloads.xenproject.org/release/xen/X.Y.0-rcZ/xen-X.Y.0-rcZ.tar.gz
> +https://downloads.xenproject.org/release/xen/X.Y.0-rcZ/xen-X.Y.0-rcZ.tar.[glx]z
>  
>  And the signature is at:
> -https://downloads.xenproject.org/release/xen/X.Y.0-rcZ/xen-X.Y.0-rcZ.tar.gz.sig
> +https://downloads.xenproject.org/release/xen/X.Y.0-rcZ/xen-X.Y.0-rcZ.tar.[glx]z.sig
>  
>  Please send bug reports and test reports to xen-devel@xxxxxxxxxxxxxxxxxxxx.
>  When sending bug reports, please CC relevant maintainers and me
> --- a/tools/misc/mktarball
> +++ b/tools/misc/mktarball
> @@ -5,14 +5,6 @@
>  # Takes 2 arguments, the path to the dist directory and the version
>  set -ex
>  
> -function git_archive_into {
> -    mkdir -p "$2"
> -
> -    git --git-dir="$1"/.git \
> -     archive --format=tar HEAD | \
> -     tar Cxf "$2" -
> -}
> -
>  if [[ -z "$1" || -z "$2" ]] ; then
>    echo "usage: $0 path-to-XEN_ROOT xen-version"
>    exit 1
> @@ -21,14 +13,20 @@ fi
>  xen_root="$1"
>  desc="$2"
>  
> -tdir="$xen_root/dist/tmp.src-tarball"
> +tdir="$xen_root/dist"
>  
> -rm -rf $tdir
> +rm -f $tdir/xen-$desc.tar.[glx]z

This is asymmetric with the rm at the end.  I'd remove
$tdir/xen-$desc.tar* here and remove the final rm.

Looking at the uncompressed tarball is part of my process, and it was
preserved previously.

With something along these lines, Reviewed-by: Andrew Cooper
<andrew.cooper3@xxxxxxxxxx>

>  
>  mkdir -p $tdir
>  
> -git_archive_into $xen_root $tdir/xen-$desc
> +git --git-dir="$xen_root/.git" archive --format=tar HEAD --prefix=xen-$desc/ 
> \
> +    >"$tdir/xen-$desc.tar"
> +
> +gzip -9k "$tdir/xen-$desc.tar" &
> +xz -9k "$tdir/xen-$desc.tar" &
> +lzip -9k "$tdir/xen-$desc.tar" &
> +wait

Interestingly, this wasn't fatal for not having lzip, but the error was
clear on the console given the reduced verbosity, and doing 3 at the
same time worked very nicely.

>  
> -GZIP=-9v tar cz -f $xen_root/dist/xen-$desc.tar.gz -C $tdir xen-$desc
> +rm -f $tdir/xen-$desc.tar
>  
> -echo "Source tarball in $xen_root/dist/xen-$desc.tar.gz"
> +echo "Source tarball in" $tdir/xen-$desc.tar.[glx]z

This was grammatically awkward to begin with, but is now pretty useless,
especially combined with the set -x so it gets printed twice.

Something like this:

echo "Source tarballs:"
ls -lah $tdir/xen-$desc.tar*

generates:

-rw-rw-r-- 1 andrew andrew  32M Sep  2 17:13 
/home/andrew/xen.git/dist/xen-4.21-unstable.tar
-rw-rw-r-- 1 andrew andrew 6.8M Sep  2 17:13 
/home/andrew/xen.git/dist/xen-4.21-unstable.tar.gz
-rw-rw-r-- 1 andrew andrew 4.7M Sep  2 17:13 
/home/andrew/xen.git/dist/xen-4.21-unstable.tar.lz
-rw-rw-r-- 1 andrew andrew 4.7M Sep  2 17:13 
/home/andrew/xen.git/dist/xen-4.21-unstable.tar.xz


on my system and is rather more useful IMO.

~Andrew