Re: [PATCH v2] releases: use newer compression methods for tarballs

[Jan Beulich <jbeulich@xxxxxxxx>]

On 02.09.2025 18:15, Andrew Cooper wrote:
> On 25/08/2025 2:54 pm, Jan Beulich wrote:
>> --- a/tools/misc/mktarball
>> +++ b/tools/misc/mktarball
>> @@ -5,14 +5,6 @@
>>  # Takes 2 arguments, the path to the dist directory and the version
>>  set -ex
>>  
>> -function git_archive_into {
>> -    mkdir -p "$2"
>> -
>> -    git --git-dir="$1"/.git \
>> -    archive --format=tar HEAD | \
>> -    tar Cxf "$2" -
>> -}
>> -
>>  if [[ -z "$1" || -z "$2" ]] ; then
>>    echo "usage: $0 path-to-XEN_ROOT xen-version"
>>    exit 1
>> @@ -21,14 +13,20 @@ fi
>>  xen_root="$1"
>>  desc="$2"
>>  
>> -tdir="$xen_root/dist/tmp.src-tarball"
>> +tdir="$xen_root/dist"
>>  
>> -rm -rf $tdir
>> +rm -f $tdir/xen-$desc.tar.[glx]z
> 
> This is asymmetric with the rm at the end.  I'd remove
> $tdir/xen-$desc.tar* here and remove the final rm.

Can do, but ...

> Looking at the uncompressed tarball is part of my process, and it was
> preserved previously.

... afaics none was ever generated previously. git_archive_into() piped git's
output into an "untar", and then the resulting tree was all that was ever
acted upon, with tar directly piping into gzip.

If we were retaining the uncompressed tarball, it wasn't quite clear to me
whether that might get in the way of the uploading process: We surely want to
upload only the compressed ones.

> With something along these lines, Reviewed-by: Andrew Cooper
> <andrew.cooper3@xxxxxxxxxx>

Thanks, but the above first need clarifying (at the very least because I would
want to somehow mention the behavioral change in the description, yet then
there may be none if my observation was wrong).

>>  mkdir -p $tdir
>>  
>> -git_archive_into $xen_root $tdir/xen-$desc
>> +git --git-dir="$xen_root/.git" archive --format=tar HEAD 
>> --prefix=xen-$desc/ \
>> +    >"$tdir/xen-$desc.tar"
>> +
>> +gzip -9k "$tdir/xen-$desc.tar" &
>> +xz -9k "$tdir/xen-$desc.tar" &
>> +lzip -9k "$tdir/xen-$desc.tar" &
>> +wait
> 
> Interestingly, this wasn't fatal for not having lzip, but the error was
> clear on the console given the reduced verbosity, and doing 3 at the
> same time worked very nicely.

Well, obviously, because the exit status is effectively lost for async lists.

>> -GZIP=-9v tar cz -f $xen_root/dist/xen-$desc.tar.gz -C $tdir xen-$desc
>> +rm -f $tdir/xen-$desc.tar
>>  
>> -echo "Source tarball in $xen_root/dist/xen-$desc.tar.gz"
>> +echo "Source tarball in" $tdir/xen-$desc.tar.[glx]z
> 
> This was grammatically awkward to begin with, but is now pretty useless,
> especially combined with the set -x so it gets printed twice.
> 
> Something like this:
> 
> echo "Source tarballs:"
> ls -lah $tdir/xen-$desc.tar*
> 
> generates:
> 
> -rw-rw-r-- 1 andrew andrew  32M Sep  2 17:13 
> /home/andrew/xen.git/dist/xen-4.21-unstable.tar
> -rw-rw-r-- 1 andrew andrew 6.8M Sep  2 17:13 
> /home/andrew/xen.git/dist/xen-4.21-unstable.tar.gz
> -rw-rw-r-- 1 andrew andrew 4.7M Sep  2 17:13 
> /home/andrew/xen.git/dist/xen-4.21-unstable.tar.lz
> -rw-rw-r-- 1 andrew andrew 4.7M Sep  2 17:13 
> /home/andrew/xen.git/dist/xen-4.21-unstable.tar.xz
> 
> 
> on my system and is rather more useful IMO.

I was indeed wondering whether to do something like this, but didn't because
it again would have extended the scope of the patch. Now that you're asking
for it, I will. I won't pass 'a' to ls though, as the glob doesn't allow for
file names starting with '.'.

Jan