AW: Problems booting Ubuntu Jammy 22.04 PV domU from Jammy dom0

[Klaus Darilion <klaus.darilion@xxxxxx>]

Sorry for the late repsonse. I just found some time for testing.

Thanks for the detailed answer - that made things much clearer.

As suggested by you and others: using PVH instead of PV solved the problem.

Thanks
Klaus

> -----Ursprüngliche Nachricht-----
> Von: Xen-users <xen-users-bounces@xxxxxxxxxxxxxxxxxxxx> Im Auftrag von
> Andy Smith
> Gesendet: Mittwoch, 27. April 2022 23:54
> An: xen-users@xxxxxxxxxxxxxxxxxxxx
> Betreff: Re: Problems booting Ubuntu Jammy 22.04 PV domU from Jammy
> dom0
> 
> Hi Klaus,
> 
> On Wed, Apr 27, 2022 at 12:01:54AM +0200, Klaus Darilion wrote:
> > When using pygrub the error is:
> > xc: error: panic: xg_dom_bzimageloader.c:619: xc_try_zstd_decode: ZSTD
> decompress support unavailable
> >
> > So it seems that pygrub does not support the ZSTD compressed kernel.
> 
> pygrub extracts a kernel and initrd from the guest block device and
> then Xen direct boots it. Your Xen does not support zstd-compressed
> kernels for direct boot so that doesn't work.
> 
> You can use the extract-vmlinux script to decompress a guest kernel
> to vmlinux and direct boot that. It isn't hard to bodge a test into
> pygrub for zstd (and lz4) compressed kernels and automatically call
> extract-vmlinux on them, then boot the resulting vmlinux.
> 
> However, pygrub is a really really bad way to boot guests, so you
> would be much better off figuring out how to stop using it
> altogether. Maybe now is a good time for that, since you're going to
> have to make some changes anyway?
> 
> TL;DR: PVH mode is probably your saviour as long as you don't need
> to boot anything older than kernel version 4.20 or so.
> 
> > So I tried to boot the domU with pvgrub2 by setting:
> > kernel = "/usr/lib/grub-xen/grub-x86_64-xen.bin"
> 
> pvgrub also doesn't support LZ4 or zstd compressed kernels. It seems
> to be abandoned in that no one is interested in adding such support
> to it.
> 
> Again you can lash something together that decompresses the kernels
> first. This time a bit different since it needs to run in guest
> context due to pvgrub. Here's something I came up with to make the
> guest decompress its kernel when the kernel is installed:
> 
>     https://github.com/bitfolk/decompress-lz4-kernels
> 
> > https://wiki.xenproject.org/wiki/Xen_Project_4.15_Feature_List states
> "Support for zstd-compressed dom0 (x86) and domU kernels.".
> >
> > So I need some advice/confirmations:
> > Can I boot Linux 5.15 (ZSTD kernel) with pygrub? If not, is it planned to 
> > add
> ZSTD support to pygrub?
> 
> pygrub doesn't boot the kernel, Xen does, so if your Xen supports
> direct booting LZ4/zstd then that will work. It's not a question of
> what pygrub supports as pygrub does not parse or manipulate the
> kernel file in any way, it just passes it to Xen.
> 
> However you'd still be using pygrub, which is bad.
> 
> > If pygrub will not support ZSTD, what are my options? Should it
> > work with pvgrub2 and I am doing something wrong? How to correctly
> > use pvgrub2.
> 
> pvgrub seems to be a dead end for guests using LZ4/zstd kernels as
> there seems to be no interest in supporting that.
> 
> > If pvgrub2 does not support ZSTD, what are my options? Would it
> > help to move from PV to PVH,PVHVM,...?
> 
> Yes. Every guest that uses LZ4 or zstd kernels also supports being
> booted in PVH mode, so I switched to PVH for every guest that
> supports it and stopped using my extract kernel hack that was
> necessary for getting pvgrub to boot them.
> 
> One of the other reasons why pvhgrub is better is that it uses the
> standard parts of grub for loading images, so if grub supports the
> compression format then pvhgrub does too. So no problem loading
> lz4/zstd compressed kernels, as long as the grub package you started
> with is new enough.
> 
> > Can I automate building
> > domUs similar to xen-create-image also with non-PV domUs?
> 
> As far as I know xen-tools and xen-create-image are also dead
> upstream so it would be best not to rely on them. Maybe a project
> for a later time though, as I can't see any reason why something
> that xen-create-image makes would not boot under PVH mode.
> 
> For Debian/Ubuntu-based guests you can just boot the installer
> kernel/initrd in PVH mode and run through a normal text mode
> install. Or you can use debootstrap from your dom0 to download
> everything you need into a directory. Which is basically what
> xen-create-image is doing.
> 
> (This all changed with Ubuntu >= 21.10 as they have moved away from
> the Debian installer. For newer Ubuntu it's going to be either
> debootstrap or download the Ubuntu cloud image and boot that (with
> optional cloud-init for any configuration you need).)
> 
> Cheers,
> Andy