Re: pvhgrub docs

[Andy Smith <andy@xxxxxxxxxxxxxx>]

Hi Chris,

On Thu, Oct 07, 2021 at 07:09:35AM -0500, Chris Myers wrote:
> Out of curiosity, could someone point me to some documentation for pvhgrub?

I haven't found much, I've had to teach myself from other people's
examples and some out of date articles on the Xen wiki. Here is what I
learned so far and have had working for a year or so.

If you're currently running a PV mode guest and it has a kernel of
4.19 or greater, you can/should change it to a PVH mode guest. You
can also boot it using pvhgrub (instead of pvgrub, pygrub or direct
kernel booting).

Inside the domU all you need is a grub.cfg file generated that grub
understands. The easiest way to do this in my experience is to
actually install grub inside the guest, even though only its config
file is needed. So inside a Debian guest you'd install grub-pc and
make sure that the install process leaves you with a correct
/boot/grub/grub.cfg. I didn't have to do anything special on
Debian/Ubuntu to make it generate a correct config, and keep it up
to date.

On the dom0 the kernel file you use for PVH guests is a grub2 image
built as a PVH binary. On Debian it can be found in the
grub-xen-host package and once installed that lives at
/usr/lib/grub-xen/grub-i386-xen_pvh.bin. I'll refer to that as "the
pvhgrub image/binary".

So in your domU config:

[…]
type   = pvh
kernel = /usr/lib/grub-xen/grub-i386-xen_pvh.bin
[…]

I recommend just comment out your existing "kernel", "ramdisk" and
"extra" lines so you can put them back if this doesn't work.

The default behaviour of that pvhgrub binary is to run grub and
search all available block devices for the following file paths:

    /boot/grub/grub.cfg
    /grub/grub.cfg

If it finds any such file then it assumes that is a grub config and
loads it. The second path is for the case where /boot is its own
block device. The path on that device would be /grub/grub.cfg.

Thus, the domU admin can install their own kernels, initramfs and
kernel command line arguments and boot them without dom0 admin
assistance. The domU admin can do anything that grub can do, because
it really is grub. The domU config file in dom0 doesn't have to be
changed every time there is a kernel upgrade inside the guest.

I personally wanted a bit more complex initial grub config because I
also wanted it to search for /boot/grub2/grub.cfg (Red Hat / Fedora
path) and /boot/grub/menu.lst (legacy grub config file), so I
compile my own pvhgrub binary. Instructions for that are here:

    
https://wiki.xenproject.org/wiki/Xen_4.12_RC_test_instructions#PVH_domU_boot_via_grub2

Except I seem to have modified my process a little:

    $ git clone https://git.savannah.gnu.org/git/grub.git
    $ cd grub
    $ ./configure --prefix=/opt/grub \
            --target=x86-64 \
            --with-platform=xenpvh && \
      make && sudo make install
    $ edit /opt/grub/share/memdisk/grub.cfg # YOUR BOOTSTRAP CONFIG
    $ echo "normal (memdisk)/grub.cfg" | \
        sudo tee /opt/grub/etc/grub-bootstrap.cfg
    $ sudo tar -C /opt/grub/share/memdisk \
            -cvf /opt/grub/share/memdisk.tar grub.cfg && \
      sudo /opt/grub/bin/grub-mkimage -O i386-xenpvh \
            -c /opt/grub/etc/grub-bootstrap.cfg \
            -m /opt/grub/share/memdisk.tar \
            -o /opt/grub/lib/pvhgrub.bin \
            -p /boot/grub \
            -v /opt/grub/lib/grub/i386-xenpvh/*.mod

So, that works for me, but you don't need to do any of that if
you're happy with the Debian default behaviour of just looking for
boot/grub.cfg on the domU's block devices.

Even if you do want a different bootstrap behaviour maybe you can
even get away without completely compiling your own pvhgrub binary —
as I do there — by just coming up with your own config and using
grub-mkimage from the package, I am doing this because I started
before pvhgrub was packaged and only just committed in upstream
grub.

If you run into any trouble put back the "kernel", "ramdisk" and
"extra" lines to return to a booting configuration for that domU.

Perhaps some or all of this should go in an article on the Xen wiki.
I do worry about it becoming yet another out of date reference
though.

Cheers,
Andy