Re: [Xen-users] pvgrub2(-like?) booting methods for PVHv2 guests

[Hans van Kranenburg <hans@xxxxxxxxxxx>]

On 01/25/2018 07:01 AM, Juergen Gross wrote:
> On 24/01/18 17:37, Hans van Kranenburg wrote:
>> On 01/23/2018 06:41 AM, Juergen Gross wrote:
>>> On 23/01/18 05:46, Andy Smith wrote:
>>>> Hi,
>>>>
>>>> What are the current options for booting PVHv2 guests in Xen 4.10.0?
>>>>
>>>> As far as I can see there is only direct kernel booting, i.e.
>>>> supplying "kernel = /path/to/vmlinux".
>>>>
>>>> In PV land there is pvgrub which is very convenient for being able
>>>> to use kernel and initramfs, but I am not seeing support for that in
>>>> upstream grub2. I see that Juergen Gross post a patch series that
>>>> claims to add this feature:
>>>>
>>>>     <https://lists.gnu.org/archive/html/grub-devel/2017-11/msg00054.html>
>>>>
>>>> …but this doesn't seem to have been applied yet. Is it an ongoing
>>>> idea or has that method been abandoned?
>>>
>>> Still waiting for review.
>>
>> Since I wanted to try to test this, I scraped the patches off the
>> grub-devel list and did the following:
>>
>> 1. clone https://git.savannah.gnu.org/git/grub.git
>> 2. apply patches on top of master, plus an extra one with a puzzle piece
>> to make it actually compile (see attachment)
>>
>> 3. build:
>> ./autogen.sh
>> ./configure --target=i386 --with-platform=xenpvh
>> make
>>
>> 4. use mkstandalone to create a grub image to boot
>> ./grub-mkstandalone -d grub-core --locale-directory=/ -o
>> ~/grub-i386-xenpvh-fire-ze-missile -O i386-xenpvh
>> ~/build/pvgrub2/pvgrub2/xvda-fire-ze-missile.cfg
>>
>> contents of the config file that I want to have inserted into it (same
>> file I use for booting PV now):
>>
>> root='(xen/xvda)'
>> insmod xzio
>> insmod gzio
>> insmod btrfs
>> insmod ext2
>> echo 'Loading Linux ...'
>> linux /vmlinuz root=/dev/xvda ro elevator=noop
>> echo 'Loading initial ramdisk ...'
>> initrd /initrd.img
>> boot
>>
>> 5. Try to use it already, to see what happens. (Xen 4.10.0)
>>
>> -# cat yolo-pvgrub2
>>
>> type="pvh"
>> kernel = "/yolo/grub-i386-xenpvh-fire-ze-missile"
>> name = "yolo-pvgrub2"
>> memory = 1024
>> vcpus = 10
>> vif = [
>>     "mac=02:00:0a:8c:d9:0d,bridge=ovs0.217",
>> ]
>> disk = [
>>     '/dev/test/yolo,raw,xvda,rw',
>> ]
>>
>> -# xen create -c yolo-pvgrub2
>> Parsing config from yolo-pvgrub2
>> libxl: info: libxl_create.c:109:libxl__domain_build_info_setdefault:
>> qemu-xen is unavailable, using qemu-xen-traditional instead: No such
>> file or directory
>> S3 disabled
>> S4 disabled
>> CONV disabled
>>
>>                      GNU GRUB  version 2.03
>>
>>    Minimal BASH-like line editing is supported. For the first word, TAB
>>    lists possible command completions. Anywhere else TAB lists possible
>>    device or file completions.
>>
>>
>> grub>
>>
>> Ok, so for some reason it seems to ignore the config? But I can type
>> things manually:
>>
>> grub> insmod xzio
>> grub> insmod gzio
>> grub> insmod btrfs
>> grub> insmod ext2
>> grub> search.file /vmlinuz
>>  xen/xvda
>>
>> Ok, so it can see the disk and fine the symlink to the kernel image. Now
>> let's try the rest of it:
>>
>> grub> root='(xen/xvda)'
>> grub> linux /vmlinuz root=/dev/xvda ro elevator=noop
>> grub> initrd /initrd.img
>> grub> boot
>>
>> And now it just hangs, no output any more.
>>
>> 6. Apply linux patches to the domU kernel and Xen
>>
>> domU Linux kernel:
>> - Branch off from linux 4.14.14
>> - add https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git as
>> remote and cherry-pick the 4 patches mentioned in
>>   https://lists.gnu.org/archive/html/grub-devel/2017-12/msg00017.html
>>
>> Xen: pick this one on top of 4.10.0:
>>  https://lists.xen.org/archives/html/xen-devel/2017-11/msg01807.html
>>  4a5733771e6f33918eba07b584cccce564a67ac1
>>
>> 7. Test again after dom0 reboot into the new xen and put the new domU
>> kernel on it.
>>  -> same result: grub command prompt, and no activity after 'boot'.
>>
>> Extra things:
>>
>> 8. Start domU after copying kernel/initrd to the dom0:
>>
>> type="pvh"
>> kernel = "/yolo/vmlinuz-4.14.14-yolo"
>> ramdisk = "/yolo/initrd.img-4.14.14-yolo"
>> root = "/dev/xvda ro elevator=noop console=hvc0"
>>
>> This boots, dmesg: http://paste.debian.net/plainh/7120cef2
>>
>> 9. Just for fun, start domU with previous kernel/initrd without the patches:
>>
>> type="pvh"
>> kernel = "/yolo/vmlinuz-4.14.0-3-amd64"
>> ramdisk = "/yolo/initrd.img-4.14.0-3-amd64"
>> root = "/dev/xvda ro elevator=noop console=hvc0"
>>
>> This boots and has 1 vcpu and no ACPI lines in dmesg, which is expected
>> breakage, I believe, reading xen-devel.
>>
>> So, what am I missing?
> 
> I used the attached script to build it (derived from the pvgrub build
> script used in SUSE's grub-xen RPM).

Thanks. This gets me a step further.

Using the "graft point syntax" thingie
("/boot/grub/grub.cfg=./grub.cfg") helped getting the config actually
executed instead of getting the prompt.

Next I got:

Welcome to GRUB!

Loading Linux ...
Loading initial ramdisk ...

...and then it stops.

So I modified my grub config a bit from the one shown above, adding
explicit console=hvc0:

root='(xen/xvda)'
insmod xzio
insmod gzio
insmod btrfs
insmod ext2
echo 'Loading Linux ...'
linux /vmlinuz root=/dev/xvda ro elevator=noop console=hvc0
echo 'Loading initial ramdisk ...'
initrd /initrd.img
echo 'There we go! ...'
boot

And now I get console output and things happen. Only it can't find the disk.

[...]
Begin: Running /scripts/local-block ... done.
Begin: Running /scripts/local-block ... done.
done.
Gave up waiting for root file system device.  Common problems:
 - Boot args (cat /proc/cmdline)
   - Check rootdelay= (did the system wait long enough?)
 - Missing modules (cat /proc/modules; ls /dev)
ALERT!  /dev/xvda does not exist.  Dropping to a shell!
(initramfs)

Full log at http://paste.debian.net/plainh/426bed60

Now I have to find out why's that. But first time for lunch. ;)

Hans

_______________________________________________
Xen-users mailing list
Xen-users@xxxxxxxxxxxxxxxxxxxx
https://lists.xenproject.org/mailman/listinfo/xen-users