Re: kexec failure with Xen 4.19-rc4 and 4.20-dev on linux host

[A Kundu <avikalpa@xxxxxxxxx>]

On 8/2/24 13:25, Jan Beulich wrote:
> On 02.08.2024 09:28, A Kundu wrote:
>> On 8/2/24 09:06, Baoquan He wrote:
>>> On 07/31/24 at 06:34pm, A Kundu wrote:
>>>> I am experiencing issues using kexec to load Xen 4.17(debian's apt 
version),
>>>> Xen 4.19-rc4 (compiled from source) and 4.20-dev (compiled from 
source) on a
>>>> debian host.
>>> You should CC this to XEN dev list so that XEN dev knows this and may
>>> provide help. Not everyone is interested in and knows XEN.
>>>
>>>> System information:
>>>> $ uname -a
>>>> Linux host 6.9.10-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.9.10-1 
(2024-07-19)
>>>> x86_64 GNU/Linux
>>>>
>>>> $ kexec --version # compiled from source tarball with ./configure 
--with-xen
>>>> kexec-tools 2.0.29
>>>>
>>>> Steps to reproduce:
>>>>
>>>> 1. Set variables:
>>>>
>>>> XEN_HYPERVISOR="/boot/xen.gz"
>>>> XEN_CMD="dom0_mem=6G dom0_max_vcpus=6 dom0_vcpus_pin cpufreq=xen"
>>>>
>>>> 2. Attempt to load Xen 4.19-rc4:
>>>>
>>>> # kexec -l "$XEN_HYPERVISOR" --command-line="$XEN_CMD"
>>>> Could not find a free area of memory of 0x3b6001 bytes...
>>>> elf_exec_build_load_relocatable: ELF exec load failed
>>>>
>>>> 3. Attempt to load Xen 4.20-dev:
>>>>
>>>> # kexec -l "$XEN_HYPERVISOR" --command-line="$XEN_CMD"
>>>> Could not find a free area of memory of 0x3f8001 bytes...
>>>> elf_exec_build_load_relocatable: ELF exec load failed
>>>>
>>>> 4. Attempt to load Xen 4.17 (from debian rrepositories):
>>>> # kexec -l /boot/xen-4.17-amd64.gz --command-line="$XEN_CMD"
>>>> Could not find a free area of memory of 0x3b4001 bytes...
>>>> elf_exec_build_load_relocatable: ELF exec load failed
>
> And with all of them saying effectively the same, did you verify you
> actually have a sufficiently large area reserved? The obvious
> place for you to look at is Xen's boot log (obtained via serial
> console or "xl dmesg" immediately after booting the system). If you
> find everything as expected there, ...
>
>>>> If you need any further information to investigate this problem,
>>>> please let me know.
>
> ... please provide that boot log.

Dear Jan,

Thank you for your response and suggestions.

As mentioned in my original message, I have already attempted to load
Xen using kexec with various versions (4.17, 4.19-rc4, and 4.20-dev),
all of which resulted in the same error:

Could not find a free area of memory of 0x3b6001 bytes...
elf_exec_build_load_relocatable: ELF exec load failed

I have also followed up on your suggestion to check the Xen boot log
using "xl dmesg", but unfortunately, I received the following error:

xencall: error: Could not obtain handle on privileged command interface: 
No such file or directory
libxl: error: libxl.c:102:libxl_ctx_alloc: cannot open libxc handle: No 
such file or directory
cannot init xl context

This indicates that Xen did not boot successfully, so there are no logs
available.

>
> And with all of them saying effectively the same, did you verify you
> actually have a sufficiently large area reserved? The obvious
> place for you to look at is Xen's boot log (obtained via serial
> console or "xl dmesg" immediately after booting the system). If you
> find everything as expected there, ...
>

In an attempt to resolve the memory allocation issue, I have tried the
following:

Added a crashkernel=<size>@<offset> parameter to the host kernel command
line to reserve a dedicated memory region for kexec, and attempted to
load Xen into that area.

Experimented with the mem=<size> parameter to limit the amount of memory
used by the host kernel, hoping to leave more contiguous space available
for loading Xen.

Unfortunately, neither of these approaches resolved the issue, and I
still encounter the same memory allocation error when attempting to load
Xen with kexec.

I have also reviewed the available documentation for kexec and Xen, but
much of it appears to be outdated, referencing archaic options like
--vmm which no longer exist in the current version of kexec-tools.

> Jan

At this point, I'm unsure of how to proceed. I suspect the issue may be
related to changes in the kexec codebase and its interaction with Xen,
as kexec docs for Xen has not been actively maintained since 2017.

git log -1 --format="%ad" -- docs/misc/kexec_and_kdump.txt
Fri Jun 9 14:11:37 2017 +0200

If it's not too much to ask, could you please check if you can
successfully launch the Xen kernel using kexec with kexec-tools
version > 2.x.x and a Linux kernel version >= 6.6?

I have never been able to boot the Xen kernel with anything (syslinux,
zfsbootmenu)other than GRUB and mboot.c32, as used by Citrix or XCP-ng.

In fact, I have never come across a Xen distribution that doesn't rely
on GRUB for xen, which makes me question whether kexec'ing the Xen
kernel is even possible at this point.


For reference, I'm providing a log of my attempts to boot Xen using
ZFSBootMenu (a barebones linux kernel trying to boot via kexec; other 
details irrelevant):
tee: /zfsbootmenu/environments/zroot/ROOT/debian/mnt/var/zbm.log: 
Read-only file system
=== Attempting to Launch Xen ===
Selected Kernel     : /boot/vmlinuz-6.9.12-amd64
Selected initramfs  : /boot/initrd.img-6.9.12-amd64
Selected BE         : zroot/ROOT/debian
Selected Mountpoint : /zfsbootmenu/environments/zroot/ROOT/debian/mnt
ZFSBootMenu kernel commandline: quiet loglevel=0

zfs get org.zfsbootmenu:commandline zroot/ROOT/debian
NAME               PROPERTY VALUE                                        
SOURCE
zroot/ROOT/debian  org.zfsbootmenu:commandline  quiet crashkernel=0M 
memmap=64M$0x100000000  local
(^^^ This is the linux/dom0 commandline, executed by kexec)

=== Memory reported by head /proc/meminfo ===
MemTotal:        8044084 kB
MemFree:         7822332 kB
MemAvailable:    7687320 kB
Buffers:              0 kB
Cached:           49136 kB
SwapCached:           0 kB
Active:           45868 kB
Inactive:             0 kB
Active(anon):     45868 kB
Inactive(anon):       0 kB
=== End of Memory Report ===

Xen kernel found. Attempting kexec...
Xen Kernel Path     : 
/zfsbootmenu/environments/zroot/ROOT/debian/mnt/boot/xen.gz
Try gzip decompression.
Could not find a free area of memory of 0x3ba001 bytes...
elf_exec_build_load_relocatable: ELF exec load failed
Executing kexec...
Nothing has been loaded!

After booting into linux what happens, I have already shared.

If you have any further insights or suggestions on how to overcome this
memory allocation issue or successfully boot Xen using kexec on a modern
system, I would greatly appreciate your guidance.

Thank you for your time and assistance.

Best regards,
A Kundu