Re: Imagebuilder can't compute correctly the memory addresses of our images

[Stefano Stabellini <sstabellini@xxxxxxxxxx>]

Hi Mario,

Replies inline below


On Wed, 1 Nov 2023, Mario Marietto wrote:
> Hello to everyone.
> 
> We are a team of linux enthusiasts who are trying to boot Xen on a Samsung 
> XE303C12 Chromebook aka "snow" , following the suggestions in
> the slide show presentation here:
> 
> https://www.slideshare.net/xen_com_mgr/xpds16-porting-xen-on-arm-to-a-new-soc-julien-grall-arm
> 
> This device uses an exynos5250 SOC dual core 1.7 GHz with 2 MB RAM, it is
> a Samsung armv7 chip with virtualization extensions.
> 
> In particular, we have it working fairly well both on the bare metal with
> a recent 6.1.59 Linux LTS kernel and also with a recent 5.4.257 LTS
> kernel with KVM, the older LTS kernel version is used to test KVM because
> support for KVM on arm v7 was removed from Linux around kernel version
> 5.7. So we know we have the hypervisor mode enabled because we were able
> to use it with KVM.
> 
> For Xen, we are using the latest Debian build of Xen 4.17 for the Debian
> armhf architecture:
> 
> (XEN) Xen version 4.17.2-pre (Debian 4.17.1+2-gb773c48e36-1)
> (pkg-xen-devel@xxxxxxxxxxxxxxxxxxxxxxx) (arm-linux-gnueabihf-gcc (Debian 
> 12.2.0-14) 12.2.0) debug=n Thu May 18 19:26:30 UTC 2023
> 
> The Linux kernel is a custom build that adds the Xen config kernel
> options (CONFIG_XEN_DOM0, etc) on top of a kernel that works well on the
> same Chromebook model on the bare metal.
> 
> Our method of booting is to have u-boot boot the Xen hypervisor and load
> the device tree after adding the dom0 to the otherwise unaltered device
> tree from the Linux kernel using u-boot fdt commands to add a /chosen
> node, as described on the Xen wiki and in the pages linked from there. We
> have also tried adding and loading an initrd.img using the device tree
> /chosen node but that made no difference in our tests.
> 
> The workflow is the following :
> 
> a) let's create the file xen-config-6.1.59 :
> 
> MEMORY_START="0x0"
> MEMORY_END="0x80000000"
> LOAD_CMD="ext2load mmc 1:3"
> BOOT_CMD="bootm"
> DEVICE_TREE="exynos5250-snow-6.1.59-stb-xen-cbe+.dtb"
> XEN="xen-4.17-armhf"
> XEN_CMD="console=dtuart dtuart=serial0 dom0_mem=1G dom0_max_vcpus=2 
> bootscrub=0 vwfi=native sched=null"
> DOM0_KERNEL="zImage-6.1.59-xen-mixer-off"
> DOM0_CMD="console=tty earlycon=xen earlyprintk=xen root=/dev/mmcblk1p4 rw 
> rootwait clk_ignore_unused"
> UBOOT_SOURCE="xen.source"
> UBOOT_SCRIPT="xen.scr"
> 
> b) let's create gen-script.sh :
> 
> bash ./uboot-script-gen -c xen-config-6.1.59 -d .
> 
> c) let's run the script gen-script.sh :
> 
> # ./gen-script
> 
> Image Name:    
> Created:      Wed Nov  1 14:34:23 2023
> Image Type:   ARM Linux Kernel Image (uncompressed)
> Data Size:    884744 Bytes = 864.01 KiB = 0.84 MiB
> Load Address: 01800000
> Entry Point:  01800000
> Generated uboot script xen.scr, to be loaded at address 0xC00000:
> ext2load mmc 1:3 0xC00000 xen.scr; source 0xC00000
 
This step is confusing: step b) should directly generate xen.scr. There
is no ./gen-script to run.

 
> d) let's give a look inside the file xen.scr because we need some memory 
> address that it has generated :
> 
> 
> ext2load mmc 1:3 0x1800000 xen-4.17-armhf.ub
> ext2load mmc 1:3 0x1A00000 exynos5250-snow-6.1.59-stb-xen-cbe+.dtb

Here the Dom0 kernel is missing?


> fdt addr 0x1A00000
> fdt resize 1024
> fdt set /chosen \#address-cells <0x2>
> fdt set /chosen \#size-cells <0x2>
> fdt set /chosen xen,xen-bootargs "console=dtuart dtuart=serial0 dom0_mem=1G 
> dom0_max_vcpus=2 bootscrub=0 vwfi=native sched=null"
> fdt mknod /chosen dom0
> fdt set /chosen/dom0 compatible  "xen,linux-zimage" "xen,multiboot-module" 
> "multiboot,module"
> fdt set /chosen/dom0 reg <0x0 0xE00000 0x0 0x816110 >
> fdt set /chosen xen,dom0-bootargs "console=tty earlycon=xen earlyprintk=xen 
> root=/dev/mmcblk1p4 rw rootwait clk_ignore_unused"
> setenv fdt_high 0xffffffffffffffff
> bootm 0x1800000 - 0x1A00000
> 
> As you can see it has generated a lot of interesting memory addresses,but 
> unfortunately they are all wrong,except for one,this :
> 
> 0x816110
> 
> This is well calculated.

What do you mean they are wrong? Which ones are wrong?

Imagebuilder will pick addresses 2MB aligned starting from a 2MB offset
from MEMORY_START.

So both 0x1800000 and 0x1A00000 should be good addresses because they
are in the MEMORY_START-MEMORY_END range.

There is one issue though: the dom0 kernel address 0xE00000 is not
present among the loading commands. 


> e) let's write this template,called "bootxen.source" :
> 
> mmc dev 1
> ext2load mmc 1:3 0x42000000 zImage-6.1.59-xen-mixer-off
> ext2load mmc 1:3 0x51000000 xen-4.17-armhf-armmp-0x51004000.ub
> ext2load mmc 1:3 0x5ffec000 exynos5250-snow-6.1.59-stb-xen-cbe+.dtb
> fdt addr 0x5ffec000
> fdt resize 1024
> fdt set /chosen \#address-cells <0x2>
> fdt set /chosen \#size-cells <0x2>
> fdt set /chosen xen,xen-bootargs "console=dtuart dtuart=serial0 dom0_mem=1G 
> dom0_max_vcpus=2 bootscrub=0 vwfi=native sched=null"
> fdt mknod /chosen dom0
> fdt set /chosen/dom0 compatible  "xen,linux-zimage" "xen,multiboot-module" 
> "multiboot,module"
> fdt set /chosen/dom0 reg <0x0 0x42000000 0x0 0x816110 >
> fdt set /chosen xen,dom0-bootargs "console=tty1 root=/dev/mmcblk1p4 rw 
> rootwait clk_ignore_unused --no-log"
> bootm 0x51000000 - 0x5ffec000
> 
> As you can see,we can use only one value generated by your script : 0x816110
> The other values have been generated manually by us,not by your script,
> because the values that it produces are wrong.

If you explain why they are wrong, we can find a way to fix them. As
mentioned, they are just 2MB aligned address started at an offset from
MEMORY_START. Is it possible that you need to configure MEMORY_START and
MEMORY_END properly for your device?

For instance you should be able to get ImageBuilder to generate
addresses similar to the ones you chose if you use:

MEMORY_START="0x41e00000"
MEMORY_END="0x60000000"



> f) let's create the script to-scr.sh :
> 
> mkimage -A arm -T script -C none -a 0x50000000 -e 0x50000000 -d 
> bootxen.source bootxen.scr
> 
> g) type these commands to boot Xen and Linux 6.1.59 as dom0 :
> 
> SMDK5250 # mmc dev 1
> SMDK5250 # ext2load mmc 1:3 0x50000000 bootxen.scr; source 0x50000000
> 
> We would like that your script was able to generate every value correctly.
> Is this possible ? Why does it,instead,fail in most cases ?

Sure it should be possible.