AW: Colibri imx8qxp: Missing kernel boot module

[Daniel Wagner2 <Daniel.Wagner2@xxxxxxxxxxxxxxxxxx>]

Hello,

> On 22/08/2020 13:00, Daniel Wagner2 wrote:
> >>>>>>>> On 10/08/2020 16:32, Daniel Wagner2 wrote:
> >>>>>>>>> Hello xen-users,
> >>>>>>>>> we are trying to get Xen running on a Toradex "Colibri iMX8X"
> >>>>>>>>> module (see [1] at the bottom of this email), which features
> >>>>>>>>> the
> >>>>>>>>> iMX8 QXP
> >>>>>>> prozessor.
> >>>>>>>>> We found that NXP has a Xen reference implementation [2][3]
> >>>>>>>>> for their MEK Module and tried to port that to the Toradex
> >>> module.
> >>>>>>>>>
> >>>>>>>>> When booting via the bootscript [4], which is unaltered except
> >>>>>>>>> for the "dom0fdt_file" and "xenhyper_bootargs" variables, we
> >>>>>>>>> get the following
> >>>>>>>>> error:
> >>>>>>>>>
> >>>>>>>>> [...] (See [5] for complete bootlog)
> >>>>>>>>> (XEN) *** LOADING DOMAIN 0 ***
> >>>>>>>>> (XEN) Missing kernel boot module?
> >>>>>>>>> (XEN)
> >>>>>>>>> (XEN) ****************************************
> >>>>>>>>> (XEN) Panic on CPU 0:
> >>>>>>>>> (XEN) Could not set up DOM0 guest OS
> >>>>>>>>> (XEN) ****************************************
> >>>>>>>>> (XEN)
> >>>>>>>>> (XEN) Reboot in five seconds...
> >>>>>>>>>
> >>>>>>>>> For Dom0 we took the linux-toradex kernel source, ran "make
> >>>>> xenconfig"
> >>>>>>>>> [6], which should add Xen-support and rebuilt the Image with
> >>>>>>>>> "make
> >>>>>>> Image".
> >>>>>>>>> For dom0 DTB [7] we copied the dom0 DTS NXP uses for their
> >>>>>>> MEK-Module
> >>>>>>>>> [8] and only adjusted the "bootargs" parameter.
> >>>>>>>>> This Linux Image was able to run after we rebuilt and booted
> >>>>>>>>> it without
> >>>>>>> Xen.
> >>>>>>>>
> >>>>>>>> Which Device-Tree did you for boot Linux without Xen?
> >>>>>>>
> >>>>>>> Used the same Device-Tree Binary that we want to use with Xen
> >>>>>>> which is described in [7]. In [7] you can see rows 23 and 24,
> >>>>>>> which include the Devicetree as it is supplied by Toradex.
> >>>>>>> To boot linux with our dom0 DTB, the u-boot variable fdt_file
> >>>>>>> was changed from fsl-imx8qxp-colibri-eval-v3.dtb to
> >>>>>>> fsl-imx8qxp-colibri-eval-v3-dom0.dtb.
> >>>>>>
> >>>>>>
> >>>>>> Did you use the U-Boot cmd 'xenmmcboot' or 'xennetboot'?
> >>>>>
> >>>>> To boot linux without Xen the u-boot cmd 'boot' is used.
> >>>>> To boot Xen we use 'run xenmmcboot' (in an adjusted form to
> >>>>> account for different filenames and bootargs, see [4]).
> >>>>
> >>>> So kernel Image loaded to 0x80a00000, right?
> >>>>
> >>>> Please load kernel image before run xenmmcboot if not.
> >>>
> >>> The xenmmcboot script [4] line 25 loads the linux image.
> >>> Inserting the values for the variables in this line gives:
> >>> 'fatload mmc 0:1 0x92000000 image'
> >>> So we load the kernel image to 0x92000000. Is this not the right
adress?
> >>> How can the right adress, to load the kernel image to, be determined?
> >>
> >> With U-boot, you will have to unfortunately determine the address
> > manually. I
> >> usually use the address provided by U-boot for baremetal boot and
> >> load
> > both
> >> Xen and Linux close to each other.
> >>
> >>> Loading it to 0x80a00000 yields the same error ("Missing kernel boot
> >>> module?")
> >>
> >> It may be possible that the Device-Tree node is incorrect. Would it
> >> be
> > possible to
> >> dump the Device-Tree node /chosen/ in either U-boot or Xen?
> >>
> >
> > We tried to load xen, dom0 und fdt close together in an effort to
> > replicate your approach.
> > Thats why we now load
> > xen         @ 0x8200 0000
> > dom0        @ 0x8210 0000
> > fdt @ 0x8370 0000
> > To account for these changes, the /chosen node was also adjusted.
> > /chosen (for readibility as pastebin):
> > https://pastebin.com/v6pmm1iq
> >
> > chosen {
> >     bootargs = "console=ttyLP3,115200
> > earlycon=lpuart32,0x5a090000,115200";
> >     stdout-path = "/serial@5a090000";
> >     #address-cells = <0x00000002>;
> >     #size-cells = <0x00000002>;
> >     module@0 {
> >             bootargs = "earlycon=xen console=hvc0 loglevel=8
> > root=/dev/mmcblk0p2 rootwait rw";
> >             xen,dom0-bootargs = "console=dtuart dtuart=ttyLP3
> > earlycon=lpuart3,0x5a090000,115200";
> >             compatible = "xen,linux-zimage", "xen,multiboot-module";
> >             reg = <0x00000000 0x82100000 0x00000000 0x01490a00>;
> >     };
> > };
> 
> Thank you for the input. At first glance, I don't see the issue with the
node
> but it looks like Xen is not happy with it.
> 
> I am not entirely sure why yet. Do you mind to enable earlyprintk for your
> platform? With that you should be able to get more information about the
> list of modules discovered.

I didn't manage to enable earlyprintk. 
But I enabled kconfig "Developer Checks": "Verbose debug messages" and
"Devicetree debug messages" and put some extra printks in the Xen Source and
was able to locate the problem. 

When parsing the fdt, the memory@80000000 node throws an error, which stops
the parsing before the /chosen node was found and so no kernel boot module
was found for dom0. 

I bypassed this by putting the /chosen node at the top of the fdt, so that
module@0 gets parsed before the functions arrives at the memory node. After
this change, Dom0 was successfully booted by Xen.

I have created a pastebin https://pastebin.com/JBjKNvPP for future
reference. 
At pastebin line 215 Xen already found the kernel boot module.
So I put 
"boot_fdt_info(device_tree_flattened, fdt_paddr);"
in start_xen (arch/arm/setup.c) after the 
"console_init_postirq();"
call, so tha the function is run a second time, but this time the outputs
are shown in the bootlog (starting pastebin l. 247).
Starting at pastebin line 337, parser arrives at memory node.
Line 349 is what is specified in the fdt (which I copy here for reference)

memory@80000000 {
        device_type = "memory";
        reg = <0x00000000 0x80000000 0 0x40000000>;
};

The first bank found (l. 349) is the one from the fdt node. I am not sure
where the second bank (l. 350) comes from. The the second bank's size=0 ist
what causes the parse to fail and will Xen prevent from finding the boot
kernel for dom0 if the /chosen node comes after the memory node in the fdt.

Thanks for the help wih this issue Julien and Peng!

Regards,

Daniel

Attachment: smime.p7s
Description: S/MIME cryptographic signature