Re: Xen on Pi4: Xen doesn't work with overlays from Raspberry Pi 5.4 kernel

[Christopher Clark <christopher.w.clark@xxxxxxxxx>]

On Mon, Jun 15, 2020 at 6:49 PM Roman Shaposhnik <roman@xxxxxxxxxx> wrote:
>
> On Mon, Jun 15, 2020 at 6:45 PM Corey Minyard <cminyard@xxxxxxxxxx> wrote:
> >
> > On Mon, Jun 15, 2020 at 06:05:57PM -0700, Roman Shaposhnik wrote:
> > > On Mon, Jun 15, 2020 at 6:02 PM Corey Minyard <cminyard@xxxxxxxxxx> wrote:
> > > >
> > > > On Mon, Jun 15, 2020 at 05:14:21PM -0700, Stefano Stabellini wrote:
> > > > > On Mon, 15 Jun 2020, Christopher Clark wrote:
> > > > > > On Wed, Jun 10, 2020 at 7:21 PM Roman Shaposhnik <roman@xxxxxxxxxx> 
> > > > > > wrote:
> > > > > > >
> > > > > > > On Wed, Jun 10, 2020 at 11:54 AM Corey Minyard 
> > > > > > > <cminyard@xxxxxxxxxx> wrote:
> > > > > > > >
> > > > > > > > I had been working on Xen on the Pi4 by throwing kernels I 
> > > > > > > > compiled onto
> > > > > > > > existing sd cards, and this was working fine.  I finally got to 
> > > > > > > > a full
> > > > > > > > yocto build of the system, and it didn't boot.
> > > > > > > >
> > > > > > > > In fact, Xen didn't print anything at all, and nothing happens 
> > > > > > > > that
> > > > > > > > might suggest it's booting without any console output.
> > > > > >
> > > > > > I've reproduced this. Linux 4.19 from the Raspberry Pi kernel branch
> > > > > > works fine, whereas I see no console output from the kernel once Xen
> > > > > > tries to hand off to dom0 with either a 5.4 or 5.6 kernel.
> > > > > >
> > > > > > > > I traced the issue down to the vc4-fkms-v3d dtoverly.  With 
> > > > > > > > everything
> > > > > > > > else the same, the 4.19 version of that overlay works, and the 
> > > > > > > > 5.4
> > > > > > > > version does not work.  It also didn't work if I completely 
> > > > > > > > removed the
> > > > > > > > overlay.  The base device trees are the same between the two 
> > > > > > > > kernels.
> > > > > > > >
> > > > > > > > Looking at the overlay changes between the versions and Xen 
> > > > > > > > source, I
> > > > > > > > can't trace down anything that would cause an issue.  Has 
> > > > > > > > anyone seen
> > > > > > > > this issue of have any ideas?
> > > > > >
> > > > > > Seen it: yes, but no progress on resolving it to report at this 
> > > > > > point.
> > > > > >
> > > > > > > FWIW: I ran into very similar issues, ditched 5.4 kernel and 
> > > > > > > moved to 5.6.x:
> > > > > > >     https://github.com/raspberrypi/linux/tree/rpi-5.6.y
> > > > > > >
> > > > > > > The 5.6.14 seems to be working quite nicely with Xen for me (and 
> > > > > > > Stefano).
> > > > > >
> > > > > > Hi Roman - is there a specific commit in that rpi-5.6.y branch that
> > > > > > you guys have working ok?
> > > > > > It looks like the bcm2711_defconfig file wasn't included in the 
> > > > > > kernel
> > > > > > source tree of that branch at the point the kernel version was 
> > > > > > bumped
> > > > > > up to 5.6.14, so is there somewhere else to look for a matching 
> > > > > > kernel
> > > > > > config?
> > > > >
> > > > > I don't know if that is the issue but beware that some device trees
> > > > > invert serial0 with serial1. Make sure to use /soc/serial@7e215040. 
> > > > > You
> > > > > can do that by passing dtuart=/soc/serial@7e215040 to the Xen command
> > > > > line.
> > > >
> > > > I already have that set as part of the boot process, but it still
> > > > doesn't print anything out once Xen is started.
> > > >
> > > > I tried the 5.6 device tree, and no help there, eithers.  I'm wondering
> > > > if everyone is still running with the 4.19 device trees.
> > >
> > > As I pointed out in the EVE link above -- we're very happily running
> > > with 5.6 device trees. They are, of course, taken from RPI kernel
> > > tree.
> >
> > Ok, what version of Xen are you running?  I'm using 4.13 with the Pi
> > patches, but I have not tried the master branch.
>
> We're running 4.14 + additional patches (not sure which ones will make
> it into 4.14 proper yet):
>     https://github.com/lf-edge/eve/tree/master/pkg/xen/arch/aarch64
>
> FWIW: the first patch is basically the delta between 4.13 and 4.14

Corey: try adding these to your u-boot script:

# Define the size and address cells
fdt set /chosen '#size-cells' <1>
fdt set /chosen '#address-cells' <1>

It looks like the 4.19 kernel device tree has those nodes set by the
time the u-boot script picks up after the GPU's work on the device
tree, but the more recent 5.4 and 5.6 kernels do not. Adding those in
within the u-boot script, I've been able to successfully boot the
Linux 5.4 and 5.6 kernels from meta-raspberrypi as well as a local
build of the Eve 5.6 kernel.

Stefano has a patch to do something similar in the Eve series here:
https://github.com/lf-edge/eve/blob/master/pkg/new-kernel/patches-5.6.x/0013-adding-stdout-path-overlay.patch

I've just posted an updated series to meta-virt to enable Xen on Rpi4
with Linux 5.4 (the current Yocto kernel version) here:
https://lists.yoctoproject.org/g/meta-virtualization/message/5449

Christopher