Re: Xen on Pi4: Xen doesn't work with overlays from Raspberry Pi 5.4 kernel

[Roman Shaposhnik <roman@xxxxxxxxxx>]

On Mon, Jun 15, 2020 at 4:35 PM Christopher Clark
<christopher.w.clark@xxxxxxxxx> wrote:
>
> On Wed, Jun 10, 2020 at 7:21 PM Roman Shaposhnik <roman@xxxxxxxxxx> wrote:
> >
> > On Wed, Jun 10, 2020 at 11:54 AM Corey Minyard <cminyard@xxxxxxxxxx> wrote:
> > >
> > > I had been working on Xen on the Pi4 by throwing kernels I compiled onto
> > > existing sd cards, and this was working fine.  I finally got to a full
> > > yocto build of the system, and it didn't boot.
> > >
> > > In fact, Xen didn't print anything at all, and nothing happens that
> > > might suggest it's booting without any console output.
>
> I've reproduced this. Linux 4.19 from the Raspberry Pi kernel branch
> works fine, whereas I see no console output from the kernel once Xen
> tries to hand off to dom0 with either a 5.4 or 5.6 kernel.
>
> > > I traced the issue down to the vc4-fkms-v3d dtoverly.  With everything
> > > else the same, the 4.19 version of that overlay works, and the 5.4
> > > version does not work.  It also didn't work if I completely removed the
> > > overlay.  The base device trees are the same between the two kernels.
> > >
> > > Looking at the overlay changes between the versions and Xen source, I
> > > can't trace down anything that would cause an issue.  Has anyone seen
> > > this issue of have any ideas?
>
> Seen it: yes, but no progress on resolving it to report at this point.
>
> > FWIW: I ran into very similar issues, ditched 5.4 kernel and moved to 5.6.x:
> >     https://github.com/raspberrypi/linux/tree/rpi-5.6.y
> >
> > The 5.6.14 seems to be working quite nicely with Xen for me (and Stefano).
>
> Hi Roman - is there a specific commit in that rpi-5.6.y branch that
> you guys have working ok?

Pretty much the latest really. The problem is that it seems RPi ppl.
keep forcepushing to that branch. Hence what we had to do on
the EVE side is to basically "snapshot" it with this gigantic patch:
     
https://github.com/lf-edge/eve/blob/master/pkg/new-kernel/patches-5.6.x/0000-rpi-kernel-changes.patch

This needs to be applied on top of straight up upstream 5.6.14 (since
that's what the closest common ancestor was on that branch at the time)

In fact, the entire build process is captured here (if you're curious):
      https://github.com/lf-edge/eve/blob/master/pkg/new-kernel/Dockerfile
(you can literally just docker build . build the whole thing)

> It looks like the bcm2711_defconfig file wasn't included in the kernel
> source tree of that branch at the point the kernel version was bumped
> up to 5.6.14, so is there somewhere else to look for a matching kernel
> config?

Yes ;-) I can share the kind of config that has been pretty extensively
tested by us here at Project EVE:
     
https://github.com/lf-edge/eve/blob/master/pkg/new-kernel/kernel_config-5.6.x-aarch64

None of it is terribly EVE specific -- so you should have a reasonable
base-line.

Thanks,
Roman.