Re: Xen on Pi4: Xen doesn't work with overlays from Raspberry Pi 5.4 kernel

[Roman Shaposhnik <roman@xxxxxxxxxx>]

On Mon, Jun 15, 2020 at 6:45 PM Corey Minyard <cminyard@xxxxxxxxxx> wrote:
>
> On Mon, Jun 15, 2020 at 06:05:57PM -0700, Roman Shaposhnik wrote:
> > On Mon, Jun 15, 2020 at 6:02 PM Corey Minyard <cminyard@xxxxxxxxxx> wrote:
> > >
> > > On Mon, Jun 15, 2020 at 05:14:21PM -0700, Stefano Stabellini wrote:
> > > > On Mon, 15 Jun 2020, Christopher Clark wrote:
> > > > > On Wed, Jun 10, 2020 at 7:21 PM Roman Shaposhnik <roman@xxxxxxxxxx> 
> > > > > wrote:
> > > > > >
> > > > > > On Wed, Jun 10, 2020 at 11:54 AM Corey Minyard 
> > > > > > <cminyard@xxxxxxxxxx> wrote:
> > > > > > >
> > > > > > > I had been working on Xen on the Pi4 by throwing kernels I 
> > > > > > > compiled onto
> > > > > > > existing sd cards, and this was working fine.  I finally got to a 
> > > > > > > full
> > > > > > > yocto build of the system, and it didn't boot.
> > > > > > >
> > > > > > > In fact, Xen didn't print anything at all, and nothing happens 
> > > > > > > that
> > > > > > > might suggest it's booting without any console output.
> > > > >
> > > > > I've reproduced this. Linux 4.19 from the Raspberry Pi kernel branch
> > > > > works fine, whereas I see no console output from the kernel once Xen
> > > > > tries to hand off to dom0 with either a 5.4 or 5.6 kernel.
> > > > >
> > > > > > > I traced the issue down to the vc4-fkms-v3d dtoverly.  With 
> > > > > > > everything
> > > > > > > else the same, the 4.19 version of that overlay works, and the 5.4
> > > > > > > version does not work.  It also didn't work if I completely 
> > > > > > > removed the
> > > > > > > overlay.  The base device trees are the same between the two 
> > > > > > > kernels.
> > > > > > >
> > > > > > > Looking at the overlay changes between the versions and Xen 
> > > > > > > source, I
> > > > > > > can't trace down anything that would cause an issue.  Has anyone 
> > > > > > > seen
> > > > > > > this issue of have any ideas?
> > > > >
> > > > > Seen it: yes, but no progress on resolving it to report at this point.
> > > > >
> > > > > > FWIW: I ran into very similar issues, ditched 5.4 kernel and moved 
> > > > > > to 5.6.x:
> > > > > >     https://github.com/raspberrypi/linux/tree/rpi-5.6.y
> > > > > >
> > > > > > The 5.6.14 seems to be working quite nicely with Xen for me (and 
> > > > > > Stefano).
> > > > >
> > > > > Hi Roman - is there a specific commit in that rpi-5.6.y branch that
> > > > > you guys have working ok?
> > > > > It looks like the bcm2711_defconfig file wasn't included in the kernel
> > > > > source tree of that branch at the point the kernel version was bumped
> > > > > up to 5.6.14, so is there somewhere else to look for a matching kernel
> > > > > config?
> > > >
> > > > I don't know if that is the issue but beware that some device trees
> > > > invert serial0 with serial1. Make sure to use /soc/serial@7e215040. You
> > > > can do that by passing dtuart=/soc/serial@7e215040 to the Xen command
> > > > line.
> > >
> > > I already have that set as part of the boot process, but it still
> > > doesn't print anything out once Xen is started.
> > >
> > > I tried the 5.6 device tree, and no help there, eithers.  I'm wondering
> > > if everyone is still running with the 4.19 device trees.
> >
> > As I pointed out in the EVE link above -- we're very happily running
> > with 5.6 device trees. They are, of course, taken from RPI kernel
> > tree.
>
> Ok, what version of Xen are you running?  I'm using 4.13 with the Pi
> patches, but I have not tried the master branch.

We're running 4.14 + additional patches (not sure which ones will make
it into 4.14 proper yet):
    https://github.com/lf-edge/eve/tree/master/pkg/xen/arch/aarch64

FWIW: the first patch is basically the delta between 4.13 and 4.14

Thanks,
Roman.