Re: Troubles running Xen on Raspberry Pi 4 with 5.6.1 DomU

[Julien Grall <julien@xxxxxxx>]

Hi,

On 02/05/2020 03:16, Corey Minyard wrote:
> On Fri, May 01, 2020 at 06:06:11PM -0700, Roman Shaposhnik wrote:
> > On Fri, May 1, 2020 at 4:42 AM Corey Minyard <minyard@xxxxxxx> wrote:
> > > On Thu, Apr 30, 2020 at 07:20:05PM -0700, Roman Shaposhnik wrote:
> > > > Hi!
> > > >
> > > > I'm trying to run Xen on Raspberry Pi 4 with 5.6.1 stock,
> > > > upstream kernel. The kernel itself works perfectly well
> > > > on the board. When I try booting it as Dom0 under Xen,
> > > > it goes into a stacktrace (attached).
> > > Getting Xen working on the Pi4 requires a lot of moving parts, and they
> > > all have to be right.
> > Tell me about it! It is a pretty frustrating journey to align
> > everything just right.
> > On the other hand -- it seems worth to enable RPi as an ARM development
> > platform for Xen given how ubiquitous it is.
> >
> > Hence me trying to combine pristine upstream kernel (5.6.1) with
> > pristine upstream
> > Xen to enable 100% upstream developer workflow on RPi.
> >
> > > > Looking at what nice folks over at Dornerworks have previously
> > > > done to make RPi kernels boot as Dom0 I've come across these
> > > > 3 patches:
> > > >      https://github.com/dornerworks/xen-rpi4-builder/tree/master/patches/linux
> > > >
> > > > The first patch seems irrelevant (unless I'm missing something
> > > > really basic here).
> > > It might be irrelevant for your configuration, assuming that Xen gets
> > > the right information from EFI.  I haven't tried EFI booting.
> > I'd doing a bit of belt-and-suspenders strategy really -- I'm actually
> > using UEFI u-boot implementation that pre-populates device trees
> > and then I'm also forcing an extra copy of it to be load explicitly
> > via the GRUB devicetree command (GRUB runs as a UEFI payload).
> >
> > I also have access to the semi-official TianoCore RPi4 port that seems
> > to be working pretty well: https://github.com/pftf/RPi4/releases/tag/v1.5
> > for booting all sort of UEFI payloads on RPi4.
> >
> > > > The 2nd patch applied with no issue (but
> > > > I don't think it is related) but the 3d patch failed to apply on
> > > > account of 5.6.1 kernel no longer having:
> > > >      dev->archdata.dev_dma_ops
> > > > E.g.
> > > >      
> > > > https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/arch/arm64/mm/dma-mapping.c?h=v5.6.1#n55
> > > >
> > > > I've tried to emulate the effect of the patch by simply introducing
> > > > a static variable that would signal that we already initialized
> > > > dev->dma_ops -- but that didn't help at all.
> > > >
> > > > I'm CCing Jeff Kubascik to see if the original authors of that Corey Minyard
> > > > to see if may be they have any suggestions on how this may be dealt
> > > > with.
> > > >
> > > > Any advice would be greatly appreciated!
> > > What's your Pi4 config.txt file look like? The GIC is not being handled
> > > correctly, and I'm guessing that configuration is wrong.  You can't use
> > > the stock config.txt file with Xen, you have to modify the configuration a
> > > bit.
> > Understood. I'm actually using a custom one:
> >      https://github.com/lf-edge/eve/blob/master/pkg/u-boot/rpi/config.txt
> >
> > I could swear that I had a GIC setting in it -- but apparently no -- so
> > I added the following at the top of what you could see at the URL above:
> >
> > total_mem=4096
> > enable_gic=1
> >
> > > I think just adding:
> > >
> > > enable_gic=1
> > > total_mem=1024
> > Right -- but my board has 4G memory -- so I think what I did above should work.
> Nope.  If you say 4096M of RAM, your issue is almost certainly DMA, but
> it's not (just) the Linux code.  On the Raspberry Pi 4, several devices
> cannot DMA to above 1024M of RAM, but Xen does not handle this.  The
> 1024M of RAM is a limitation you will have to live with until Xen has a
> fix.
IIUC, dom0 would need to have some memory below 1GB for this to work, am 
I correct?
If so could you try the following patch?

diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c
index 430708753642..002f49dba74b 100644
--- a/xen/arch/arm/domain_build.c
+++ b/xen/arch/arm/domain_build.c
@@ -282,7 +282,7 @@ static void __init allocate_memory_11(struct domain *d,
      */
     while ( order >= min_low_order )
     {
-        for ( bits = order ; bits <= (lowmem ? 32 : PADDR_BITS); bits++ )
+        for ( bits = order ; bits <= (lowmem ? 30 : PADDR_BITS); bits++ )
         {
             pg = alloc_domheap_pages(d, order, MEMF_bits(bits));
             if ( pg != NULL )
@@ -313,7 +313,7 @@ static void __init allocate_memory_11(struct domain *d,
     order = get_allocation_size(kinfo->unassigned_mem);
     while ( kinfo->unassigned_mem && kinfo->mem.nr_banks < NR_MEM_BANKS )
     {
-        pg = alloc_domheap_pages(d, order, lowmem ? MEMF_bits(32) : 0);
+        pg = alloc_domheap_pages(d, order, lowmem ? MEMF_bits(30) : 0);
         if ( !pg )
         {
             order --;


Cheers,

--
Julien Grall