Re: Troubles running Xen on Raspberry Pi 4 with 5.6.1 DomU

[Roman Shaposhnik <roman@xxxxxxxxxx>]

On Wed, May 6, 2020 at 10:34 AM Stefano Stabellini
<sstabellini@xxxxxxxxxx> wrote:
>
> On Wed, 6 May 2020, Nataliya Korovkina wrote:
> > On Wed, May 6, 2020 at 9:43 AM Boris Ostrovsky
> > <boris.ostrovsky@xxxxxxxxxx> wrote:
> > >
> > >
> > > On 5/6/20 9:08 AM, Nataliya Korovkina wrote:
> > > > Hello,
> > > >
> > > > What I found out: rpi_firmware_property_list() allocates memory from
> > > > dma_atomic_pool which was mapped to VMALLOC region, so virt_to_page()
> > > > is not eligible in this case.
> > >
> > >
> > > So then it seems it didn't go through xen_swiotlb_alloc_coherent(). In
> > > which case it has no business calling xen_swiotlb_free_coherent().
> > >
> > >
> > > -boris
> > >
> > >
> > >
> > >
> >
> > It does go.
> > dma_alloc_coherent() indirectly calls xen_swiotlb_alloc_coherent(),
> > then  xen_alloc_coherent_pages() eventually calls arch_dma_alloc() in
> > remap.c which successfully allocates pages from atomic pool.
> >
> > The patch Julien offered for domian_build.c moved Dom0 banks in the
> > first G of RAM.
> > So it covered the previous symptom (a crash during allocation) because
> > now we avoid pathway  when we mark a page "XenMapped".
> >
> > But the symptom still remains in xen_swiotlb_free_coherent() because
> > "TestPage..." is called unconditionally. virt_to_page() is not
> > applicable to such allocations.
>
> Ah! So this is the crux of the issue. I saw this kind of problem before
> on ARM32 (in fact there are several comments warning not to use
> virt_to_phys on ARM in drivers/xen/swiotlb-xen.c).
>
>
> So, to recap we have 2 issues as far as I can tell:
>
> - virt_to_page not working in some cases on ARM, leading to a crash
> - WARN_ON for range_straddles_page_boundary which is normal on ARM
>
> The appended patch addresses them by:
>
> - using pfn_to_page instead virt_to_page
> - moving the WARN_ON under a #ifdef (Juergen might have a better
>   suggestion on how to rework the WARN_ON)
>
> Please let me know if this patch works!
>
> Cheers,
>
> Stefano
>
>
> diff --git a/drivers/xen/swiotlb-xen.c b/drivers/xen/swiotlb-xen.c
> index b6d27762c6f8..0a40ac332a4c 100644
> --- a/drivers/xen/swiotlb-xen.c
> +++ b/drivers/xen/swiotlb-xen.c
> @@ -322,7 +322,7 @@ xen_swiotlb_alloc_coherent(struct device *hwdev, size_t 
> size,
>                         xen_free_coherent_pages(hwdev, size, ret, 
> (dma_addr_t)phys, attrs);
>                         return NULL;
>                 }
> -               SetPageXenRemapped(virt_to_page(ret));
> +               SetPageXenRemapped(pfn_to_page(PFN_DOWN(phys)));
>         }
>         memset(ret, 0, size);
>         return ret;
> @@ -346,9 +346,14 @@ xen_swiotlb_free_coherent(struct device *hwdev, size_t 
> size, void *vaddr,
>         /* Convert the size to actually allocated. */
>         size = 1UL << (order + XEN_PAGE_SHIFT);
>
> -       if (!WARN_ON((dev_addr + size - 1 > dma_mask) ||
> -                    range_straddles_page_boundary(phys, size)) &&
> -           TestClearPageXenRemapped(virt_to_page(vaddr)))
> +#ifdef CONFIG_X86
> +       if (WARN_ON(dev_addr + size - 1 > dma_mask) ||
> +                   range_straddles_page_boundary(phys, size)) {
> +           return;
> +       }
> +#endif
> +
> +       if (TestClearPageXenRemapped(pfn_to_page(PFN_DOWN(phys))))
>                 xen_destroy_contiguous_region(phys, order);
>
>         xen_free_coherent_pages(hwdev, size, vaddr, (dma_addr_t)phys, attrs);

Stefano, with your patch applied, I'm still getting:

[    0.590705] Unable to handle kernel paging request at virtual
address fffffe0003700000

However, Boris' patch seems to get us much closer. It would be awesome if you
can take a look at that (plus the additional DMA issue that seems to
be dependent
on how much memory I allocate to Dom0).

Thanks,
Roman.