Re: xen-swiotlb issue when NVMe driver is enabled in Dom0 on ARM

[Stefano Stabellini <sstabellini@xxxxxxxxxx>]

On Fri, 15 Apr 2022, Christoph Hellwig wrote:
> On Thu, Apr 14, 2022 at 01:39:23PM -0700, Stefano Stabellini wrote:
> > OK, now we know that the code path with Xen is correct and it is the
> > same code path taken (dma_alloc_direct) as when !CONFIG_XEN and !SMMU.
> > That is how it should be.
> > 
> > I cannot explain why dma_alloc_direct() would fail when called from
> > xen_swiotlb_alloc_coherent(), but it would succeed when called from
> > dma_alloc_attrs() without Xen.
> > 
> > I am not aware of any restrictions that xen or swiotlb-xen would
> > introduce in that regard. Unless you are just running out of memory
> > because dom0_mem too low.
> 
> The crash is deep down in the page allocator.  Even if memory was low
> it should no crash.  So there is some odd interaction between Xen
> and the page allocator going on.  I think nvme and dma-direct really
> are only the messenger here.


I cannot think of anything but if that is the case I guess it is more
likely related to reserved-memory not properly advertised or ACPI tables
not properly populated.


Rahul,

What happens if you boot Linux on Xen with swiotlb-xen disabled?


diff --git a/arch/arm64/mm/dma-mapping.c b/arch/arm64/mm/dma-mapping.c
index 93e87b287556..26d9da58f2c6 100644
--- a/arch/arm64/mm/dma-mapping.c
+++ b/arch/arm64/mm/dma-mapping.c
@@ -51,9 +51,4 @@ void arch_setup_dma_ops(struct device *dev, u64 dma_base, u64 
size,
        dev->dma_coherent = coherent;
        if (iommu)
                iommu_setup_dma_ops(dev, dma_base, size);
-
-#ifdef CONFIG_XEN
-       if (xen_initial_domain())
-               dev->dma_ops = &xen_swiotlb_dma_ops;
-#endif
 }
 

I thought Bertrand said that "it works" but I can't imagine how
swiotlb-xen could be the cause of the Linux stacktrace now.