Re: [Xen-devel] Question about DMA on 1:1 mapping dom0 of arm64

[Chen Baozi <cbz@xxxxxxxxxx>]

On Fri, Apr 17, 2015 at 02:21:45PM +0100, Ian Campbell wrote:
> On Fri, 2015-04-17 at 19:24 +0800, Chen Baozi wrote:
> > Hi all,
> > 
> > According to my recent experience, there might be some problems of swiotlb
> > dma map on 1:1 mapping arm64 dom0 with large memory. The issue is like 
> > below:
> > 
> > For those arm64 server with large memory, it is possible to set dom0_mem >
> > 4G (e.g. I have one set with 16G). In this case, according to my 
> > understanding,
> > there is chance that the dom0 kernel needs to map some buffers above 4G to 
> > do
> 
>                                                                  ^below?
> 
> > DMA operations (e.g. in snps,dwmac ethernet driver). However, most DMA 
> > engines
> > support only 32-bit physical address, thus aren't able to operate directly 
> > on
> > those memory.
> 
> Even on arm64 systems with RAM above 4GB? That seems.... short-sighted.
> Oh well, I suppose we have to live with it.

I understand for most ARM SoCs, the DMA engines come from third party IP 
companies
which is arm32/arm64 independent. Thus, 32-bit address DMA engine should be 
common
even on arm64 system. The preferred way is to use/enable SMMU(IOMMU). However, 
we
are focusing on 1:1 mapping right now...

> 
> >  IIUC, swiotlb is implemented to solve this (using bounce buffer),
> > if there is no IOMMU or IOMMU is not enabled on the system. Sadly, it seems
> > that xen_swiotlb_map_page in my dom0 kernel allocates
> > (start_dma_addr = 0x944800000) the buffers for DMA above 4G which fails
> > dma_capable() checking and was then unable to return from 
> > xen_swiotlb_map_page()
> > successfully.
> 
> The swiotlb bounce buffer have been allocated below 4GB? 

I have no idea (about the exact behavior of bounce buffer). But I don't think
it has been allocated below 4GB on my board, for in that case it won't fail
dma_capable() in the end of xen_swiotlb_map_page().

> I suspect that
> xen_swiotlb_init is buggy for ARM -- it allocates some random pages and
> then swizzles the backing pages for ones < 4G, but that won't work on an
> ARM dom0 with a 1:1 mapping, I don't think. Do you see error messages
> along those lines?
> 
> Essentially I think either xen_swiotlb_fixup is unable to work on ARM,
> or the following:
>         start_dma_addr = xen_virt_to_bus(xen_io_tlb_start);
> is returning 1:1 and not reflecting the fixup.

Yes. It seems very likely what happened in my system.

> 
> > If I set dom0_mem to a small value (e.g. 512M), which makes all physical 
> > memory
> > of dom0 below 4G, everything goes fine.
> 
> So you are getting allocated memory below 4G?

If all the banks of memory that xen populate to dom0 is below 4G, yes. However,
if some banks of memory for dom0 is above 4G, usually not.

> 
> You message on IRC suggested you weren't, did you hack around this?

Yes. I did some hacks to help understand my situation earlier. What I have done
and observed is as below:

1. At the very beginning, I used the default dom0_mem value to boot the system,
which is 128M. And I didn't realize the DMA buffer problem.

2. I started to try more dom0_mem (16G). Then the ethernet driver reported
that it cannot initiate rx buffers (DMA buffers). And I found out that
allocate_memory_11 didn't populate any banks of memory below 4G for dom0.
At that time, I guessed the failure might be introduced because there is no
memory banks below 4G was populated. (there is only a 2GB address space below 4G
for physical memory on my platform, and there is a hole for PCI memory address
space above 4G before the memory address space continue.)

3. So I did some hacks to let lowmem=true manually in allocate_memory_11, which
made xen on arm64 acts similar as it is on arm32 that populates at least one
bank of memory below 4G to dom0. (this is the point when I send you message
on IRC.) I thought that can solve the problem, but it doesn't.

4. Then I found out once xen populated any banks of memory which is above 4G,
the ethernet driver would have chances (very likely, almost every time if
dom0_mem=16G) to use buffers above 4G, regardless whether dom0 has banks of
memory below 4G.

> 
> I think we have two options, either xen_swiotlb_init allocates pages
> below 4GB (e.g. __GFP_DMA) or we do something to allow xen_swiotlb_fixup
> to actually work even on a 1:1 dom0.
> 
> Although the first option seems preferable at first glance it has the
> short coming that it requires dom0 to have some memory below 4GB, which
> might not necessarily be the case. The second option seems like it might
> be uglier but doesn't suffer from this issue.
> 
> Can you please look and find out if the IPA at 0x944800000 is actually
> backed by 1:1 RAM or if xen_swiotlb_fixup has done it's job and updated
> things such that the associated PAs are below 4GB?

I am at home now and will check it out tomorrow. But I guess it should be
the first situation you mentioned.

Cheers,

Baozi.

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel