Re: [Xen-devel] Question about DMA on 1:1 mapping dom0 of arm64

[Ian Campbell <ian.campbell@xxxxxxxxxx>]

On Fri, 2015-04-17 at 19:24 +0800, Chen Baozi wrote:
> Hi all,
> 
> According to my recent experience, there might be some problems of swiotlb
> dma map on 1:1 mapping arm64 dom0 with large memory. The issue is like below:
> 
> For those arm64 server with large memory, it is possible to set dom0_mem >
> 4G (e.g. I have one set with 16G). In this case, according to my 
> understanding,
> there is chance that the dom0 kernel needs to map some buffers above 4G to do

                                                                 ^below?

> DMA operations (e.g. in snps,dwmac ethernet driver). However, most DMA engines
> support only 32-bit physical address, thus aren't able to operate directly on
> those memory.

Even on arm64 systems with RAM above 4GB? That seems.... short-sighted.
Oh well, I suppose we have to live with it.

>  IIUC, swiotlb is implemented to solve this (using bounce buffer),
> if there is no IOMMU or IOMMU is not enabled on the system. Sadly, it seems
> that xen_swiotlb_map_page in my dom0 kernel allocates
> (start_dma_addr = 0x944800000) the buffers for DMA above 4G which fails
> dma_capable() checking and was then unable to return from 
> xen_swiotlb_map_page()
> successfully.

The swiotlb bounce buffer have been allocated below 4GB? I suspect that
xen_swiotlb_init is buggy for ARM -- it allocates some random pages and
then swizzles the backing pages for ones < 4G, but that won't work on an
ARM dom0 with a 1:1 mapping, I don't think. Do you see error messages
along those lines?

Essentially I think either xen_swiotlb_fixup is unable to work on ARM,
or the following:
        start_dma_addr = xen_virt_to_bus(xen_io_tlb_start);
is returning 1:1 and not reflecting the fixup.

> If I set dom0_mem to a small value (e.g. 512M), which makes all physical 
> memory
> of dom0 below 4G, everything goes fine.

So you are getting allocated memory below 4G?

You message on IRC suggested you weren't, did you hack around this?

I think we have two options, either xen_swiotlb_init allocates pages
below 4GB (e.g. __GFP_DMA) or we do something to allow xen_swiotlb_fixup
to actually work even on a 1:1 dom0.

Although the first option seems preferable at first glance it has the
short coming that it requires dom0 to have some memory below 4GB, which
might not necessarily be the case. The second option seems like it might
be uglier but doesn't suffer from this issue.

Can you please look and find out if the IPA at 0x944800000 is actually
backed by 1:1 RAM or if xen_swiotlb_fixup has done it's job and updated
things such that the associated PAs are below 4GB?

Ian.

> I am not familiar with swiotlb-xen, so there would be misunderstanding about
> the current situation. Fix me if I did/understood anything wrong.
> 
> Any ideas?
> 
> Cheers,
> 
> Chen Baozi



_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel