Re: [Xen-devel] Question about DMA on 1:1 mapping dom0 of arm64

[Stefano Stabellini <stefano.stabellini@xxxxxxxxxxxxx>]

On Fri, 17 Apr 2015, Ian Campbell wrote:
> On Fri, 2015-04-17 at 19:24 +0800, Chen Baozi wrote:
> > Hi all,
> > 
> > According to my recent experience, there might be some problems of swiotlb
> > dma map on 1:1 mapping arm64 dom0 with large memory. The issue is like 
> > below:
> > 
> > For those arm64 server with large memory, it is possible to set dom0_mem >
> > 4G (e.g. I have one set with 16G). In this case, according to my 
> > understanding,
> > there is chance that the dom0 kernel needs to map some buffers above 4G to 
> > do
> 
>                                                                  ^below?
> 
> > DMA operations (e.g. in snps,dwmac ethernet driver). However, most DMA 
> > engines
> > support only 32-bit physical address, thus aren't able to operate directly 
> > on
> > those memory.
> 
> Even on arm64 systems with RAM above 4GB? That seems.... short-sighted.
> Oh well, I suppose we have to live with it.
> 
> >  IIUC, swiotlb is implemented to solve this (using bounce buffer),
> > if there is no IOMMU or IOMMU is not enabled on the system. Sadly, it seems
> > that xen_swiotlb_map_page in my dom0 kernel allocates
> > (start_dma_addr = 0x944800000) the buffers for DMA above 4G which fails
> > dma_capable() checking and was then unable to return from 
> > xen_swiotlb_map_page()
> > successfully.
> 
> The swiotlb bounce buffer have been allocated below 4GB? I suspect that
> xen_swiotlb_init is buggy for ARM -- it allocates some random pages and
> then swizzles the backing pages for ones < 4G, but that won't work on an
> ARM dom0 with a 1:1 mapping, I don't think. Do you see error messages
> along those lines?
> 
> Essentially I think either xen_swiotlb_fixup is unable to work on ARM,
> or the following:
>         start_dma_addr = xen_virt_to_bus(xen_io_tlb_start);
> is returning 1:1 and not reflecting the fixup.

The swiotlb on arm doesn't necessarily get memory under 4G, see my other reply.


> > If I set dom0_mem to a small value (e.g. 512M), which makes all physical 
> > memory
> > of dom0 below 4G, everything goes fine.
> 
> So you are getting allocated memory below 4G?
> 
> You message on IRC suggested you weren't, did you hack around this?
> 
> I think we have two options, either xen_swiotlb_init allocates pages
> below 4GB (e.g. __GFP_DMA) or we do something to allow xen_swiotlb_fixup
> to actually work even on a 1:1 dom0.

I don't think that making xen_swiotlb_fixup work on ARM is a good idea:
it would break the 1:1.


> Although the first option seems preferable at first glance it has the
> short coming that it requires dom0 to have some memory below 4GB, which
> might not necessarily be the case.

I think we should arrange dom0 to get some memory under 4G to begin
with, not necessarily all of it.


> The second option seems like it might
> be uglier but doesn't suffer from this issue.
> 
> Can you please look and find out if the IPA at 0x944800000 is actually
> backed by 1:1 RAM or if xen_swiotlb_fixup has done it's job and updated
> things such that the associated PAs are below 4GB?

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel