Re: [Xen-devel] Question about DMA on 1:1 mapping dom0 of arm64

[Chen Baozi <cbz@xxxxxxxxxx>]

Hi Stefano,

On Fri, Apr 17, 2015 at 03:32:20PM +0100, Stefano Stabellini wrote:
> On Fri, 17 Apr 2015, Chen Baozi wrote:
> > Hi all,
> > 
> > According to my recent experience, there might be some problems of swiotlb
> > dma map on 1:1 mapping arm64 dom0 with large memory. The issue is like 
> > below:
> > 
> > For those arm64 server with large memory, it is possible to set dom0_mem >
> > 4G (e.g. I have one set with 16G). In this case, according to my 
> > understanding,
> > there is chance that the dom0 kernel needs to map some buffers above 4G to 
> > do
> > DMA operations (e.g. in snps,dwmac ethernet driver). However, most DMA 
> > engines
> > support only 32-bit physical address, thus aren't able to operate directly 
> > on
> > those memory. IIUC, swiotlb is implemented to solve this (using bounce 
> > buffer),
> > if there is no IOMMU or IOMMU is not enabled on the system. Sadly, it seems
> > that xen_swiotlb_map_page in my dom0 kernel allocates
> > (start_dma_addr = 0x944800000) the buffers for DMA above 4G which fails
> > dma_capable() checking and was then unable to return from 
> > xen_swiotlb_map_page()
> > successfully.
> >
> > If I set dom0_mem to a small value (e.g. 512M), which makes all physical 
> > memory
> > of dom0 below 4G, everything goes fine.
> > 
> > I am not familiar with swiotlb-xen, so there would be misunderstanding about
> > the current situation. Fix me if I did/understood anything wrong.
> > 
> > Any ideas?
> 
> I think that the problem is that xen_swiotlb_init doesn't necessarely allocate
> memory under 4G on arm/arm64.
> 
> xen_swiotlb_init calls __get_free_pages to allocate memory, so the pages
> could easily be above 4G.  Subsequently xen_swiotlb_fixup is called on
> the allocated memory range, calling xen_create_contiguous_region and
> passing an address_bits mask. However xen_create_contiguous_region
> doesn't actually do anything at all on ARM.
> 
> I think that given that dom0 is mapped 1:1 on ARM, the easiest and best
> fix would be to simply allocate memory under 4G to begin with. Something
> like (maybe with an ifdef ARM around it):
> 
> diff --git a/drivers/xen/swiotlb-xen.c b/drivers/xen/swiotlb-xen.c
> index 810ad41..22ac33a 100644
> --- a/drivers/xen/swiotlb-xen.c
> +++ b/drivers/xen/swiotlb-xen.c
> @@ -235,7 +235,7 @@ retry:
>  #define SLABS_PER_PAGE (1 << (PAGE_SHIFT - IO_TLB_SHIFT))
>  #define IO_TLB_MIN_SLABS ((1<<20) >> IO_TLB_SHIFT)
>               while ((SLABS_PER_PAGE << order) > IO_TLB_MIN_SLABS) {
> -                     xen_io_tlb_start = (void 
> *)__get_free_pages(__GFP_NOWARN, order);
> +                     xen_io_tlb_start = (void 
> *)__get_free_pages(__GFP_NOWARN|__GFP_DMA32, order);
>                       if (xen_io_tlb_start)
>                               break;
>                       order--;

I have no idea if __GFP_DMA32 on arm64 has something wrong. But It looks like
that it doesn't help...

Here is the memory info about what xen has populated to dom0 (I did some hacks
to allocate_memory_11 to make it map some low memory banks to dom0):

(XEN) Allocating 1:1 mappings totalling 16384MB for dom0:
(XEN) BANK[0] 0x00000088000000-0x00000098000000 (256MB)
(XEN) BANK[1] 0x000000a0000000-0x000000f8000000 (1408MB)
(XEN) BANK[2] 0x00000400000000-0x00000600000000 (8192MB)
(XEN) BANK[3] 0x00000680000000-0x00000700000000 (2048MB)
(XEN) BANK[4] 0x00000800000000-0x00000900000000 (4096MB)
(XEN) BANK[5] 0x00000940000000-0x00000958000000 (384MB)

And Here is the printk info I got when trying to map a dma page:

enter xen_swiotlb_map_page.
phys = 0x9444e4042, dev_addr = 0x9444e4042, size = 0x600
start_dma_addr = 0x944800000
virt_to_phys(xen_io_tlb_start) = 0x944800000
Oh Well, have to allocate and map a bounce buffer.
map = 0x944800000
dev_addr = 0x944800000
*dev->dma_mask = 0xffffffff
!dma_capable(0xffffffc8bd384810, 0x944800000, 0x600)

And the patch I used for dom0 hacking:

diff --git a/drivers/xen/swiotlb-xen.c b/drivers/xen/swiotlb-xen.c
index 810ad41..96465cf 100644
--- a/drivers/xen/swiotlb-xen.c
+++ b/drivers/xen/swiotlb-xen.c
@@ -235,7 +235,7 @@ retry:
 #define SLABS_PER_PAGE (1 << (PAGE_SHIFT - IO_TLB_SHIFT))
 #define IO_TLB_MIN_SLABS ((1<<20) >> IO_TLB_SHIFT)
                while ((SLABS_PER_PAGE << order) > IO_TLB_MIN_SLABS) {
-                       xen_io_tlb_start = (void 
*)__get_free_pages(__GFP_NOWARN, order);
+                       xen_io_tlb_start = (void 
*)__get_free_pages(__GFP_NOWARN|__GFP_DMA32, order);
                        if (xen_io_tlb_start)
                                break;
                        order--;
@@ -391,6 +391,13 @@ dma_addr_t xen_swiotlb_map_page(struct device *dev, struct 
page *page,
        dma_addr_t dev_addr = xen_phys_to_bus(phys);
 
        BUG_ON(dir == DMA_NONE);
+       printk("enter xen_swiotlb_map_page.\n");
+       printk("phys = 0x%lx, dev_addr = 0x%lx, size = 0x%lx\n",
+               phys, dev_addr, size);
+       printk("start_dma_addr = 0x%lx\n", start_dma_addr);
+       printk("virt_to_phys(xen_io_tlb_start) = 0x%lx\n",
+               virt_to_phys(xen_io_tlb_start));
+
        /*
         * If the address happens to be in the device's DMA window,
         * we can safely return the device addr and not worry about bounce
@@ -403,27 +410,35 @@ dma_addr_t xen_swiotlb_map_page(struct device *dev, 
struct page *page,
                /* we are not interested in the dma_addr returned by
                 * xen_dma_map_page, only in the potential cache flushes 
executed
                 * by the function. */
+               printk("The address happens to be in the device's DMA 
window\n");
                xen_dma_map_page(dev, page, dev_addr, offset, size, dir, attrs);
                return dev_addr;
        }
 
+
        /*
         * Oh well, have to allocate and map a bounce buffer.
         */
+       printk("Oh Well, have to allocate and map a bounce buffer.\n");
        trace_swiotlb_bounced(dev, dev_addr, size, swiotlb_force);
 
        map = swiotlb_tbl_map_single(dev, start_dma_addr, phys, size, dir);
        if (map == SWIOTLB_MAP_ERROR)
                return DMA_ERROR_CODE;
+       printk("map = 0x%lx\n", map);
 
        xen_dma_map_page(dev, pfn_to_page(map >> PAGE_SHIFT),
                                        dev_addr, map & ~PAGE_MASK, size, dir, 
attrs);
        dev_addr = xen_phys_to_bus(map);
+       printk("dev_addr = 0x%lx\n", dev_addr);
+       printk("*dev->dma_mask = 0x%lx\n", *dev->dma_mask);
 
        /*
         * Ensure that the address returned is DMA'ble
         */
        if (!dma_capable(dev, dev_addr, size)) {
+               printk("!dma_capable(0x%p, 0x%lx, 0x%lx\n)",
+                       dev, dev_addr, size);
                swiotlb_tbl_unmap_single(dev, map, size, dir);
                dev_addr = 0;
        }

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel