[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] Re: [Xen-devel] Shattering superpages impact on IOMMU in Xen
Hi, Andrew On Mon, Apr 3, 2017 at 7:42 PM, Andrew Cooper <andrew.cooper3@xxxxxxxxxx> wrote: > On 03/04/17 17:24, Oleksandr Tyshchenko wrote: >> Hi, all. >> >> Playing with non-shared IOMMU in Xen on ARM I faced one interesting >> thing. I found out that the superpages were shattered during domain >> life cycle. >> This is the result of mapping of foreign pages, ballooning memory, >> even if domain maps Xen shared pages, etc. >> I don't bother with the memory fragmentation at the moment. But, >> shattering bothers me from the IOMMU point of view. >> As the Xen owns IOMMU it might manipulate IOMMU page tables when >> passthoughed/protected device doing DMA in Linux. It is hard to detect >> when the DMA transaction isn't in progress >> in order to prevent this race. So, if we have inflight transaction >> from a device when changing IOMMU mapping we might get into trouble. >> Unfortunately, not in all the cases the >> faulting transaction can be restarted. The chance to hit the problem >> increases during shattering. >> >> I did next test: >> The dom0 on my setup contains ethernet IP that are protected by IOMMU. >> What is more, as the IOMMU I am playing with supports superpages (2M, >> 1G) the IOMMU driver >> takes into account these capabilities when building page tables. As I >> gave 256 MB for dom0, the IOMMU mapping was built by 2M memory blocks >> only. As I am using NFS for both dom0 and domU the ethernet IP >> performs DMA transactions almost all the time. >> Sometimes, I see the IOMMU page faults during creating guest domain. I >> think, it happens during Xen is shattering 2M mappings 4K mappings (it >> unmaps dom0 pages by one 4K page at a time, then maps domU pages there >> for copying domU images). >> But, I don't see any page faults when the IOMMU page table was built >> by 4K pages only. >> >> I had a talk with Julien on IIRC and we came to conclusion that the >> safest way would be to use 4K pages to prevent shattering, so the >> IOMMU shouldn't report superpage capability. >> On the other hand, if we build IOMMU from 4K pages we will have >> performance drop (during building, walking page tables), TLB pressure, >> etc. >> Another possible solution Julien was suggesting is to always >> ballooning with 2M, 1G, and not using 4K. That would help us to >> prevent shattering effect. >> The discussion was moved to the ML since it seems to be a generic >> issue and the right solution should be think of. >> >> What do you think is the right way to follow? Use 4K pages and don't >> bother with shattering or try to optimize? And if the idea to make >> balloon mechanism smarter makes sense how to teach balloon to do so? >> Thank you. > > Ballooning and foreign mappings are terrible for trying to retain > superpage mappings. No OS, not even Linux, can sensibly provide victim > pages in a useful way to avoid shattering. > > If you care about performance, don't ever balloon. Foreign mappings in > translated guests should start from the top of RAM, and work upwards. I understand about disabling ballooning mechanism. I will keep it in mind. > > > As for the IOMMU specifically, things are rather easier. It is the > guests responsibility to ensure that frames offered up for ballooning or > foreign mappings are unused. Therefore, if anything cares about the > specific 4K region becoming non-present in the IOMMU mappings, it is the > guest kernels fault for offering up a frame already in use. > > For the shattering however, It is Xen's responsibility to ensure that > all other mappings stay valid at all points. The correct way to do this > is to construct a new L1 table, mirroring the L2 superpage but lacking > the specific 4K mapping in question, then atomically replace the L2 > superpage entry with the new L1 table, then issue an IOMMU TLB > invalidation to remove any cached mappings. I think I do almost the same. > > By following that procedure, all DMA within the 2M region, but not > hitting the 4K frame, won't observe any interim lack of mappings. It > appears from your description that Xen isn't following the procedure. > > ~Andrew Thank you. -- Regards, Oleksandr Tyshchenko _______________________________________________ Xen-devel mailing list Xen-devel@xxxxxxxxxxxxx https://lists.xen.org/xen-devel
|
Lists.xenproject.org is hosted with RackSpace, monitoring our |