Re: [Xen-devel] [RFC PATCH v1 0/7] IPMMU-VMSA support on ARM

[Oleksandr Tyshchenko <olekstysh@xxxxxxxxx>]

Hi, Julien

On Tue, Aug 1, 2017 at 3:27 PM, Julien Grall <julien.grall@xxxxxxx> wrote:
> On 26/07/17 16:09, Oleksandr Tyshchenko wrote:
>>
>> From: Oleksandr Tyshchenko <oleksandr_tyshchenko@xxxxxxxx>
>>
>> Hi, all.
>
>
> Hi,
>
> Please CC maintainers and any relevant person on the cover letter. This is
> quite useful to have in the inbox.
Yes. I CCed guys who, I think, are/were involved in IPMMU-VMSA
development from Linux side +
IOMMU maintainers (mostly ARM). Sorry, if I missed someone or mistakenly added.

>
>> The purpose of this patch series is to add IPMMU-VMSA support to Xen on
>> ARM.
>> It is VMSA-compatible IOMMU that integrated in the newest Renesas R-Car
>> Gen3 SoCs (ARM).
>> And this IOMMU can't share the page table with the CPU since it doesn't
>> use the same page-table format
>> as the CPU on ARM therefore I name it "Non-shared" IOMMU.
>> This all means that current patch series must be based on "Non-shared"
>> IOMMU support [1]
>> for the IPMMU-VMSA to be functional inside Xen.
>>
>> The IPMMU-VMSA driver as well as the ARM LPAE allocator were directly
>> ported from BSP for Linux the vendor provides:
>> git://git.kernel.org/pub/scm/linux/kernel/git/horms/renesas-bsp.git
>> rcar-3.5.3
>
>
> I think this is probably a good starting point to discuss about IOMMU
> support in Xen. I skimmed through the patches and saw the words "rfc" and
> "ported from BSP".
Well, at the time of porting IPMMU-VMSA driver, BSP [1] had more
complete support than mainline [2]
and seems to have at the moment.
For example, mainline driver still has single IPMMU context while BSP
driver can have up to 8 contexts,
the maximum uTLBs mainline driver can handle is 32, but for BSP driver
this value was increased to 48, etc.
But, I see attempts to get all required support in [3]. So, when this
support reaches upsteam, I hope,
it won't be a big problem to rebase on mainline driver if we decide to
align with it.

>
> At the moment for IOMMU we rely on the Linux community to do the review, but
> this is not the case here as it is an RFC.  I can definitely help to check
> if it comply for Xen,
yes, please.

> but I don't have the competence to tell whether it is
> valid for the hardware.
>
> We may want to find a compromise to get it merged in Xen, but surely we
> don't want to build it by default at least until we had feedback from the
> community about the validity of the code here.
agree.

>
> As I mentioned above, we are currently borrowing drivers from Linux and
> adapting for Xen. Today we support SMMUv{1,2} (we need to resync it) and
> there are plan to add IPMMU-VMSA (this series) and SMMUv3.
It would be really nice to have IPMMU-VMSA support in Xen. Without
this support the SCF [4] we are developing right now
and even the Passthrough feature won't be fully functional on R-Car
Gen3 based boards powered by Xen hypervisor.

>
> I am aware that Linux IOMMU subsystem has growing quite a lot making more
> tricky to get support in Xen. I wanted to get feedback how complex from you
> and Sameer how complex it was and whether we should consider doing our own.

Yes, the IPMMU-VMSA Linux driver relies on some Linux functional
(IOMMU/DMA/io-pgtable frameworks) the Xen doesn't have (it is
expected). So, it took *some time*
to make Linux driver happy inside Xen). Moreover, this all resulted in
the fact that the driver looks complicated a bit).
A lot of different wrappers, #if 0, code style, etc.
On the other hand, I think, I will be able to fairly quickly align
with new BSP, etc.

But, I really don't know should we continue to follow this direction
or not, perhaps it will depend on
how complex the entity is and how much things we must pull together
with it to make it happy.

>
>> Patch series was rebased on Xen 4.9.0 release and tested on Renesas R-Car
>> Gen3 H3 ES2.0/M3 based boards
>> with devices assigned to different domains.
>>
>> You can find patch series here:
>> repo: https://github.com/otyshchenko1/xen.git branch: ipmmu_v2
>>
>> P.S. There is one more patch which needs to be brought back to life [2]
>> Any reasons why this patch hasn't been upstremed yet?
>
>
> The series didn't make it upstream. Feel free to resend it separately.
ok.

>
>>
>> Thank you.
>>
>> [1] [Xen-devel] [PATCH v2 00/13] "Non-shared" IOMMU support on ARM
>> https://www.mail-archive.com/xen-devel@xxxxxxxxxxxxx/msg115901.html
>>
>> [2] [Xen-devel] [PATCH v8 02/28] xen: Add log2 functionality
>> https://lists.xenproject.org/archives/html/xen-devel/2016-02/msg00031.html
>>
>> Oleksandr Tyshchenko (7):
>>   iommu/arm: ipmmu-vmsa: Add IPMMU-VMSA support
>>   iommu/arm: ipmmu-vmsa: Add Xen changes for main driver
>>   iommu/arm: ipmmu-vmsa: Add io-pgtables support
>>   iommu/arm: ipmmu-vmsa: Add Xen changes for io-pgtables
>>   iommu/arm: Build IPMMU-VMSA related stuff
>>   iommu/arm: ipmmu-vmsa: Deallocate page table asynchronously
>>   iommu/arm: ipmmu-vmsa: Enable VMSAv8-64 mode if IPMMU HW supports it
>>
>>  xen/drivers/passthrough/arm/Makefile         |    3 +
>>  xen/drivers/passthrough/arm/io-pgtable-arm.c | 1331 +++++++++++++
>>  xen/drivers/passthrough/arm/io-pgtable.c     |   91 +
>>  xen/drivers/passthrough/arm/io-pgtable.h     |  220 +++
>>  xen/drivers/passthrough/arm/ipmmu-vmsa.c     | 2611
>> ++++++++++++++++++++++++++
>>  5 files changed, 4256 insertions(+)
>>  create mode 100644 xen/drivers/passthrough/arm/io-pgtable-arm.c
>>  create mode 100644 xen/drivers/passthrough/arm/io-pgtable.c
>>  create mode 100644 xen/drivers/passthrough/arm/io-pgtable.h
>>  create mode 100644 xen/drivers/passthrough/arm/ipmmu-vmsa.c
>>
>
> Cheers,
>
> --
> Julien Grall

[1] 
https://git.kernel.org/pub/scm/linux/kernel/git/horms/renesas-bsp.git/tree/drivers/iommu/ipmmu-vmsa.c?h=v4.9/rcar-3.5.3
[2] 
http://elixir.free-electrons.com/linux/latest/source/drivers/iommu/ipmmu-vmsa.c
[3] https://lwn.net/Articles/725769/
[4] https://lists.xenproject.org/archives/html/xen-devel/2017-07/msg02124.html

-- 
Regards,

Oleksandr Tyshchenko

_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
https://lists.xen.org/xen-devel