Re: [XEN PATCH v1] xen/arm : Add support for SMMUv3 driver

[Rahul Singh <Rahul.Singh@xxxxxxx>]

Hello Oleksandr,

> On 6 Nov 2020, at 2:22 pm, Oleksandr Andrushchenko 
> <Oleksandr_Andrushchenko@xxxxxxxx> wrote:
> 
> Hi, Rahul!
> 
> On 11/6/20 3:58 PM, Rahul Singh wrote:
>> Hello Oleksandr,
>> 
>>> On 6 Nov 2020, at 1:00 pm, Oleksandr Andrushchenko 
>>> <Oleksandr_Andrushchenko@xxxxxxxx> wrote:
>>> 
>>> Hello, Rahul!
>>> 
>>> On 11/6/20 2:48 PM, Rahul Singh wrote:
>>>> Hello Oleksandr,
>>>> 
>>>>> On 2 Nov 2020, at 10:12 am, Oleksandr Andrushchenko 
>>>>> <Oleksandr_Andrushchenko@xxxxxxxx> wrote:
>>>>> 
>>>>> Hi,
>>>>> 
>>>>> On 11/2/20 11:55 AM, Bertrand Marquis wrote:
>>>>>> Hi,
>>>>>> 
>>>>>>> On 2 Nov 2020, at 05:55, Oleksandr Andrushchenko <andr2000@xxxxxxxxx> 
>>>>>>> wrote:
>>>>>>> 
>>>>>>> Hi, Julien!
>>>>>>> 
>>>>>>> On 10/30/20 7:18 PM, Julien Grall wrote:
>>>>>>>> Hi Oleksandr,
>>>>>>>> 
>>>>>>>> On 30/10/2020 10:44, Oleksandr Andrushchenko wrote:
>>>>>>>>> On 10/20/20 6:25 PM, Rahul Singh wrote:
>>>>>>>>>> Add support for ARM architected SMMUv3 implementations. It is based 
>>>>>>>>>> on
>>>>>>>>>> the Linux SMMUv3 driver.
>>>>>>>>>> 
>>>>>>>>>> Major differences between the Linux driver are as follows:
>>>>>>>>>> 1. Only Stage-2 translation is supported as compared to the Linux 
>>>>>>>>>> driver
>>>>>>>>>>      that supports both Stage-1 and Stage-2 translations.
>>>>>>>>> First of all thank you for the efforts!
>>>>>>>>> 
>>>>>>>>> I tried the patch with QEMU and would like to know if my 
>>>>>>>>> understanding correct
>>>>>>>>> 
>>>>>>>>> that this combination will not work as of now:
>>>>>>>>> 
>>>>>>>>> (XEN) SMMUv3: /smmuv3@9050000: SMMUv3: DT value = eventq
>>>>>>>>> (XEN) Data Abort Trap. Syndrome=0x1940010
>>>>>>>>> (XEN) Walking Hypervisor VA 0x40031000 on CPU0 via TTBR 
>>>>>>>>> 0x00000000b8469000
>>>>>>>>> (XEN) 0TH[0x0] = 0x00000000b8468f7f
>>>>>>>>> 
>>>>>>>>> [snip]
>>>>>>>>> 
>>>>>>>>> If this is expected then is there any plan to make QEMU work as well?
>>>>>>>>> 
>>>>>>>>> I see [1] says that "Only stage 1 and AArch64 PTW are supported." on 
>>>>>>>>> QEMU side.
>>>>>>>> Just for clarication, you are trying to boot Xen on QEMU, right?
>>>>>>> Exactly
>>>>>>>> You might be able to use the stage-1 page-tables to isolate each 
>>>>>>>> device in Xen. However, I don't think you will be able to share the 
>>>>>>>> P2M because the page-tables layout between stage-1 and stage-2 is 
>>>>>>>> different.
>>>>>>> So, it is even more work then
>>>>>> Overall it would make more sense to spend some time adding proper 
>>>>>> support in Qemu then trying to modify the driver to support Qemu right 
>>>>>> now.
>>>>>> 
>>>>>>>>> We are interested in QEMU/SMMUv3 as a flexible platform for PCI 
>>>>>>>>> passthrough
>>>>>>>>> 
>>>>>>>>> implementation, so it could allow testing different setups and 
>>>>>>>>> configurations with QEMU.
>>>>>>>> I would recommend to get the SMMU supporting supporting stage-2 
>>>>>>>> page-tables.
>>>>>>> You mean in QEMU?
>>>>>> See before.
>>>>>> 
>>>>>>>> Regardless that, I think Xen should be able to say the SMMU is not 
>>>>>>>> supported rather than crashing.
>>>>>>> Yes, that would be nice
>>>>>> Fully agree and we will look into that.
>>>>>> 
>>>>>> Anything you could share so that we could quickly reproduce your setup 
>>>>>> would be more then great.
>>>>> Nothing special,
>>>>> 
>>>>> qemu/aarch64-softmmu/qemu-system-aarch64 -machine type=virt -machine 
>>>>> virt,gic-version=2 \
>>>>> 
>>>>> -machine virtualization=true -cpu cortex-a57 -smp 4 -m 2048 -nic 
>>>>> user,hostfwd=tcp:127.0.0.1:2222-:22 \
>>>>> 
>>>>> -nographic -serial mon:stdio [..snip..]
>>>>> 
>>>>> I also set iommu to smmuv3 in my tests, QEMU emulator version 4.2.1
>>>> I just checked and confirmed that QEMU is booting with XEN SMMUv3 patch 
>>>> and XEN is able to say SMMU translation is not supported. As XEN supports 
>>>> Stage-2 translation and QEMU supports Stage-1 only.
>>>> 
>>>> 
>>>> (XEN) SMMUv3: /smmuv3@9050000: SMMUv3: DT value = eventq
>>>> (XEN) SMMUv3: /smmuv3@9050000: IDR0.COHACC overridden by FW configuration 
>>>> (false)
>>>> (XEN) SMMUv3: /smmuv3@9050000: no translation support!
>>>> (XEN) I/O virtualisation disabled
>>>> 
>>>> Only difference I observed is that you have to add option "-machine 
>>>> virt,iommu=smmuv3 “ when launching the QEMU.
>>> I do use the option
>> I used "-machine virt,iommu=smmuv3 “  option while creating the virt-dtb and 
>> while launching the QEMU.
>> I also observed the same error what you observed if I am not using the 
>> "-machine virt,iommu=smmuv3 “ options when launching the QEMU so I thought 
>> this might be case for you also but anyways you have use the options it 
>> might be other issue.
> 
> Hm, probably that was on my side as now I can see:
> 
> (XEN) SMMUv3: /smmuv3@9050000: SMMUv3: DT value = eventq
> (XEN) SMMUv3: /smmuv3@9050000: IDR0.COHACC overridden by FW configuration 
> (false)
> (XEN) SMMUv3: /smmuv3@9050000: no translation support!
> (XEN) I/O virtualisation disabled
> (XEN)
> (XEN) ****************************************
> (XEN) Panic on CPU 0:
> (XEN) Couldn't configure correctly all the IOMMUs.
> (XEN) ****************************************
> (XEN)
> (XEN) Manual reset required ('noreboot' specified)
> 
> So, sorry for the noise, I might have misconfigured something it seems
> 
> When you say "Xen is booting" do you mean you see the same panic?

Yes I observe the same. 

We have to decide now if for SMMUv3 there is no translation support do we have 
to print the logs and move forward  or as above return error to iommu_setup 
that will cal panic().

Regards,
Rahul

> 
> Thank you,
> 
> Oleksandr
> 
>> 
>>>> Please let me know if it also works for you.
>>> Well, I should have reported that earlier that I do not use the staging Xen 
>>> at the moment,
>>> 
>>> it is 4.14.0. So, can this be a problem with that Xen version?
>> I don’t think so this is the problem with the XEN version.
>>> Anyways, if it works with the staging then everything looks ok
>>> 
>>> Thank you,
>>> 
>>> Oleksandr
>>> 
>>>>>> Regards
>>>>>> Bertrand
>>>>>> 
>>>>>>>> Cheers,
>>>>>>>> 
>>>>>>> Thank you,
>>>>>>> 
>>>>>>> Oleksandr
>> Regards,
>> Rahul