Re: problem running on rcar gen3 board (iommu?)

[Oleksandr Tyshchenko <Oleksandr_Tyshchenko@xxxxxxxx>]

On 08.10.22 22:14, Julien Grall wrote:
> (+Oleksandr)


Hello all


>
> On 29/09/2022 18:06, stsp wrote:
>> Hi!
>>
>> I am trying to run xen on an rcar H3
>> StarterKit (r8a77951 ES2.0).
>> First I created a working linux setup.
>> It uses latest linux git code and is
>> working well. >
>> Then I built xen, also from git,
>> at rev 211d8419ef.


I didn’t find the 211d8419ef revision in vanilla Xen.


>> Now when runnign xen, I get this:
>>
>> ---
>>
>> (XEN) ipmmu: /soc/iommu@e6740000: Failed to init IPMMU (-11)
>> (XEN) ipmmu: /soc/iommu@e7740000: Failed to init IPMMU (-11)
>> (XEN) ipmmu: /soc/iommu@e6570000: Failed to init IPMMU (-11)
>> (XEN) ipmmu: /soc/iommu@ff8b0000: Failed to init IPMMU (-11)
>> (XEN) ipmmu: P2M sharing is not supported in current SoC revision
>> (XEN) ipmmu: /soc/iommu@e67b0000: Failed to init IPMMU (-19)
>> (XEN) ipmmu: /soc/iommu@ec670000: Failed to init IPMMU (-11)
>> (XEN) ipmmu: /soc/iommu@fd800000: Failed to init IPMMU (-11)
>> (XEN) ipmmu: /soc/iommu@ffc80000: Failed to init IPMMU (-11)
>> (XEN) ipmmu: /soc/iommu@fe6b0000: Failed to init IPMMU (-11)
>> (XEN) ipmmu: /soc/iommu@fe6f0000: Failed to init IPMMU (-11)
>> (XEN) ipmmu: /soc/iommu@febd0000: Failed to init IPMMU (-11)
>> (XEN) ipmmu: /soc/iommu@fe990000: Failed to init IPMMU (-11)
>> (XEN) ipmmu: /soc/iommu@ec680000: Failed to init IPMMU (-11)
>> (XEN) ipmmu: /soc/iommu@e7730000: Failed to init IPMMU (-11)
>> (XEN) I/O virtualisation disabled
>> (XEN)
>> (XEN) ****************************************
>> (XEN) Panic on CPU 0:
>> (XEN) Couldn't configure correctly all the IOMMUs.
>> (XEN) ****************************************
>
> Adding Oleksandr who wrote the driver and also worked with the Renesas 
> SOCs.
>
> Cheers,
>
>> ---
>>
>> So I added the "iommu=0" option.
>> Now linux kernel starts but hangs
>> the entire system somewhere
>> around the first sdcard transfer.
>> I see xen is saying this on load:
>>
>> ---
>>
>> (XEN) Loading d0 kernel from boot module @ 000000007c936000
>> (XEN) Allocating 1:1 mappings totalling 512MB for dom0:
>> (XEN) BANK[0] 0x00000050000000-0x00000070000000 (512MB)
>>
>> ---
>>
>> If it does 1:1 mappings then perhaps
>> it is supposed to work w/o IOMMU?


Currently Dom0 on Arm is 1:1 mapped, so yes, DMA devices in Dom0 can 
work without IOMMU being enabled in Xen.


>> But why would it then insist on me
>> adding "iommu=0"?


Good question. Well, the IOMMUs initialization failed because that SoCs 
revision (r8a77951 ES2.0) does not support P2M sharing so cannot be used 
(and this is reported by the driver).
I assume, Xen was built with CONFIG_IPMMU_VMSA option explicitly enabled 
(this option is disabled by default) although config IPMMU_VMSA mentions 
the following:

       "Say Y here if you are using newest R-Car Gen3 SoCs revisions
       (H3 ES3.0, M3-W+, etc) or Gen4 SoCs which IPMMU hardware supports 
stage 2
       translation table format and is able to use CPU's P2M table as is."

So, the CONFIG_IPMMU_VMSA just shouldn’t be enabled if target H/W 
doesn’t match.

Or, if we indeed want/need to relax the behavior (do not panic and 
continue to operate with I/O virtualization disabled) if such a case happens
(I mean when driver honestly reports it cannot work due to objective 
reason(s)), the code should be updated (fixed?) a bit. I will let Julien 
to clarify, how he thinks it would be more correct.

 From the log you provided we see the following:
- Driver for all cache IPMMUs returns -EAGAIN because root IPMMU has 
been registered yet.
- Driver for root IPMMU returns -ENODEV because it detects incompatible 
SoC revision.

The Arm’s iommu_hardware_setup() after scanning all IOMMU devices and 
having num_iommus == 0 makes a decision to return -EAGAIN instead of 
-ENODEV as the deferred list is not empty.
And any errors except -ENODEV is treated by the upper layer as a reason 
to panic (see start_xen()). This is what you are facing.
With the attached patch Xen can continue to boot in such particular case 
with IOMMU disabled. You may want to give it a try.


>> And there is a hang and I don't know
>> if it is related to IOMMU or not...


I don’t think it is related to IOMMU, the IOMMU is neither used by Xen 
nor by Dom0 (we do not insert the IOMMU nodes into Dom0’s device tree).


>>
>> Any suggestions?


As you mentioned that "entire system hangs" I assume that Xen "hangs" as 
well (not only the Dom0), so the one thing which comes to mind is to 
re-check whether the "clk_ignore_unused" property is indeed passed via 
xen,dom0-bootargs to Linux.
Otherwise, the SCIF clock which supplies UART H/W used for Xen console 
will be disabled by Linux as unused, so the Xen console won't be 
functional afterwards and that may create the impression that system hangs.

Also you may want to look at the reference (which I wrote last year) and 
adopt it for the StarterKit:
https://wiki.xenproject.org/wiki/Xen_ARM_with_Virtualization_Extensions/Salvator-XS


>> Is rcar H3 board known to work well
>> with xen?


Yes. Everything needed for R-Car Gen3 is already in upstream Xen.
Note that using old SoC revision you won’t be able to get DMA devices 
working in non 1:1 mapped DomUs.


-- 
Regards,

Oleksandr Tyshchenko

Attachment: 0001-iommu-arm-Always-return-ENODEV-if-num_iommus-in-iomm.patch
Description: 0001-iommu-arm-Always-return-ENODEV-if-num_iommus-in-iomm.patch