Re: HYPERVISOR_console_io + CONSOLEIO_write needs COPY_flush_dcache

[Oleksandr Andrushchenko <Oleksandr_Andrushchenko@xxxxxxxx>]

On 6/10/20 3:50 PM, Julien Grall wrote:
> On 10/06/2020 09:13, Oleksandr Andrushchenko wrote:
>> Hello,
>
> Hi,
>
>> While adding support for para-virtualized block device in u-boot
>>
>> I faced an issue with HYPERVISOR_console_io + CONSOLEIO_write.
>>
>> I tried to use the hypercall during MMU setup stage while enabling data cache
>>
>> on arm64 platform (e.g. data cache is not yet enabled) and saw in guest's 
>> console
>>
>> either old data or some sort of garbage. Printing constant strings, just 
>> like mini-os
>>
>> does on boot, works as expected. 
>
> Per the hypercall ABI (see include/public/arch-arm.h) all the buffers must 
> reside in memory which is mapped as Normal Inner Write-Back Inner-Shareable.
>
> You are passing non-cacheable memory, so the behavior you see is expected.
This is what we come up with as well
>
>> So, looking at the Xen code [1] it seems
>>
>> that we should copy guest's buffer with COPY_flush_dcache set in this case
>
> COPY_flush_dcache is only meant to be used when copy to guest memory to make 
> sure the data reached the point of coherency in your system. It is not meant 
> to be used when copying from guest.
Understood and agree
>
>> as (usually?) this hypercall is used in guest's code which doesn't have any
>> other means to print yet, e.g. early printk case.
>
> I have been using it after the MMU is setup 
The thing is that in u-boot a lot of things happen before the MMU setup...
> but before the console is properly setup by the guest (there is a lot that 
> can happen in Linux between the two). In my case, the flush is not necessary 
> and would be wrong.
... and without the flush you lose the ability to debug that code.
>
> In general, the cache is never off on Arm64 because you may have system cache 
> or, in the cache of virtualization, cacheable mapping in the hypervisor (all 
> the memory is mapped).
>
> When updating data with MMU/D-cache off the normal approach is:
>    1) Remove any dirty lines from the cache, otherwise they may get evicted 
> while updating the memory and override any value written with MMU off.
>    2) Update the memory
>    3) Remove any lines that may have been speculated so when you turn onthe 
> MMU and D-cache, U-boot can obverse the correct data.
>
> Step 1 is only necessary if you are going to write outside of the loaded 
> Image (BSS is not part of it).
>
> You most likely already have similar steps in U-boot for other part of the 
> memory access with MMU/D-Cache off. So I think U-boot is the best place to 
> invalidate the cache before issuing the hypercall.

Yes, I have invalidated the buffers and everything works like a charm now, 
thanks

Probably my use-case should be somewhere documented, so another time someone 
steps into similar issues it is explained.

>
> Cheers,
>
Thank you for helping,

Oleksandr