Re: HYPERVISOR_console_io + CONSOLEIO_write needs COPY_flush_dcache

[Julien Grall <julien@xxxxxxx>]

On 10/06/2020 09:13, Oleksandr Andrushchenko wrote:
> Hello,
Hi,

> While adding support for para-virtualized block device in u-boot
>
> I faced an issue with HYPERVISOR_console_io + CONSOLEIO_write.
>
> I tried to use the hypercall during MMU setup stage while enabling data cache
>
> on arm64 platform (e.g. data cache is not yet enabled) and saw in guest's 
> console
>
> either old data or some sort of garbage. Printing constant strings, just like 
> mini-os
>
> does on boot, works as expected. 
Per the hypercall ABI (see include/public/arch-arm.h) all the buffers 
must reside in memory which is mapped as Normal Inner Write-Back 
Inner-Shareable.
You are passing non-cacheable memory, so the behavior you see is expected.

> So, looking at the Xen code [1] it seems
>
> that we should copy guest's buffer with COPY_flush_dcache set in this case
COPY_flush_dcache is only meant to be used when copy to guest memory to 
make sure the data reached the point of coherency in your system. It is 
not meant to be used when copying from guest.
> as (usually?) this hypercall is used in guest's code which doesn't have any
> other means to print yet, e.g. early printk case.
I have been using it after the MMU is setup but before the console is 
properly setup by the guest (there is a lot that can happen in Linux 
between the two). In my case, the flush is not necessary and would be wrong.
In general, the cache is never off on Arm64 because you may have system 
cache or, in the cache of virtualization, cacheable mapping in the 
hypervisor (all the memory is mapped).
When updating data with MMU/D-cache off the normal approach is:
   1) Remove any dirty lines from the cache, otherwise they may get 
evicted while updating the memory and override any value written with 
MMU off.
   2) Update the memory
   3) Remove any lines that may have been speculated so when you turn 
onthe MMU and D-cache, U-boot can obverse the correct data.
Step 1 is only necessary if you are going to write outside of the loaded 
Image (BSS is not part of it).
You most likely already have similar steps in U-boot for other part of 
the memory access with MMU/D-Cache off. So I think U-boot is the best 
place to invalidate the cache before issuing the hypercall.
Cheers,

--
Julien Grall,