Re: [Xen-devel] [PATCH] xen/arm64: Use __flush_dcache_area instead of __flush_dcache_all

[Ian Campbell <ian.campbell@xxxxxxxxxx>]

On Tue, 2014-10-14 at 12:23 +0100, Mark Rutland wrote:
> On Tue, Oct 14, 2014 at 11:39:37AM +0100, Ian Campbell wrote:
> > On Tue, 2014-10-14 at 11:32 +0100, Mark Rutland wrote:
> > > On Tue, Oct 14, 2014 at 10:35:23AM +0100, Ian Campbell wrote:
> > > > On Tue, 2014-10-14 at 10:21 +0100, Mark Rutland wrote:
> > > > > Hi Roy,
> > > > > 
> > > > > [...]
> > > > > 
> > > > > > It seems that for Xen we do need to flush the FDT as well - I get a
> > > > > > variety of crashes
> > > > > > with a corrupt FDT when cache state is modeled on the FVP model, and
> > > > > > Suravee sees similar
> > > > > > behavior on Seattle. I was not expecting this, as I looked at the 
> > > > > > code
> > > > > > in Xen and the caches/TLB
> > > > > > are enabled quite early on, before the FDT is accessed by Xen.  I 
> > > > > > then
> > > > > > looked at the mappings
> > > > > > used by  edk2 and Xen, and found some differences.  Even after
> > > > > > modifying edk2 to use the same
> > > > > > configuration as Xen, the flushing of the FDT is still required. Xen
> > > > > > and edk2 use the same memory
> > > > > > attributes  in the MAIR_EL2 register (0xFF), but had different
> > > > > > sharing, access perm, and nG settings.
> > > > > 
> > > > > I don't think the access perm or nG settings should have any effect, 
> > > > > but
> > > > > the shareability forms part of the memory attributes (along with the
> > > > > memory type and cacheability), and there are several rules that apply
> > > > > when accessing a memory location with mismatched attributes. See the
> > > > > ARMv8 ARM - The AArch64 Application Level Memory Model - Mismatched
> > > > > memory attributes.
> > > > > 
> > > > > In Linux we're likely getting lucky, and the shareability we use 
> > > > > varies
> > > > > for an SMP or UP kernel. So we need maintenance in at least one of 
> > > > > those
> > > > > cases. This would also apply to any initrd or other image.
> > > > > 
> > > > > Do you happen to know the shareability used by EDK2 and Xen?
> > > > 
> > > > Xen maps everything inner-shareable. Dunno about EDK2.
> > > 
> > > Ok. That matches what an SMP Linux kernel will do, so it looks like
> > > we're just getting lucky with Linux. I'lll have a play and see if I can
> > > trigger similar issues.
> > > 
> > > > Is the real issue here not a lack of specification for some corner cases
> > > > of the boot protocol? Can we get that fixed somehow?
> > > 
> > > To an extent, yes. We can try to fix up the Linux side with patche to
> > > Documentation/arm64/booting.txt. As far as I am aware, for UEFI that
> > > will require membership of the UEFI forum.
> > > 
> > 
> > Is Documentation/arm64/booting.txt relevant here since the kernel is
> > being launched as an EFI app, which already has a standardised calling
> > convention of its own. I suppose booting.txt is in addition to the UEFI
> > convention. It probably would be best to formalise that (what if a
> > second OS comes along with contradictory requirements?)
> 
> If we're trying to fix up UEFI, that needs to happen at the UEFI forum
> level. I believe there are some additional reqwuirements in SBSA/SBBR,
> but I haven't studied them in detail.
> 
> If there are requirements that Linux needs to have met regardless of
> UEFI, we should ensure we mention that in booting.txt.
> 
> It would be nice to have cross-OS agreement on boot protocols, but at
> the moment the table is somewhat empty beyond Linux and Xen. I had a
> conversation with the FreeBSD guys working on 64-bit ARM stuff, but
> they're still at an early stage, and I can't recall the specifics of
> their boot process.

I was thinking (perhaps naÃvely) that these problems would be mostly the
same for any OS and that the solution ought to be specified in terms
which allow any OS to know what to expect and/or what is expected of
them. Really OSes ought to be designing their boot protocols within the
set of constraints implied by the (improved) UEFI launching spec, not
vice versa.

> > > > Part of me wants to suggest that UEFI (and bootloaders generally) ought
> > > > to be cleaning caches for anything they have loaded into RAM before
> > > > launching an OS as a matter of good hygiene.
> > > 
> > > In general, yes.
> > > 
> > > Unfortunately, UEFI can't perform the maintenance in this case, because
> > > the stub modifies things. I was under the impression it copied and
> > > modified the FDT to embed the command line -- UEFI has no visibiltiy of
> > > this and therefore cannot be in charge of flushing it. So in this case,
> > > the stub needs to be thought of as the bootloader, and needs to be in
> > > charge of any required maintenance.
> > 
> > Right, that's what I was thinking. UEFI enters bootloader with
> > everything it has done all nice and clean and consistent. Anything the
> > stub then does it is responsible for maintaining the cleanliness.
> 
> There are two horrible parts here:
> 
>  * EFI has no idea what a boot loader is. As far as it's aware, the
>    kernel + efi stub is just another UEFI application until it calls
>    ExitBootServices. For all UEFI knows, it may as well be a calculator
>    until that point, and flushing the entire cache hierarchy for a
>    calculator seems a little extreme.

Most EFI applications are not that trivial though, and any non-trivial
app is going to (with some reasonably high probability) need to touch
the MMU. I don't see the problem with doing something which always works
even if it might be overkill for some small subset of things you might
be launching.

>  * Defining "nice and clean and consistent".
>   
>    As far as I am aware, UEFI may have an arbitrary set of mappings
>    present during boot services time, with arbitrary drivers active. 
>    That means that UEFI can create dirty cache entries concurrently with
>    the bootloader, in addition to the usual clean entries that can be
>    allocated at any time thanks to speculative fetches.
>    
>    So while we're in the bootloader, any system level caches can have
>    entries allocated to it, and as those aren't architected the only
>    thing we can do is flush those by VA for the portions we care about.
>    
> So we can have "initially consistent", but that might not be useful.

Hrm, yes, rather unfortunate.

> 
> > > There are a tonne of subtleties here, and certain properties we would
> > > like (e.g. a completely clean cache hierarchy upon entry to the OS)
> > > aren't necessarily possible to provide in general (thanks to the wonders
> > > of non-architected system level caches, interaction with bootloaders,
> > > etc).
> > 
> > I suppose it is easier for the UEFI implementation, since it knows the
> > platform it runs on and there knows about the caches. Harder for the
> > stub though :-/
> 
> Yeah. System-level caches interact badly with pretty much any scenario
> where ownership of the MMU is transferred (UEFI boot, kexec), and there
> doesn't seem to be a single agent that can be charged with ownership of
> maintenance.
> 
> This is something I've been meaning to revisit, but it takes a while to
> get back up to speed on the minutiae of the cache architecture and the
> rules for memory attributes, and I haven't had the time recently.
> 
> We do have a very heavy hammer that we know will work: flushing the
> memory by PA in the stub once the MMU and caches are disabled. A
> back-of-the-envelope calculation shows that could take minutes to issue
> on a server machine (say 2GHz, with 16GB of RAM), so that's very much a
> last resort.

Ouch...

> We could try to manage the system caches explicitly, but then we need
> code to do so very early, we need to have them described in the
> appropriate firmware tables, and they need to be manageable from the
> non-secure side (which I believe is not always the case). That somewhat
> defeat the portability aspect of booting as an EFI application.
> 
> So yes, it's harder for the stub :

Indeed.

Probably this isn't even close to the correct venue. I'm not sure where
better to transfer it though. One of the Linaro lists perhaps?

Ian.


_______________________________________________
Xen-devel mailing list
Xen-devel@xxxxxxxxxxxxx
http://lists.xen.org/xen-devel