Re: [RFC PATCH v2 00/15] xen/arm: port Linux LL/SC and LSE atomics helpers to Xen

[Ash Wilding <ash.j.wilding@xxxxxxxxx>]

Having pondered note (1) in my previous email a bit more, I imagine the reason
for using a DMB instead of acq/rel semantics is to prevent accesses following
the STLXR from being reordered between it and the LDAXR.

I won't be winning any awards for this ASCII art but hopefully it helps convey
the point.

Using just an LDAXR/STLXR pair and ditching the DMB, accesses to [D] and [E]
can be reodered between the LDAXR and STLXR:

                    ...
        +---------- LDR   [A]
        |           ...
        |           ...
        |    +----- STR   [B]
        |    |      ...
    ====|====|======LDAXR [C]================
        |    |      ...                  X
        |    +----> ...                  |
        |           ...                  |
        |           ...   <----------+   |
        X           ...              |   |
    ================STLXR [C]========|===|===
                    ...              |   |
                    ...              |   |
                    LDR   [D]--------+   |
                    ...                  |
                    STR   [E]------------+
                    ...


While dropping the acq semantics from the LDAXR and using a DMB instead will
prevent accesses to [D] and [E] being reordered between the LDXR/STLXR pair,
and keeping the rel semantics on the STLXR to prevents accesses to [A] and [B]
from being reordered after the STLXR:

                    ...
        +---------- LDR   [A]
        |           ...
        |           ...
        |    +----- STR   [B]
        |    |      ...
        |    |      LDXR  [C]
        |    |      ...
        |    +----> ...
        |           ...
        X           ...
    ================STLXR [C]================
    ================DMB======================
                    ...          X    X
                    ...          |    |
                    LDR   [D]----+    |  
                    ...               |
                    STR   [E]---------+
                    ...


As mentioned in my original email, the LSE atomic is a single instruction so
we can give it acq/rel semantics and not worry about any accesses to [A], [B],
[D], or [E] being reordered relative to that atomic:

                    ...
        +---------- LDR   [A]
        |           ...
        |           ...
        |    +----- STR   [B]
        |    |      ...
        X    X      ...
    ================LDADDAL [C]================
                    ...          X    X
                    ...          |    |
                    LDR   [D]----+    |  
                    ...               |
                    STR   [E]---------+
                    ...


So, makes sense that Linux uses acq/rel and no DMB for LSE, but Linux (and Xen)
are forced to use rel semantics and a DMB for the LL/SC case.

Anyway, point (2) from my earlier email is the one that's potentially more
concerning as we only have rel semantics and no DMB on the Linux LL/SC version
of atomic_cmpxchg(), in contrast to the existing Xen LL/SC implementation being
sandwiched between two DMBs and the Linux LSE version having acq/rel semantics.

Cheers,
Ash.