[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] Re: [Xen-devel] [PATCH 1/2] xen: arm: update arm64 assembly primitives to Linux v3.16-rc6
Hi Ian, On 07/25/2014 04:22 PM, Ian Campbell wrote: > The only really interesting changes here are the updates to mem* which update > to actually optimised versions and introduce an optimised memcmp. I didn't read the whole code as I assume it's just a copy with few changes from Linux. Acked-by: Julien Grall <julien.grall@xxxxxxxxxx> Regards, > bitops: No change to the bits we import. Record new baseline. > > cmpxchg: Import: > 60010e5 arm64: cmpxchg: update macros to prevent warnings > Author: Mark Hambleton <mahamble@xxxxxxxxxxxx> > Signed-off-by: Mark Hambleton <mahamble@xxxxxxxxxxxx> > Signed-off-by: Mark Brown <broonie@xxxxxxxxxx> > Signed-off-by: Catalin Marinas <catalin.marinas@xxxxxxx> > > e1dfda9 arm64: xchg: prevent warning if return value is unused > Author: Will Deacon <will.deacon@xxxxxxx> > Signed-off-by: Will Deacon <will.deacon@xxxxxxx> > Signed-off-by: Catalin Marinas <catalin.marinas@xxxxxxx> > > e1dfda9 resolves the warning which previous caused us to skip 60010e508111. > > Since arm32 and arm64 now differ (as do Linux arm and arm64) here the > existing definition in asm/system.h gets moved to asm/arm32/cmpxchg.h. > Previously this was shadowing the arm64 one but they happened to be > identical. > > atomics: Import: > 8715466 arch,arm64: Convert smp_mb__*() > Author: Peter Zijlstra <peterz@xxxxxxxxxxxxx> > Signed-off-by: Peter Zijlstra <peterz@xxxxxxxxxxxxx> > > This just drops some unused (by us) smp_mb__*_atomic_*. > > spinlocks: No change. Record new baseline. > > mem*: Import: > 808dbac arm64: lib: Implement optimized memcpy routine > Author: zhichang.yuan <zhichang.yuan@xxxxxxxxxx> > Signed-off-by: Zhichang Yuan <zhichang.yuan@xxxxxxxxxx> > Signed-off-by: Deepak Saxena <dsaxena@xxxxxxxxxx> > Signed-off-by: Catalin Marinas <catalin.marinas@xxxxxxx> > 280adc1 arm64: lib: Implement optimized memmove routine > Author: zhichang.yuan <zhichang.yuan@xxxxxxxxxx> > Signed-off-by: Zhichang Yuan <zhichang.yuan@xxxxxxxxxx> > Signed-off-by: Deepak Saxena <dsaxena@xxxxxxxxxx> > Signed-off-by: Catalin Marinas <catalin.marinas@xxxxxxx> > b29a51f arm64: lib: Implement optimized memset routine > Author: zhichang.yuan <zhichang.yuan@xxxxxxxxxx> > Signed-off-by: Zhichang Yuan <zhichang.yuan@xxxxxxxxxx> > Signed-off-by: Deepak Saxena <dsaxena@xxxxxxxxxx> > Signed-off-by: Catalin Marinas <catalin.marinas@xxxxxxx> > d875c9b arm64: lib: Implement optimized memcmp routine > Author: zhichang.yuan <zhichang.yuan@xxxxxxxxxx> > Signed-off-by: Zhichang Yuan <zhichang.yuan@xxxxxxxxxx> > Signed-off-by: Deepak Saxena <dsaxena@xxxxxxxxxx> > Signed-off-by: Catalin Marinas <catalin.marinas@xxxxxxx> > > These import various routines from Linaro's Cortex Strings library. > > Added assembler.h similar to on arm32 to define the various magic symbols > which these imported routines depend on (e.g. CPU_LE() and CPU_BE()) > > str*: No changes. Record new baseline. > > Correct the paths in the README. > > *_page: No changes. Record new baseline. > > README previous said clear_page was unused while clear page was, which was > backwards. > > Signed-off-by: Ian Campbell <ian.campbell@xxxxxxxxxx> > --- > xen/arch/arm/README.LinuxPrimitives | 36 +++-- > xen/arch/arm/arm64/lib/Makefile | 2 +- > xen/arch/arm/arm64/lib/assembler.h | 13 ++ > xen/arch/arm/arm64/lib/memchr.S | 1 + > xen/arch/arm/arm64/lib/memcmp.S | 258 > +++++++++++++++++++++++++++++++++++ > xen/arch/arm/arm64/lib/memcpy.S | 193 +++++++++++++++++++++++--- > xen/arch/arm/arm64/lib/memmove.S | 191 ++++++++++++++++++++++---- > xen/arch/arm/arm64/lib/memset.S | 208 +++++++++++++++++++++++++--- > xen/include/asm-arm/arm32/cmpxchg.h | 3 + > xen/include/asm-arm/arm64/atomic.h | 5 - > xen/include/asm-arm/arm64/cmpxchg.h | 35 +++-- > xen/include/asm-arm/string.h | 5 + > xen/include/asm-arm/system.h | 3 - > 13 files changed, 844 insertions(+), 109 deletions(-) > create mode 100644 xen/arch/arm/arm64/lib/assembler.h > create mode 100644 xen/arch/arm/arm64/lib/memcmp.S > > diff --git a/xen/arch/arm/README.LinuxPrimitives > b/xen/arch/arm/README.LinuxPrimitives > index 6cd03ca..69eeb70 100644 > --- a/xen/arch/arm/README.LinuxPrimitives > +++ b/xen/arch/arm/README.LinuxPrimitives > @@ -6,29 +6,26 @@ were last updated. > arm64: > ===================================================================== > > -bitops: last sync @ v3.14-rc7 (last commit: 8e86f0b) > +bitops: last sync @ v3.16-rc6 (last commit: 8715466b6027) > > linux/arch/arm64/lib/bitops.S xen/arch/arm/arm64/lib/bitops.S > linux/arch/arm64/include/asm/bitops.h xen/include/asm-arm/arm64/bitops.h > > --------------------------------------------------------------------- > > -cmpxchg: last sync @ v3.14-rc7 (last commit: 95c4189) > +cmpxchg: last sync @ v3.16-rc6 (last commit: e1dfda9ced9b) > > linux/arch/arm64/include/asm/cmpxchg.h xen/include/asm-arm/arm64/cmpxchg.h > > -Skipped: > - 60010e5 arm64: cmpxchg: update macros to prevent warnings > - > --------------------------------------------------------------------- > > -atomics: last sync @ v3.14-rc7 (last commit: 95c4189) > +atomics: last sync @ v3.16-rc6 (last commit: 8715466b6027) > > linux/arch/arm64/include/asm/atomic.h xen/include/asm-arm/arm64/atomic.h > > --------------------------------------------------------------------- > > -spinlocks: last sync @ v3.14-rc7 (last commit: 95c4189) > +spinlocks: last sync @ v3.16-rc6 (last commit: 95c4189689f9) > > linux/arch/arm64/include/asm/spinlock.h xen/include/asm-arm/arm64/spinlock.h > > @@ -38,30 +35,31 @@ Skipped: > > --------------------------------------------------------------------- > > -mem*: last sync @ v3.14-rc7 (last commit: 4a89922) > +mem*: last sync @ v3.16-rc6 (last commit: d875c9b37240) > > -linux/arch/arm64/lib/memchr.S xen/arch/arm/arm64/lib/memchr.S > -linux/arch/arm64/lib/memcpy.S xen/arch/arm/arm64/lib/memcpy.S > -linux/arch/arm64/lib/memmove.S xen/arch/arm/arm64/lib/memmove.S > -linux/arch/arm64/lib/memset.S xen/arch/arm/arm64/lib/memset.S > +linux/arch/arm64/lib/memchr.S xen/arch/arm/arm64/lib/memchr.S > +linux/arch/arm64/lib/memcmp.S xen/arch/arm/arm64/lib/memcmp.S > +linux/arch/arm64/lib/memcpy.S xen/arch/arm/arm64/lib/memcpy.S > +linux/arch/arm64/lib/memmove.S xen/arch/arm/arm64/lib/memmove.S > +linux/arch/arm64/lib/memset.S xen/arch/arm/arm64/lib/memset.S > > -for i in memchr.S memcpy.S memmove.S memset.S ; do > +for i in memchr.S memcmp.S memcpy.S memmove.S memset.S ; do > diff -u linux/arch/arm64/lib/$i xen/arch/arm/arm64/lib/$i > done > > --------------------------------------------------------------------- > > -str*: last sync @ v3.14-rc7 (last commit: 2b8cac8) > +str*: last sync @ v3.16-rc6 (last commit: 2b8cac814cd5) > > -linux/arch/arm/lib/strchr.S xen/arch/arm/arm64/lib/strchr.S > -linux/arch/arm/lib/strrchr.S xen/arch/arm/arm64/lib/strrchr.S > +linux/arch/arm64/lib/strchr.S xen/arch/arm/arm64/lib/strchr.S > +linux/arch/arm64/lib/strrchr.S xen/arch/arm/arm64/lib/strrchr.S > > --------------------------------------------------------------------- > > -{clear,copy}_page: last sync @ v3.14-rc7 (last commit: f27bb13) > +{clear,copy}_page: last sync @ v3.16-rc6 (last commit: f27bb139c387) > > -linux/arch/arm64/lib/clear_page.S unused in Xen > -linux/arch/arm64/lib/copy_page.S xen/arch/arm/arm64/lib/copy_page.S > +linux/arch/arm64/lib/clear_page.S xen/arch/arm/arm64/lib/clear_page.S > +linux/arch/arm64/lib/copy_page.S unused in Xen > > ===================================================================== > arm32 > diff --git a/xen/arch/arm/arm64/lib/Makefile b/xen/arch/arm/arm64/lib/Makefile > index b895afa..2e7fb64 100644 > --- a/xen/arch/arm/arm64/lib/Makefile > +++ b/xen/arch/arm/arm64/lib/Makefile > @@ -1,4 +1,4 @@ > -obj-y += memcpy.o memmove.o memset.o memchr.o > +obj-y += memcpy.o memcmp.o memmove.o memset.o memchr.o > obj-y += clear_page.o > obj-y += bitops.o find_next_bit.o > obj-y += strchr.o strrchr.o > diff --git a/xen/arch/arm/arm64/lib/assembler.h > b/xen/arch/arm/arm64/lib/assembler.h > new file mode 100644 > index 0000000..84669d1 > --- /dev/null > +++ b/xen/arch/arm/arm64/lib/assembler.h > @@ -0,0 +1,13 @@ > +#ifndef __ASM_ASSEMBLER_H__ > +#define __ASM_ASSEMBLER_H__ > + > +#ifndef __ASSEMBLY__ > +#error "Only include this from assembly code" > +#endif > + > +/* Only LE support so far */ > +#define CPU_BE(x...) > +#define CPU_LE(x...) x > + > +#endif /* __ASM_ASSEMBLER_H__ */ > + > diff --git a/xen/arch/arm/arm64/lib/memchr.S b/xen/arch/arm/arm64/lib/memchr.S > index 3cc1b01..b04590c 100644 > --- a/xen/arch/arm/arm64/lib/memchr.S > +++ b/xen/arch/arm/arm64/lib/memchr.S > @@ -18,6 +18,7 @@ > */ > > #include <xen/config.h> > +#include "assembler.h" > > /* > * Find a character in an area of memory. > diff --git a/xen/arch/arm/arm64/lib/memcmp.S b/xen/arch/arm/arm64/lib/memcmp.S > new file mode 100644 > index 0000000..9aad925 > --- /dev/null > +++ b/xen/arch/arm/arm64/lib/memcmp.S > @@ -0,0 +1,258 @@ > +/* > + * Copyright (C) 2013 ARM Ltd. > + * Copyright (C) 2013 Linaro. > + * > + * This code is based on glibc cortex strings work originally authored by > Linaro > + * and re-licensed under GPLv2 for the Linux kernel. The original code can > + * be found @ > + * > + * http://bazaar.launchpad.net/~linaro-toolchain-dev/cortex-strings/trunk/ > + * files/head:/src/aarch64/ > + * > + * This program is free software; you can redistribute it and/or modify > + * it under the terms of the GNU General Public License version 2 as > + * published by the Free Software Foundation. > + * > + * This program is distributed in the hope that it will be useful, > + * but WITHOUT ANY WARRANTY; without even the implied warranty of > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > + * GNU General Public License for more details. > + * > + * You should have received a copy of the GNU General Public License > + * along with this program. If not, see <http://www.gnu.org/licenses/>. > + */ > + > +#include <xen/config.h> > +#include "assembler.h" > + > +/* > +* compare memory areas(when two memory areas' offset are different, > +* alignment handled by the hardware) > +* > +* Parameters: > +* x0 - const memory area 1 pointer > +* x1 - const memory area 2 pointer > +* x2 - the maximal compare byte length > +* Returns: > +* x0 - a compare result, maybe less than, equal to, or greater than ZERO > +*/ > + > +/* Parameters and result. */ > +src1 .req x0 > +src2 .req x1 > +limit .req x2 > +result .req x0 > + > +/* Internal variables. */ > +data1 .req x3 > +data1w .req w3 > +data2 .req x4 > +data2w .req w4 > +has_nul .req x5 > +diff .req x6 > +endloop .req x7 > +tmp1 .req x8 > +tmp2 .req x9 > +tmp3 .req x10 > +pos .req x11 > +limit_wd .req x12 > +mask .req x13 > + > +ENTRY(memcmp) > + cbz limit, .Lret0 > + eor tmp1, src1, src2 > + tst tmp1, #7 > + b.ne .Lmisaligned8 > + ands tmp1, src1, #7 > + b.ne .Lmutual_align > + sub limit_wd, limit, #1 /* limit != 0, so no underflow. */ > + lsr limit_wd, limit_wd, #3 /* Convert to Dwords. */ > + /* > + * The input source addresses are at alignment boundary. > + * Directly compare eight bytes each time. > + */ > +.Lloop_aligned: > + ldr data1, [src1], #8 > + ldr data2, [src2], #8 > +.Lstart_realigned: > + subs limit_wd, limit_wd, #1 > + eor diff, data1, data2 /* Non-zero if differences found. */ > + csinv endloop, diff, xzr, cs /* Last Dword or differences. */ > + cbz endloop, .Lloop_aligned > + > + /* Not reached the limit, must have found a diff. */ > + tbz limit_wd, #63, .Lnot_limit > + > + /* Limit % 8 == 0 => the diff is in the last 8 bytes. */ > + ands limit, limit, #7 > + b.eq .Lnot_limit > + /* > + * The remained bytes less than 8. It is needed to extract valid data > + * from last eight bytes of the intended memory range. > + */ > + lsl limit, limit, #3 /* bytes-> bits. */ > + mov mask, #~0 > +CPU_BE( lsr mask, mask, limit ) > +CPU_LE( lsl mask, mask, limit ) > + bic data1, data1, mask > + bic data2, data2, mask > + > + orr diff, diff, mask > + b .Lnot_limit > + > +.Lmutual_align: > + /* > + * Sources are mutually aligned, but are not currently at an > + * alignment boundary. Round down the addresses and then mask off > + * the bytes that precede the start point. > + */ > + bic src1, src1, #7 > + bic src2, src2, #7 > + ldr data1, [src1], #8 > + ldr data2, [src2], #8 > + /* > + * We can not add limit with alignment offset(tmp1) here. Since the > + * addition probably make the limit overflown. > + */ > + sub limit_wd, limit, #1/*limit != 0, so no underflow.*/ > + and tmp3, limit_wd, #7 > + lsr limit_wd, limit_wd, #3 > + add tmp3, tmp3, tmp1 > + add limit_wd, limit_wd, tmp3, lsr #3 > + add limit, limit, tmp1/* Adjust the limit for the extra. */ > + > + lsl tmp1, tmp1, #3/* Bytes beyond alignment -> bits.*/ > + neg tmp1, tmp1/* Bits to alignment -64. */ > + mov tmp2, #~0 > + /*mask off the non-intended bytes before the start address.*/ > +CPU_BE( lsl tmp2, tmp2, tmp1 )/*Big-endian.Early bytes are at MSB*/ > + /* Little-endian. Early bytes are at LSB. */ > +CPU_LE( lsr tmp2, tmp2, tmp1 ) > + > + orr data1, data1, tmp2 > + orr data2, data2, tmp2 > + b .Lstart_realigned > + > + /*src1 and src2 have different alignment offset.*/ > +.Lmisaligned8: > + cmp limit, #8 > + b.lo .Ltiny8proc /*limit < 8: compare byte by byte*/ > + > + and tmp1, src1, #7 > + neg tmp1, tmp1 > + add tmp1, tmp1, #8/*valid length in the first 8 bytes of src1*/ > + and tmp2, src2, #7 > + neg tmp2, tmp2 > + add tmp2, tmp2, #8/*valid length in the first 8 bytes of src2*/ > + subs tmp3, tmp1, tmp2 > + csel pos, tmp1, tmp2, hi /*Choose the maximum.*/ > + > + sub limit, limit, pos > + /*compare the proceeding bytes in the first 8 byte segment.*/ > +.Ltinycmp: > + ldrb data1w, [src1], #1 > + ldrb data2w, [src2], #1 > + subs pos, pos, #1 > + ccmp data1w, data2w, #0, ne /* NZCV = 0b0000. */ > + b.eq .Ltinycmp > + cbnz pos, 1f /*diff occurred before the last byte.*/ > + cmp data1w, data2w > + b.eq .Lstart_align > +1: > + sub result, data1, data2 > + ret > + > +.Lstart_align: > + lsr limit_wd, limit, #3 > + cbz limit_wd, .Lremain8 > + > + ands xzr, src1, #7 > + b.eq .Lrecal_offset > + /*process more leading bytes to make src1 aligned...*/ > + add src1, src1, tmp3 /*backwards src1 to alignment boundary*/ > + add src2, src2, tmp3 > + sub limit, limit, tmp3 > + lsr limit_wd, limit, #3 > + cbz limit_wd, .Lremain8 > + /*load 8 bytes from aligned SRC1..*/ > + ldr data1, [src1], #8 > + ldr data2, [src2], #8 > + > + subs limit_wd, limit_wd, #1 > + eor diff, data1, data2 /*Non-zero if differences found.*/ > + csinv endloop, diff, xzr, ne > + cbnz endloop, .Lunequal_proc > + /*How far is the current SRC2 from the alignment boundary...*/ > + and tmp3, tmp3, #7 > + > +.Lrecal_offset:/*src1 is aligned now..*/ > + neg pos, tmp3 > +.Lloopcmp_proc: > + /* > + * Divide the eight bytes into two parts. First,backwards the src2 > + * to an alignment boundary,load eight bytes and compare from > + * the SRC2 alignment boundary. If all 8 bytes are equal,then start > + * the second part's comparison. Otherwise finish the comparison. > + * This special handle can garantee all the accesses are in the > + * thread/task space in avoid to overrange access. > + */ > + ldr data1, [src1,pos] > + ldr data2, [src2,pos] > + eor diff, data1, data2 /* Non-zero if differences found. */ > + cbnz diff, .Lnot_limit > + > + /*The second part process*/ > + ldr data1, [src1], #8 > + ldr data2, [src2], #8 > + eor diff, data1, data2 /* Non-zero if differences found. */ > + subs limit_wd, limit_wd, #1 > + csinv endloop, diff, xzr, ne/*if limit_wd is 0,will finish the cmp*/ > + cbz endloop, .Lloopcmp_proc > +.Lunequal_proc: > + cbz diff, .Lremain8 > + > +/*There is differnence occured in the latest comparison.*/ > +.Lnot_limit: > +/* > +* For little endian,reverse the low significant equal bits into MSB,then > +* following CLZ can find how many equal bits exist. > +*/ > +CPU_LE( rev diff, diff ) > +CPU_LE( rev data1, data1 ) > +CPU_LE( rev data2, data2 ) > + > + /* > + * The MS-non-zero bit of DIFF marks either the first bit > + * that is different, or the end of the significant data. > + * Shifting left now will bring the critical information into the > + * top bits. > + */ > + clz pos, diff > + lsl data1, data1, pos > + lsl data2, data2, pos > + /* > + * We need to zero-extend (char is unsigned) the value and then > + * perform a signed subtraction. > + */ > + lsr data1, data1, #56 > + sub result, data1, data2, lsr #56 > + ret > + > +.Lremain8: > + /* Limit % 8 == 0 =>. all data are equal.*/ > + ands limit, limit, #7 > + b.eq .Lret0 > + > +.Ltiny8proc: > + ldrb data1w, [src1], #1 > + ldrb data2w, [src2], #1 > + subs limit, limit, #1 > + > + ccmp data1w, data2w, #0, ne /* NZCV = 0b0000. */ > + b.eq .Ltiny8proc > + sub result, data1, data2 > + ret > +.Lret0: > + mov result, #0 > + ret > +ENDPROC(memcmp) > diff --git a/xen/arch/arm/arm64/lib/memcpy.S b/xen/arch/arm/arm64/lib/memcpy.S > index c8197c6..7cc885d 100644 > --- a/xen/arch/arm/arm64/lib/memcpy.S > +++ b/xen/arch/arm/arm64/lib/memcpy.S > @@ -1,5 +1,13 @@ > /* > * Copyright (C) 2013 ARM Ltd. > + * Copyright (C) 2013 Linaro. > + * > + * This code is based on glibc cortex strings work originally authored by > Linaro > + * and re-licensed under GPLv2 for the Linux kernel. The original code can > + * be found @ > + * > + * http://bazaar.launchpad.net/~linaro-toolchain-dev/cortex-strings/trunk/ > + * files/head:/src/aarch64/ > * > * This program is free software; you can redistribute it and/or modify > * it under the terms of the GNU General Public License version 2 as > @@ -15,6 +23,8 @@ > */ > > #include <xen/config.h> > +#include <asm/cache.h> > +#include "assembler.h" > > /* > * Copy a buffer from src to dest (alignment handled by the hardware) > @@ -26,27 +36,166 @@ > * Returns: > * x0 - dest > */ > +dstin .req x0 > +src .req x1 > +count .req x2 > +tmp1 .req x3 > +tmp1w .req w3 > +tmp2 .req x4 > +tmp2w .req w4 > +tmp3 .req x5 > +tmp3w .req w5 > +dst .req x6 > + > +A_l .req x7 > +A_h .req x8 > +B_l .req x9 > +B_h .req x10 > +C_l .req x11 > +C_h .req x12 > +D_l .req x13 > +D_h .req x14 > + > ENTRY(memcpy) > - mov x4, x0 > - subs x2, x2, #8 > - b.mi 2f > -1: ldr x3, [x1], #8 > - subs x2, x2, #8 > - str x3, [x4], #8 > - b.pl 1b > -2: adds x2, x2, #4 > - b.mi 3f > - ldr w3, [x1], #4 > - sub x2, x2, #4 > - str w3, [x4], #4 > -3: adds x2, x2, #2 > - b.mi 4f > - ldrh w3, [x1], #2 > - sub x2, x2, #2 > - strh w3, [x4], #2 > -4: adds x2, x2, #1 > - b.mi 5f > - ldrb w3, [x1] > - strb w3, [x4] > -5: ret > + mov dst, dstin > + cmp count, #16 > + /*When memory length is less than 16, the accessed are not aligned.*/ > + b.lo .Ltiny15 > + > + neg tmp2, src > + ands tmp2, tmp2, #15/* Bytes to reach alignment. */ > + b.eq .LSrcAligned > + sub count, count, tmp2 > + /* > + * Copy the leading memory data from src to dst in an increasing > + * address order.By this way,the risk of overwritting the source > + * memory data is eliminated when the distance between src and > + * dst is less than 16. The memory accesses here are alignment. > + */ > + tbz tmp2, #0, 1f > + ldrb tmp1w, [src], #1 > + strb tmp1w, [dst], #1 > +1: > + tbz tmp2, #1, 2f > + ldrh tmp1w, [src], #2 > + strh tmp1w, [dst], #2 > +2: > + tbz tmp2, #2, 3f > + ldr tmp1w, [src], #4 > + str tmp1w, [dst], #4 > +3: > + tbz tmp2, #3, .LSrcAligned > + ldr tmp1, [src],#8 > + str tmp1, [dst],#8 > + > +.LSrcAligned: > + cmp count, #64 > + b.ge .Lcpy_over64 > + /* > + * Deal with small copies quickly by dropping straight into the > + * exit block. > + */ > +.Ltail63: > + /* > + * Copy up to 48 bytes of data. At this point we only need the > + * bottom 6 bits of count to be accurate. > + */ > + ands tmp1, count, #0x30 > + b.eq .Ltiny15 > + cmp tmp1w, #0x20 > + b.eq 1f > + b.lt 2f > + ldp A_l, A_h, [src], #16 > + stp A_l, A_h, [dst], #16 > +1: > + ldp A_l, A_h, [src], #16 > + stp A_l, A_h, [dst], #16 > +2: > + ldp A_l, A_h, [src], #16 > + stp A_l, A_h, [dst], #16 > +.Ltiny15: > + /* > + * Prefer to break one ldp/stp into several load/store to access > + * memory in an increasing address order,rather than to load/store 16 > + * bytes from (src-16) to (dst-16) and to backward the src to aligned > + * address,which way is used in original cortex memcpy. If keeping > + * the original memcpy process here, memmove need to satisfy the > + * precondition that src address is at least 16 bytes bigger than dst > + * address,otherwise some source data will be overwritten when memove > + * call memcpy directly. To make memmove simpler and decouple the > + * memcpy's dependency on memmove, withdrew the original process. > + */ > + tbz count, #3, 1f > + ldr tmp1, [src], #8 > + str tmp1, [dst], #8 > +1: > + tbz count, #2, 2f > + ldr tmp1w, [src], #4 > + str tmp1w, [dst], #4 > +2: > + tbz count, #1, 3f > + ldrh tmp1w, [src], #2 > + strh tmp1w, [dst], #2 > +3: > + tbz count, #0, .Lexitfunc > + ldrb tmp1w, [src] > + strb tmp1w, [dst] > + > +.Lexitfunc: > + ret > + > +.Lcpy_over64: > + subs count, count, #128 > + b.ge .Lcpy_body_large > + /* > + * Less than 128 bytes to copy, so handle 64 here and then jump > + * to the tail. > + */ > + ldp A_l, A_h, [src],#16 > + stp A_l, A_h, [dst],#16 > + ldp B_l, B_h, [src],#16 > + ldp C_l, C_h, [src],#16 > + stp B_l, B_h, [dst],#16 > + stp C_l, C_h, [dst],#16 > + ldp D_l, D_h, [src],#16 > + stp D_l, D_h, [dst],#16 > + > + tst count, #0x3f > + b.ne .Ltail63 > + ret > + > + /* > + * Critical loop. Start at a new cache line boundary. Assuming > + * 64 bytes per line this ensures the entire loop is in one line. > + */ > + .p2align L1_CACHE_SHIFT > +.Lcpy_body_large: > + /* pre-get 64 bytes data. */ > + ldp A_l, A_h, [src],#16 > + ldp B_l, B_h, [src],#16 > + ldp C_l, C_h, [src],#16 > + ldp D_l, D_h, [src],#16 > +1: > + /* > + * interlace the load of next 64 bytes data block with store of the last > + * loaded 64 bytes data. > + */ > + stp A_l, A_h, [dst],#16 > + ldp A_l, A_h, [src],#16 > + stp B_l, B_h, [dst],#16 > + ldp B_l, B_h, [src],#16 > + stp C_l, C_h, [dst],#16 > + ldp C_l, C_h, [src],#16 > + stp D_l, D_h, [dst],#16 > + ldp D_l, D_h, [src],#16 > + subs count, count, #64 > + b.ge 1b > + stp A_l, A_h, [dst],#16 > + stp B_l, B_h, [dst],#16 > + stp C_l, C_h, [dst],#16 > + stp D_l, D_h, [dst],#16 > + > + tst count, #0x3f > + b.ne .Ltail63 > + ret > ENDPROC(memcpy) > diff --git a/xen/arch/arm/arm64/lib/memmove.S > b/xen/arch/arm/arm64/lib/memmove.S > index 1bf0936..f4065b9 100644 > --- a/xen/arch/arm/arm64/lib/memmove.S > +++ b/xen/arch/arm/arm64/lib/memmove.S > @@ -1,5 +1,13 @@ > /* > * Copyright (C) 2013 ARM Ltd. > + * Copyright (C) 2013 Linaro. > + * > + * This code is based on glibc cortex strings work originally authored by > Linaro > + * and re-licensed under GPLv2 for the Linux kernel. The original code can > + * be found @ > + * > + * http://bazaar.launchpad.net/~linaro-toolchain-dev/cortex-strings/trunk/ > + * files/head:/src/aarch64/ > * > * This program is free software; you can redistribute it and/or modify > * it under the terms of the GNU General Public License version 2 as > @@ -15,6 +23,8 @@ > */ > > #include <xen/config.h> > +#include <asm/cache.h> > +#include "assembler.h" > > /* > * Move a buffer from src to test (alignment handled by the hardware). > @@ -27,30 +37,161 @@ > * Returns: > * x0 - dest > */ > +dstin .req x0 > +src .req x1 > +count .req x2 > +tmp1 .req x3 > +tmp1w .req w3 > +tmp2 .req x4 > +tmp2w .req w4 > +tmp3 .req x5 > +tmp3w .req w5 > +dst .req x6 > + > +A_l .req x7 > +A_h .req x8 > +B_l .req x9 > +B_h .req x10 > +C_l .req x11 > +C_h .req x12 > +D_l .req x13 > +D_h .req x14 > + > ENTRY(memmove) > - cmp x0, x1 > - b.ls memcpy > - add x4, x0, x2 > - add x1, x1, x2 > - subs x2, x2, #8 > - b.mi 2f > -1: ldr x3, [x1, #-8]! > - subs x2, x2, #8 > - str x3, [x4, #-8]! > - b.pl 1b > -2: adds x2, x2, #4 > - b.mi 3f > - ldr w3, [x1, #-4]! > - sub x2, x2, #4 > - str w3, [x4, #-4]! > -3: adds x2, x2, #2 > - b.mi 4f > - ldrh w3, [x1, #-2]! > - sub x2, x2, #2 > - strh w3, [x4, #-2]! > -4: adds x2, x2, #1 > - b.mi 5f > - ldrb w3, [x1, #-1] > - strb w3, [x4, #-1] > -5: ret > + cmp dstin, src > + b.lo memcpy > + add tmp1, src, count > + cmp dstin, tmp1 > + b.hs memcpy /* No overlap. */ > + > + add dst, dstin, count > + add src, src, count > + cmp count, #16 > + b.lo .Ltail15 /*probably non-alignment accesses.*/ > + > + ands tmp2, src, #15 /* Bytes to reach alignment. */ > + b.eq .LSrcAligned > + sub count, count, tmp2 > + /* > + * process the aligned offset length to make the src aligned firstly. > + * those extra instructions' cost is acceptable. It also make the > + * coming accesses are based on aligned address. > + */ > + tbz tmp2, #0, 1f > + ldrb tmp1w, [src, #-1]! > + strb tmp1w, [dst, #-1]! > +1: > + tbz tmp2, #1, 2f > + ldrh tmp1w, [src, #-2]! > + strh tmp1w, [dst, #-2]! > +2: > + tbz tmp2, #2, 3f > + ldr tmp1w, [src, #-4]! > + str tmp1w, [dst, #-4]! > +3: > + tbz tmp2, #3, .LSrcAligned > + ldr tmp1, [src, #-8]! > + str tmp1, [dst, #-8]! > + > +.LSrcAligned: > + cmp count, #64 > + b.ge .Lcpy_over64 > + > + /* > + * Deal with small copies quickly by dropping straight into the > + * exit block. > + */ > +.Ltail63: > + /* > + * Copy up to 48 bytes of data. At this point we only need the > + * bottom 6 bits of count to be accurate. > + */ > + ands tmp1, count, #0x30 > + b.eq .Ltail15 > + cmp tmp1w, #0x20 > + b.eq 1f > + b.lt 2f > + ldp A_l, A_h, [src, #-16]! > + stp A_l, A_h, [dst, #-16]! > +1: > + ldp A_l, A_h, [src, #-16]! > + stp A_l, A_h, [dst, #-16]! > +2: > + ldp A_l, A_h, [src, #-16]! > + stp A_l, A_h, [dst, #-16]! > + > +.Ltail15: > + tbz count, #3, 1f > + ldr tmp1, [src, #-8]! > + str tmp1, [dst, #-8]! > +1: > + tbz count, #2, 2f > + ldr tmp1w, [src, #-4]! > + str tmp1w, [dst, #-4]! > +2: > + tbz count, #1, 3f > + ldrh tmp1w, [src, #-2]! > + strh tmp1w, [dst, #-2]! > +3: > + tbz count, #0, .Lexitfunc > + ldrb tmp1w, [src, #-1] > + strb tmp1w, [dst, #-1] > + > +.Lexitfunc: > + ret > + > +.Lcpy_over64: > + subs count, count, #128 > + b.ge .Lcpy_body_large > + /* > + * Less than 128 bytes to copy, so handle 64 bytes here and then jump > + * to the tail. > + */ > + ldp A_l, A_h, [src, #-16] > + stp A_l, A_h, [dst, #-16] > + ldp B_l, B_h, [src, #-32] > + ldp C_l, C_h, [src, #-48] > + stp B_l, B_h, [dst, #-32] > + stp C_l, C_h, [dst, #-48] > + ldp D_l, D_h, [src, #-64]! > + stp D_l, D_h, [dst, #-64]! > + > + tst count, #0x3f > + b.ne .Ltail63 > + ret > + > + /* > + * Critical loop. Start at a new cache line boundary. Assuming > + * 64 bytes per line this ensures the entire loop is in one line. > + */ > + .p2align L1_CACHE_SHIFT > +.Lcpy_body_large: > + /* pre-load 64 bytes data. */ > + ldp A_l, A_h, [src, #-16] > + ldp B_l, B_h, [src, #-32] > + ldp C_l, C_h, [src, #-48] > + ldp D_l, D_h, [src, #-64]! > +1: > + /* > + * interlace the load of next 64 bytes data block with store of the last > + * loaded 64 bytes data. > + */ > + stp A_l, A_h, [dst, #-16] > + ldp A_l, A_h, [src, #-16] > + stp B_l, B_h, [dst, #-32] > + ldp B_l, B_h, [src, #-32] > + stp C_l, C_h, [dst, #-48] > + ldp C_l, C_h, [src, #-48] > + stp D_l, D_h, [dst, #-64]! > + ldp D_l, D_h, [src, #-64]! > + subs count, count, #64 > + b.ge 1b > + stp A_l, A_h, [dst, #-16] > + stp B_l, B_h, [dst, #-32] > + stp C_l, C_h, [dst, #-48] > + stp D_l, D_h, [dst, #-64]! > + > + tst count, #0x3f > + b.ne .Ltail63 > + ret > ENDPROC(memmove) > diff --git a/xen/arch/arm/arm64/lib/memset.S b/xen/arch/arm/arm64/lib/memset.S > index 25a4fb6..4ee714d 100644 > --- a/xen/arch/arm/arm64/lib/memset.S > +++ b/xen/arch/arm/arm64/lib/memset.S > @@ -1,5 +1,13 @@ > /* > * Copyright (C) 2013 ARM Ltd. > + * Copyright (C) 2013 Linaro. > + * > + * This code is based on glibc cortex strings work originally authored by > Linaro > + * and re-licensed under GPLv2 for the Linux kernel. The original code can > + * be found @ > + * > + * http://bazaar.launchpad.net/~linaro-toolchain-dev/cortex-strings/trunk/ > + * files/head:/src/aarch64/ > * > * This program is free software; you can redistribute it and/or modify > * it under the terms of the GNU General Public License version 2 as > @@ -15,6 +23,8 @@ > */ > > #include <xen/config.h> > +#include <asm/cache.h> > +#include "assembler.h" > > /* > * Fill in the buffer with character c (alignment handled by the hardware) > @@ -26,27 +36,181 @@ > * Returns: > * x0 - buf > */ > + > +dstin .req x0 > +val .req w1 > +count .req x2 > +tmp1 .req x3 > +tmp1w .req w3 > +tmp2 .req x4 > +tmp2w .req w4 > +zva_len_x .req x5 > +zva_len .req w5 > +zva_bits_x .req x6 > + > +A_l .req x7 > +A_lw .req w7 > +dst .req x8 > +tmp3w .req w9 > +tmp3 .req x9 > + > ENTRY(memset) > - mov x4, x0 > - and w1, w1, #0xff > - orr w1, w1, w1, lsl #8 > - orr w1, w1, w1, lsl #16 > - orr x1, x1, x1, lsl #32 > - subs x2, x2, #8 > - b.mi 2f > -1: str x1, [x4], #8 > - subs x2, x2, #8 > - b.pl 1b > -2: adds x2, x2, #4 > - b.mi 3f > - sub x2, x2, #4 > - str w1, [x4], #4 > -3: adds x2, x2, #2 > - b.mi 4f > - sub x2, x2, #2 > - strh w1, [x4], #2 > -4: adds x2, x2, #1 > - b.mi 5f > - strb w1, [x4] > -5: ret > + mov dst, dstin /* Preserve return value. */ > + and A_lw, val, #255 > + orr A_lw, A_lw, A_lw, lsl #8 > + orr A_lw, A_lw, A_lw, lsl #16 > + orr A_l, A_l, A_l, lsl #32 > + > + cmp count, #15 > + b.hi .Lover16_proc > + /*All store maybe are non-aligned..*/ > + tbz count, #3, 1f > + str A_l, [dst], #8 > +1: > + tbz count, #2, 2f > + str A_lw, [dst], #4 > +2: > + tbz count, #1, 3f > + strh A_lw, [dst], #2 > +3: > + tbz count, #0, 4f > + strb A_lw, [dst] > +4: > + ret > + > +.Lover16_proc: > + /*Whether the start address is aligned with 16.*/ > + neg tmp2, dst > + ands tmp2, tmp2, #15 > + b.eq .Laligned > +/* > +* The count is not less than 16, we can use stp to store the start 16 bytes, > +* then adjust the dst aligned with 16.This process will make the current > +* memory address at alignment boundary. > +*/ > + stp A_l, A_l, [dst] /*non-aligned store..*/ > + /*make the dst aligned..*/ > + sub count, count, tmp2 > + add dst, dst, tmp2 > + > +.Laligned: > + cbz A_l, .Lzero_mem > + > +.Ltail_maybe_long: > + cmp count, #64 > + b.ge .Lnot_short > +.Ltail63: > + ands tmp1, count, #0x30 > + b.eq 3f > + cmp tmp1w, #0x20 > + b.eq 1f > + b.lt 2f > + stp A_l, A_l, [dst], #16 > +1: > + stp A_l, A_l, [dst], #16 > +2: > + stp A_l, A_l, [dst], #16 > +/* > +* The last store length is less than 16,use stp to write last 16 bytes. > +* It will lead some bytes written twice and the access is non-aligned. > +*/ > +3: > + ands count, count, #15 > + cbz count, 4f > + add dst, dst, count > + stp A_l, A_l, [dst, #-16] /* Repeat some/all of last store. */ > +4: > + ret > + > + /* > + * Critical loop. Start at a new cache line boundary. Assuming > + * 64 bytes per line, this ensures the entire loop is in one line. > + */ > + .p2align L1_CACHE_SHIFT > +.Lnot_short: > + sub dst, dst, #16/* Pre-bias. */ > + sub count, count, #64 > +1: > + stp A_l, A_l, [dst, #16] > + stp A_l, A_l, [dst, #32] > + stp A_l, A_l, [dst, #48] > + stp A_l, A_l, [dst, #64]! > + subs count, count, #64 > + b.ge 1b > + tst count, #0x3f > + add dst, dst, #16 > + b.ne .Ltail63 > +.Lexitfunc: > + ret > + > + /* > + * For zeroing memory, check to see if we can use the ZVA feature to > + * zero entire 'cache' lines. > + */ > +.Lzero_mem: > + cmp count, #63 > + b.le .Ltail63 > + /* > + * For zeroing small amounts of memory, it's not worth setting up > + * the line-clear code. > + */ > + cmp count, #128 > + b.lt .Lnot_short /*count is at least 128 bytes*/ > + > + mrs tmp1, dczid_el0 > + tbnz tmp1, #4, .Lnot_short > + mov tmp3w, #4 > + and zva_len, tmp1w, #15 /* Safety: other bits reserved. */ > + lsl zva_len, tmp3w, zva_len > + > + ands tmp3w, zva_len, #63 > + /* > + * ensure the zva_len is not less than 64. > + * It is not meaningful to use ZVA if the block size is less than 64. > + */ > + b.ne .Lnot_short > +.Lzero_by_line: > + /* > + * Compute how far we need to go to become suitably aligned. We're > + * already at quad-word alignment. > + */ > + cmp count, zva_len_x > + b.lt .Lnot_short /* Not enough to reach alignment. */ > + sub zva_bits_x, zva_len_x, #1 > + neg tmp2, dst > + ands tmp2, tmp2, zva_bits_x > + b.eq 2f /* Already aligned. */ > + /* Not aligned, check that there's enough to copy after alignment.*/ > + sub tmp1, count, tmp2 > + /* > + * grantee the remain length to be ZVA is bigger than 64, > + * avoid to make the 2f's process over mem range.*/ > + cmp tmp1, #64 > + ccmp tmp1, zva_len_x, #8, ge /* NZCV=0b1000 */ > + b.lt .Lnot_short > + /* > + * We know that there's at least 64 bytes to zero and that it's safe > + * to overrun by 64 bytes. > + */ > + mov count, tmp1 > +1: > + stp A_l, A_l, [dst] > + stp A_l, A_l, [dst, #16] > + stp A_l, A_l, [dst, #32] > + subs tmp2, tmp2, #64 > + stp A_l, A_l, [dst, #48] > + add dst, dst, #64 > + b.ge 1b > + /* We've overrun a bit, so adjust dst downwards.*/ > + add dst, dst, tmp2 > +2: > + sub count, count, zva_len_x > +3: > + dc zva, dst > + add dst, dst, zva_len_x > + subs count, count, zva_len_x > + b.ge 3b > + ands count, count, zva_bits_x > + b.ne .Ltail_maybe_long > + ret > ENDPROC(memset) > diff --git a/xen/include/asm-arm/arm32/cmpxchg.h > b/xen/include/asm-arm/arm32/cmpxchg.h > index 3f4e7a1..9a511f2 100644 > --- a/xen/include/asm-arm/arm32/cmpxchg.h > +++ b/xen/include/asm-arm/arm32/cmpxchg.h > @@ -40,6 +40,9 @@ static inline unsigned long __xchg(unsigned long x, > volatile void *ptr, int size > return ret; > } > > +#define xchg(ptr,x) \ > + ((__typeof__(*(ptr)))__xchg((unsigned long)(x),(ptr),sizeof(*(ptr)))) > + > /* > * Atomic compare and exchange. Compare OLD with MEM, if identical, > * store NEW in MEM. Return the initial value in MEM. Success is > diff --git a/xen/include/asm-arm/arm64/atomic.h > b/xen/include/asm-arm/arm64/atomic.h > index b5d50f2..b49219e 100644 > --- a/xen/include/asm-arm/arm64/atomic.h > +++ b/xen/include/asm-arm/arm64/atomic.h > @@ -136,11 +136,6 @@ static inline int __atomic_add_unless(atomic_t *v, int > a, int u) > > #define atomic_add_negative(i,v) (atomic_add_return(i, v) < 0) > > -#define smp_mb__before_atomic_dec() smp_mb() > -#define smp_mb__after_atomic_dec() smp_mb() > -#define smp_mb__before_atomic_inc() smp_mb() > -#define smp_mb__after_atomic_inc() smp_mb() > - > #endif > /* > * Local variables: > diff --git a/xen/include/asm-arm/arm64/cmpxchg.h > b/xen/include/asm-arm/arm64/cmpxchg.h > index 4e930ce..ae42b2f 100644 > --- a/xen/include/asm-arm/arm64/cmpxchg.h > +++ b/xen/include/asm-arm/arm64/cmpxchg.h > @@ -54,7 +54,12 @@ static inline unsigned long __xchg(unsigned long x, > volatile void *ptr, int size > } > > #define xchg(ptr,x) \ > - ((__typeof__(*(ptr)))__xchg((unsigned long)(x),(ptr),sizeof(*(ptr)))) > +({ \ > + __typeof__(*(ptr)) __ret; \ > + __ret = (__typeof__(*(ptr))) \ > + __xchg((unsigned long)(x), (ptr), sizeof(*(ptr))); \ > + __ret; \ > +}) > > extern void __bad_cmpxchg(volatile void *ptr, int size); > > @@ -144,17 +149,23 @@ static inline unsigned long __cmpxchg_mb(volatile void > *ptr, unsigned long old, > return ret; > } > > -#define cmpxchg(ptr,o,n) \ > - ((__typeof__(*(ptr)))__cmpxchg_mb((ptr), \ > - (unsigned long)(o), \ > - (unsigned long)(n), \ > - sizeof(*(ptr)))) > - > -#define cmpxchg_local(ptr,o,n) > \ > - ((__typeof__(*(ptr)))__cmpxchg((ptr), \ > - (unsigned long)(o), \ > - (unsigned long)(n), \ > - sizeof(*(ptr)))) > +#define cmpxchg(ptr, o, n) \ > +({ \ > + __typeof__(*(ptr)) __ret; \ > + __ret = (__typeof__(*(ptr))) \ > + __cmpxchg_mb((ptr), (unsigned long)(o), (unsigned long)(n), \ > + sizeof(*(ptr))); \ > + __ret; \ > +}) > + > +#define cmpxchg_local(ptr, o, n) \ > +({ \ > + __typeof__(*(ptr)) __ret; \ > + __ret = (__typeof__(*(ptr))) \ > + __cmpxchg((ptr), (unsigned long)(o), \ > + (unsigned long)(n), sizeof(*(ptr))); \ > + __ret; \ > +}) > > #endif > /* > diff --git a/xen/include/asm-arm/string.h b/xen/include/asm-arm/string.h > index 3242762..dfad1fe 100644 > --- a/xen/include/asm-arm/string.h > +++ b/xen/include/asm-arm/string.h > @@ -17,6 +17,11 @@ extern char * strchr(const char * s, int c); > #define __HAVE_ARCH_MEMCPY > extern void * memcpy(void *, const void *, __kernel_size_t); > > +#if defined(CONFIG_ARM_64) > +#define __HAVE_ARCH_MEMCMP > +extern int memcmp(const void *, const void *, __kernel_size_t); > +#endif > + > /* Some versions of gcc don't have this builtin. It's non-critical anyway. */ > #define __HAVE_ARCH_MEMMOVE > extern void *memmove(void *dest, const void *src, size_t n); > diff --git a/xen/include/asm-arm/system.h b/xen/include/asm-arm/system.h > index 7aaaf50..ce3d38a 100644 > --- a/xen/include/asm-arm/system.h > +++ b/xen/include/asm-arm/system.h > @@ -33,9 +33,6 @@ > > #define smp_wmb() dmb(ishst) > > -#define xchg(ptr,x) \ > - ((__typeof__(*(ptr)))__xchg((unsigned long)(x),(ptr),sizeof(*(ptr)))) > - > /* > * This is used to ensure the compiler did actually allocate the register we > * asked it for some inline assembly sequences. Apparently we can't trust > -- Julien Grall _______________________________________________ Xen-devel mailing list Xen-devel@xxxxxxxxxxxxx http://lists.xen.org/xen-devel
|
Lists.xenproject.org is hosted with RackSpace, monitoring our |