linux/arch/powerpc/lib
Mark Nelson 25d6e2d7c5 powerpc: Update 64bit memcpy() using CPU_FTR_UNALIGNED_LD_STD
Update memcpy() to add two new feature sections: one for aligning the
destination before copying and one for copying using aligned load
and store doubles.

These new feature sections will only affect Power6 and Cell because
the CPU feature bit was only added to these two processors.

Power6 gets its best performance in memcpy() when aligning neither the
source nor the destination, while Cell gets its best performance when
just the destination is aligned. But in order to save on CPU feature
bits we can use the previously added CPU_FTR_CP_USE_DCBTZ feature bit
to differentiate between Power6 and Cell (because CPU_FTR_CP_USE_DCBTZ
was added to Cell but not Power6).

The first feature section acts to nop out the branch that takes us to
the code that aligns us to an eight byte boundary for the destination.
We only want to nop out this branch on Power6.

So the ALT_FTR_SECTION_END() for this feature section creates a test
mask of the two feature bits ORed together and provides an expected
result of just CPU_FTR_UNALIGNED_LD_STD, thus we nop out the branch
if we're on a CPU that has CPU_FTR_UNALIGNED_LD_STD set and
CPU_FTR_CP_USE_DCBTZ unset.

For the second feature section added, if we're on a CPU that has the
CPU_FTR_UNALIGNED_LD_STD bit set then we don't want to do the copy
with aligned loads and stores (and the appropriate shifting left and
right instructions), so we want to nop out the branch to
.Lsrc_unaligned.

The andi. used for this branch is moved to just above the branch
because this allows us to nop out both instructions with just one
feature section which gives us better performance and doesn't hurt
readability which two separate feature sections did.

Moving the andi. to just above the branch doesn't have any noticeable
negative effect on the remaining 64bit processors (the ones that
didn't have this feature bit added).

On Cell this simple modification results in an improvement to measured
memcpy() bandwidth of up to 50% in the hot cache case and up to 15% in
the cold cache case.

On Power6 we get memory bandwidth results that are up to three times
faster in the hot cache case and up to 50% faster in the cold cache
case.

Commit 2a9294369b ("powerpc: Add new CPU
feature: CPU_FTR_CP_USE_DCBTZ") was where CPU_FTR_CP_USE_DCBTZ was
added.

To say that Cell gets its best performance in memcpy() with just the
destination aligned is true but only for the reason that the indirect
shift and rotate instructions, sld and srd, are microcoded on Cell.
This means that either the destination or the source can be aligned,
but not both, and seeing as we get better performance with the
destination aligned we choose this option.

While we're at it make a one line change from cmpldi r1,... to
cmpldi cr1,... for consistency.

Signed-off-by: Mark Nelson <markn@au1.ibm.com>
Signed-off-by: Paul Mackerras <paulus@samba.org>
2008-11-05 22:08:29 +11:00
..
alloc.c [POWERPC] Limit range of __init_ref_ok somewhat 2007-10-03 11:48:44 +10:00
checksum_32.S powerpc: Rename files to have consistent _32/_64 suffixes 2005-10-10 21:52:43 +10:00
checksum_64.S powerpc: Rename files to have consistent _32/_64 suffixes 2005-10-10 21:52:43 +10:00
code-patching.c PAGE_ALIGN(): correctly handle 64-bit values on 32-bit architectures 2008-07-24 10:47:21 -07:00
copy_32.S Remove obsolete #include <linux/config.h> 2006-06-30 19:25:36 +02:00
copypage_64.S powerpc: New copy_4K_page() 2008-09-15 11:07:42 -07:00
copyuser_64.S [POWERPC] Use mtocrf instruction in asm when CONFIG_POWER4_ONLY=y 2007-04-13 03:55:13 +10:00
crtsavres.S [POWERPC] Fix -Os kernel builds with newer gcc versions 2008-06-16 15:00:54 +10:00
devres.c [POWERPC] devres: Add devm_ioremap_prot() 2008-05-05 16:47:14 +10:00
div64.S powerpc: Fix a corner case in __div64_32 2005-10-20 09:37:02 +10:00
dma-noncoherent.c powerpc: Fix DMA offset for non-coherent DMA 2008-10-14 10:35:26 +11:00
feature-fixups-test.S powerpc: Fixup lwsync at runtime 2008-07-03 16:58:10 +10:00
feature-fixups.c powerpc: Use WARN_ON(1) instead of __WARN() 2008-07-22 10:39:34 +10:00
locks.c [POWERPC] Make ppc64_defconfig without CONFIG_PPC_PSERIES build 2007-03-09 15:03:24 +11:00
Makefile powerpc: Remove use of CONFIG_PPC_MERGE 2008-08-04 13:18:17 +10:00
mem_64.S [POWERPC] Use mtocrf instruction in asm when CONFIG_POWER4_ONLY=y 2007-04-13 03:55:13 +10:00
memcpy_64.S powerpc: Update 64bit memcpy() using CPU_FTR_UNALIGNED_LD_STD 2008-11-05 22:08:29 +11:00
rheap.c [POWERPC] Make rheap safe for spinlocks 2008-04-17 09:50:38 -05:00
sstep.c [POWERPC] Emulate more instructions in software 2007-04-24 21:31:57 +10:00
string.S powerpc: Use PPC_LONG and PPC_LONG_ALIGN in lib/string.S 2008-07-22 10:39:35 +10:00
usercopy_64.c powerpc: Rename files to have consistent _32/_64 suffixes 2005-10-10 21:52:43 +10:00