leandrof/linux
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
9a83c84c3a
linux/arch/arm64/include/asm
History
Julien Grall 6dcdefcde4 arm64/fpsimd: Don't disable softirq when touching FPSIMD/SVE state 
When the kernel is compiled with CONFIG_KERNEL_MODE_NEON, some part of
the kernel may be able to use FPSIMD/SVE. This is for instance the case
for crypto code.

Any use of FPSIMD/SVE in the kernel are clearly marked by using the
function kernel_neon_{begin, end}. Furthermore, this can only be used
when may_use_simd() returns true.

The current implementation of may_use_simd() allows softirq to use
FPSIMD/SVE unless it is currently in use (i.e kernel_neon_busy is true).
When in use, softirqs usually fall back to a software method.

At the moment, as a softirq may use FPSIMD/SVE, softirqs are disabled
when touching the FPSIMD/SVE context. This has the drawback to disable
all softirqs even if they are not using FPSIMD/SVE.

Since a softirq is supposed to check may_use_simd() anyway before
attempting to use FPSIMD/SVE, there is limited reason to keep softirq
disabled when touching the FPSIMD/SVE context. Instead, we can simply
disable preemption and mark the FPSIMD/SVE context as in use by setting
CPU's fpsimd_context_busy flag.

Two new helpers {get, put}_cpu_fpsimd_context are introduced to mark
the area using FPSIMD/SVE context and they are used to replace
local_bh_{disable, enable}. The functions kernel_neon_{begin, end} are
also re-implemented to use the new helpers.

Additionally, double-underscored versions of the helpers are provided to
called when preemption is already disabled. These are only relevant on
paths where irqs are disabled anyway, so they are not needed for
correctness in the current code. Let's use them anyway though: this
marks critical sections clearly and will help to avoid mistakes during
future maintenance.

The change has been benchmarked on Linux 5.1-rc4 with defconfig.

On Juno2:
    * hackbench 100 process 1000 (10 times)
    * .7% quicker

On ThunderX 2:
    * hackbench 1000 process 1000 (20 times)
    * 3.4% quicker

Reviewed-by: Dave Martin <dave.martin@arm.com>
Acked-by: Marc Zyngier <marc.zyngier@arm.com>
Signed-off-by: Julien Grall <julien.grall@arm.com>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2019-06-04 13:17:37 +01:00
..
xen arm64/xen: fix xen-swiotlb cache flushing 2019-01-23 22:14:56 +01:00
acenv.h
acpi.h arm64: KVM/mm: Move SEA handling behind a single 'claim' interface 2019-02-07 23:10:45 +01:00
alternative.h arm64: alternative: Apply alternatives early in boot process 2019-02-06 10:05:20 +00:00
arch_gicv3.h arm64: gic-v3: Implement arch support for priority masking 2019-02-06 10:05:21 +00:00
arch_timer.h Merge branch 'for-next/timers' of git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux into for-next/core 2019-05-01 15:45:36 +01:00
arm_dsu_pmu.h
arm-cci.h
asm-bug.h
asm-offsets.h
asm-prototypes.h arm64: asm-prototypes: Fix fat-fingered typo in comment 2019-01-10 11:11:46 +00:00
asm-uaccess.h arm64: Rename get_thread_info() 2019-02-26 16:57:59 +00:00
assembler.h arm64: assembler: Update comment above cond_yield_neon() macro 2019-05-14 10:52:45 +01:00
atomic_ll_sc.h Merge branch 'locking/atomics' into locking/core, to pick up WIP commits 2019-02-11 14:27:05 +01:00
atomic_lse.h Merge branch 'locking/atomics' into locking/core, to pick up WIP commits 2019-02-11 14:27:05 +01:00
atomic.h arm64, locking/atomics: Use instrumented atomics 2018-11-01 11:01:40 +01:00
barrier.h arm64: instrument smp_{load_acquire,store_release} 2019-04-16 16:28:00 +01:00
bitops.h locking/atomics/arm64, arm64/bitops: Include <asm-generic/bitops/ext2-atomic-setbit.h> 2018-06-21 12:52:12 +02:00
bitrev.h
boot.h treewide: replace #include <asm/sizes.h> with #include <linux/sizes.h> 2019-05-14 19:52:52 -07:00
brk-imm.h arm64: probes: Move magic BRK values into brk-imm.h 2019-04-09 11:21:13 +01:00
bug.h
cache.h kasan, arm64: remove redundant ARCH_SLAB_MINALIGN define 2019-01-16 12:09:11 +00:00
cacheflush.h arm64: IPI each CPU after invalidating the I-cache for kernel mappings 2018-07-05 17:24:36 +01:00
checksum.h
clocksource.h
cmpxchg.h Merge branch 'locking/atomics' into locking/core, to pick up WIP commits 2019-02-11 14:27:05 +01:00
compat.h Merge branch 'timers-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip 2018-10-25 11:14:36 -07:00
cpu_ops.h
cpu.h
cpucaps.h arm64: Handle erratum 1418040 as a superset of erratum 1188873 2019-05-23 15:40:30 +01:00
cpufeature.h arm64: mark (__)cpus_have_const_cap as __always_inline 2019-05-14 19:52:48 -07:00
cpuidle.h
cputype.h arm64: Add part number for Neoverse N1 2019-04-30 14:46:06 +01:00
current.h
daifflags.h arm64 updates for 5.1: 2019-03-10 10:17:23 -07:00
dcc.h
debug-monitors.h arm64: probes: Move magic BRK values into brk-imm.h 2019-04-09 11:21:13 +01:00
device.h arm64/xen: fix xen-swiotlb cache flushing 2019-01-23 22:14:56 +01:00
dma-mapping.h dma-mapping: add a kconfig symbol for arch_teardown_dma_ops availability 2019-02-13 19:12:50 +01:00
dmi.h
efi.h arm64: irqflags: Use ICC_PMR_EL1 for interrupt masking 2019-02-06 10:05:19 +00:00
elf.h arm64: compat: Alloc separate pages for vectors and sigpage 2019-04-23 18:01:31 +01:00
esr.h arm64: Handle trapped DC CVADP 2019-04-16 16:27:36 +01:00
exception.h arm64: kernel: Prepare for a DISR user 2018-01-16 15:07:12 +00:00
exec.h
extable.h
fb.h
fixmap.h firmware: arm_sdei: Add ACPI GHES registration helper 2019-02-11 11:07:49 +01:00
fpsimd.h arm64/fpsimd: Introduce fpsimd_save_and_flush_cpu_state() and use it 2019-06-04 13:17:30 +01:00
fpsimdmacros.h arm64/sve: Write ZCR_EL1 on context switch only if changed 2018-05-17 18:19:53 +01:00
ftrace.h arm64 festive updates for 4.21 2018-12-25 17:41:56 -08:00
futex.h arm64 updates for 5.2 2019-05-06 17:54:22 -07:00
hardirq.h arm64: Fix HCR.TGE status for NMI contexts 2019-02-06 10:05:16 +00:00
hugetlb.h treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 174 2019-05-30 11:26:41 -07:00
hw_breakpoint.h perf/hw_breakpoint: Remove default hw_breakpoint_arch_parse() 2018-06-26 09:07:58 +02:00
hwcap.h arm64: Expose SVE2 features for userspace 2019-04-23 18:02:00 +01:00
hypervisor.h
image.h arm64: add image head flag definitions 2018-12-06 14:38:51 +00:00
insn.h arm64: insn: Add BUILD_BUG_ON() for invalid masks 2019-05-24 14:58:30 +01:00
io.h arm64/io: Remove useless definition of mmiowb() 2019-04-08 11:59:57 +01:00
irq_work.h
irq.h arm64: Use the new GENERIC_IRQ_MULTI_HANDLER 2018-08-03 12:14:09 +02:00
irqflags.h arm64: sysreg: Make mrs_s and msr_s macros work with Clang and LTO 2019-04-25 14:59:49 +01:00
jump_label.h arm64/kernel: jump_label: Switch to relative references 2018-09-27 17:56:47 +02:00
kasan.h kasan: add tag related helper functions 2018-12-28 12:11:43 -08:00
Kbuild treewide: Add SPDX license identifier - Kbuild 2019-05-30 11:32:33 -07:00
kernel-pgtable.h arm64/mm: Separate boot-time page tables from swapper_pg_dir 2018-09-25 15:10:54 +01:00
kexec.h arm64: kexec_file: allow for loading Image-format kernel 2018-12-06 14:38:52 +00:00
kgdb.h
kprobes.h treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 174 2019-05-30 11:26:41 -07:00
kvm_arm.h * ARM: selftests improvements, large PUD support for HugeTLB, 2018-12-26 11:46:28 -08:00
kvm_asm.h arm64: KVM: Encapsulate kvm_cpu_context in kvm_host_data 2019-04-24 15:35:24 +01:00
kvm_coproc.h
kvm_emulate.h KVM: arm/arm64: Context-switch ptrauth registers 2019-04-24 15:30:40 +01:00
kvm_host.h KVM: arm64: Move pmu hyp code under hyp's Makefile to avoid instrumentation 2019-05-24 14:53:20 +01:00
kvm_hyp.h * ARM: support for SVE and Pointer Authentication in guests, PMU improvements 2019-05-17 10:33:30 -07:00
kvm_mmio.h
kvm_mmu.h KVM: arm/arm64: vgic-its: Take the srcu lock when writing to guest memory 2019-03-19 17:56:56 +00:00
kvm_ptrauth.h KVM: arm/arm64: Context-switch ptrauth registers 2019-04-24 15:30:40 +01:00
kvm_ras.h arm64: KVM/mm: Move SEA handling behind a single 'claim' interface 2019-02-07 23:10:45 +01:00
linkage.h
lse.h arm64: lse: Include compiler_types.h and export.h for out-of-line LL/SC 2018-03-27 13:14:49 +01:00
memory.h treewide: replace #include <asm/sizes.h> with #include <linux/sizes.h> 2019-05-14 19:52:52 -07:00
mmu_context.h arm64: Kconfig: Re-jig CONFIG options for 52-bit VA 2018-12-10 18:42:18 +00:00
mmu.h arm64: Remove asm/memblock.h 2019-01-21 17:31:15 +00:00
mmzone.h
module.h arm64/ftrace: fix inadvertent BUG() in trampoline check 2019-04-08 16:58:13 +01:00
neon-intrinsics.h arm64/neon: Disable -Wincompatible-pointer-types when building with Clang 2019-02-18 10:54:47 +00:00
neon.h arm64: drop unused kernel_neon_begin_partial() macro 2018-07-31 10:13:50 +01:00
numa.h arm64: numa: separate out updates to percpu nodeid and NUMA node cpumap 2018-07-06 13:18:18 +01:00
page-def.h
page.h arm64: Kconfig: Remove ARCH_HAS_HOLES_MEMORYMODEL 2018-09-21 12:02:45 +01:00
paravirt.h x86/paravirt: Use a single ops structure 2018-09-03 16:50:35 +02:00
pci.h PCI: remove PCI_DMA_BUS_IS_PHYS 2018-05-07 07:15:41 +02:00
percpu.h arm64: percpu: Fix LSE implementation of value-returning pcpu atomics 2018-12-12 14:43:35 +00:00
perf_event.h arm64: perf: Treat EXCLUDE_EL* bit definitions as unsigned 2018-12-13 15:34:44 +00:00
pgalloc.h arm64: mm: enable per pmd page table lock 2019-04-09 11:21:50 +01:00
pgtable-hwdef.h arm64/mm: Move PTE_VALID from SW defined to HW page table entry definitions 2019-06-03 16:59:30 +01:00
pgtable-prot.h arm64/mm: Move PTE_VALID from SW defined to HW page table entry definitions 2019-06-03 16:59:30 +01:00
pgtable-types.h
pgtable.h arm64: mm: Remove pte_unmap_nested() 2019-04-30 12:02:20 +01:00
pointer_auth.h arm64: Use the correct style for SPDX License Identifier 2019-04-16 16:28:01 +01:00
preempt.h arm64: preempt: Provide our own implementation of asm/preempt.h 2018-12-07 12:35:53 +00:00
probes.h treewide: Replace GPLv2 boilerplate/reference with SPDX - rule 174 2019-05-30 11:26:41 -07:00
proc-fns.h
processor.h arm64: compat: Reduce address limit for 64K pages 2019-04-30 11:04:50 +01:00
ptdump.h arm64: dump: no need to check return value of debugfs_create functions 2019-01-31 17:38:19 +00:00
ptrace.h arm64: ptrace: Add function argument access API 2019-04-12 17:04:27 +01:00
sdei.h arm64: Use the correct style for SPDX License Identifier 2019-04-16 16:28:01 +01:00
seccomp.h
sections.h
shmparam.h
signal32.h arm64: compat: Alloc separate pages for vectors and sigpage 2019-04-23 18:01:31 +01:00
simd.h arm64/fpsimd: Don't disable softirq when touching FPSIMD/SVE state 2019-06-04 13:17:37 +01:00
smp_plat.h
smp.h arm64: smp: Fix compilation error 2019-01-03 15:14:32 +00:00
sparsemem.h
spinlock_types.h arm64: locking: Replace ticket lock implementation with qspinlock 2018-07-05 10:05:06 +01:00
spinlock.h arm64: locking: Replace ticket lock implementation with qspinlock 2018-07-05 10:05:06 +01:00
stack_pointer.h
stackprotector.h arm64: enable per-task stack canaries 2018-12-12 18:45:31 +00:00
stacktrace.h arm64: Add stack information to on_accessible_stack 2018-07-26 11:36:07 +01:00
stage2_pgtable.h KVM: ARM: Remove pgtable page standard functions from stage-2 page tables 2019-04-09 11:21:50 +01:00
stat.h y2038: globally rename compat_time to old_time32 2018-08-27 14:48:48 +02:00
string.h arm64: lib: use C string functions with KASAN enabled 2018-10-26 16:25:18 -07:00
suspend.h
sync_bitops.h arm64, locking/atomics: Use instrumented atomics 2018-11-01 11:01:40 +01:00
syscall_wrapper.h arm64: use the correct function type in SYSCALL_DEFINE0 2019-05-29 13:45:59 +01:00
syscall.h arm64: fix syscall_fn_t type 2019-05-29 13:45:59 +01:00
sysreg.h * ARM: support for SVE and Pointer Authentication in guests, PMU improvements 2019-05-17 10:33:30 -07:00
system_misc.h arm64: mm: Make show_pte() a static function 2019-04-03 13:36:54 +01:00
thread_info.h arm64: Remove documentation about TIF_USEDFPU 2019-02-26 16:41:10 +00:00
timex.h
tlb.h arm64 updates for 5.2 2019-05-06 17:54:22 -07:00
tlbflush.h arm64 festive updates for 4.21 2018-12-25 17:41:56 -08:00
topology.h sched/topology, arch/arm64: Rebuild the sched_domain hierarchy when the CPU capacity changes 2018-09-10 11:05:47 +02:00
traps.h signal/arm64: Add and use arm64_force_sig_ptrace_errno_trap 2018-09-27 21:55:15 +02:00
uaccess.h arm64 updates for 5.1: 2019-03-10 10:17:23 -07:00
unistd32.h uapi: Wire up the mount API syscalls on non-x86 arches [ver #2] 2019-05-16 12:23:45 -04:00
unistd.h uapi: Wire up the mount API syscalls on non-x86 arches [ver #2] 2019-05-16 12:23:45 -04:00
uprobes.h
vdso_datapage.h arm64: vdso: Fix clock_getres() for CLOCK_REALTIME 2019-04-16 18:15:56 +01:00
vdso.h
virt.h arm64: capabilities: Change scope of VHE to Boot CPU feature 2018-03-26 18:01:41 +01:00
vmap_stack.h arm64: Use the correct style for SPDX License Identifier 2019-04-16 16:28:01 +01:00
word-at-a-time.h
xor.h arm64: crypto: add NEON accelerated XOR implementation 2018-12-06 16:47:06 +00:00