leandrof/linux
1
0
Fork 0
forked from Minki/linux
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Merge branches 'for-next/acpi', 'for-next/boot', 'for-next/bpf', 'for-next/cpuinfo', 'for-next/fpsimd', 'for-next/misc', 'for-next/mm', 'for-next/pci', 'for-next/perf', 'for-next/ptrauth', 'for-next/sdei', 'for-next/selftests', 'for-next/stacktrace', 'for-next/svm', 'for-next/topology', 'for-next/tpyos' and 'for-next/vdso' into for-next/core

Browse Source 
Remove unused functions and parameters from ACPI IORT code.
(Zenghui Yu via Lorenzo Pieralisi)
* for-next/acpi:
  ACPI/IORT: Remove the unused inline functions
  ACPI/IORT: Drop the unused @ops of iort_add_device_replay()

Remove redundant code and fix documentation of caching behaviour for the
HVC_SOFT_RESTART hypercall.
(Pingfan Liu)
* for-next/boot:
  Documentation/kvm/arm: improve description of HVC_SOFT_RESTART
  arm64/relocate_kernel: remove redundant code

Improve reporting of unexpected kernel traps due to BPF JIT failure.
(Will Deacon)
* for-next/bpf:
  arm64: Improve diagnostics when trapping BRK with FAULT_BRK_IMM

Improve robustness of user-visible HWCAP strings and their corresponding
numerical constants.
(Anshuman Khandual)
* for-next/cpuinfo:
  arm64/cpuinfo: Define HWCAP name arrays per their actual bit definitions

Cleanups to handling of SVE and FPSIMD register state in preparation
for potential future optimisation of handling across syscalls.
(Julien Grall)
* for-next/fpsimd:
  arm64/sve: Implement a helper to load SVE registers from FPSIMD state
  arm64/sve: Implement a helper to flush SVE registers
  arm64/fpsimdmacros: Allow the macro "for" to be used in more cases
  arm64/fpsimdmacros: Introduce a macro to update ZCR_EL1.LEN
  arm64/signal: Update the comment in preserve_sve_context
  arm64/fpsimd: Update documentation of do_sve_acc

Miscellaneous changes.
(Tian Tao and others)
* for-next/misc:
  arm64/mm: return cpu_all_mask when node is NUMA_NO_NODE
  arm64: mm: Fix missing-prototypes in pageattr.c
  arm64/fpsimd: Fix missing-prototypes in fpsimd.c
  arm64: hibernate: Remove unused including <linux/version.h>
  arm64/mm: Refactor {pgd, pud, pmd, pte}_ERROR()
  arm64: Remove the unused include statements
  arm64: get rid of TEXT_OFFSET
  arm64: traps: Add str of description to panic() in die()

Memory management updates and cleanups.
(Anshuman Khandual and others)
* for-next/mm:
  arm64: dbm: Invalidate local TLB when setting TCR_EL1.HD
  arm64: mm: Make flush_tlb_fix_spurious_fault() a no-op
  arm64/mm: Unify CONT_PMD_SHIFT
  arm64/mm: Unify CONT_PTE_SHIFT
  arm64/mm: Remove CONT_RANGE_OFFSET
  arm64/mm: Enable THP migration
  arm64/mm: Change THP helpers to comply with generic MM semantics
  arm64/mm/ptdump: Add address markers for BPF regions

Allow prefetchable PCI BARs to be exposed to userspace using normal
non-cacheable mappings.
(Clint Sbisa)
* for-next/pci:
  arm64: Enable PCI write-combine resources under sysfs

Perf/PMU driver updates.
(Julien Thierry and others)
* for-next/perf:
  perf: arm-cmn: Fix conversion specifiers for node type
  perf: arm-cmn: Fix unsigned comparison to less than zero
  arm_pmu: arm64: Use NMIs for PMU
  arm_pmu: Introduce pmu_irq_ops
  KVM: arm64: pmu: Make overflow handler NMI safe
  arm64: perf: Defer irq_work to IPI_IRQ_WORK
  arm64: perf: Remove PMU locking
  arm64: perf: Avoid PMXEV* indirection
  arm64: perf: Add missing ISB in armv8pmu_enable_counter()
  perf: Add Arm CMN-600 PMU driver
  perf: Add Arm CMN-600 DT binding
  arm64: perf: Add support caps under sysfs
  drivers/perf: thunderx2_pmu: Fix memory resource error handling
  drivers/perf: xgene_pmu: Fix uninitialized resource struct
  perf: arm_dsu: Support DSU ACPI devices
  arm64: perf: Remove unnecessary event_idx check
  drivers/perf: hisi: Add missing include of linux/module.h
  arm64: perf: Add general hardware LLC events for PMUv3

Support for the Armv8.3 Pointer Authentication enhancements.
(By Amit Daniel Kachhap)
* for-next/ptrauth:
  arm64: kprobe: clarify the comment of steppable hint instructions
  arm64: kprobe: disable probe of fault prone ptrauth instruction
  arm64: cpufeature: Modify address authentication cpufeature to exact
  arm64: ptrauth: Introduce Armv8.3 pointer authentication enhancements
  arm64: traps: Allow force_signal_inject to pass esr error code
  arm64: kprobe: add checks for ARMv8.3-PAuth combined instructions

Tonnes of cleanup to the SDEI driver.
(Gavin Shan)
* for-next/sdei:
  firmware: arm_sdei: Remove _sdei_event_unregister()
  firmware: arm_sdei: Remove _sdei_event_register()
  firmware: arm_sdei: Introduce sdei_do_local_call()
  firmware: arm_sdei: Cleanup on cross call function
  firmware: arm_sdei: Remove while loop in sdei_event_unregister()
  firmware: arm_sdei: Remove while loop in sdei_event_register()
  firmware: arm_sdei: Remove redundant error message in sdei_probe()
  firmware: arm_sdei: Remove duplicate check in sdei_get_conduit()
  firmware: arm_sdei: Unregister driver on error in sdei_init()
  firmware: arm_sdei: Avoid nested statements in sdei_init()
  firmware: arm_sdei: Retrieve event number from event instance
  firmware: arm_sdei: Common block for failing path in sdei_event_create()
  firmware: arm_sdei: Remove sdei_is_err()

Selftests for Pointer Authentication and FPSIMD/SVE context-switching.
(Mark Brown and Boyan Karatotev)
* for-next/selftests:
  selftests: arm64: Add build and documentation for FP tests
  selftests: arm64: Add wrapper scripts for stress tests
  selftests: arm64: Add utility to set SVE vector lengths
  selftests: arm64: Add stress tests for FPSMID and SVE context switching
  selftests: arm64: Add test for the SVE ptrace interface
  selftests: arm64: Test case for enumeration of SVE vector lengths
  kselftests/arm64: add PAuth tests for single threaded consistency and differently initialized keys
  kselftests/arm64: add PAuth test for whether exec() changes keys
  kselftests/arm64: add nop checks for PAuth tests
  kselftests/arm64: add a basic Pointer Authentication test

Implementation of ARCH_STACKWALK for unwinding.
(Mark Brown)
* for-next/stacktrace:
  arm64: Move console stack display code to stacktrace.c
  arm64: stacktrace: Convert to ARCH_STACKWALK
  arm64: stacktrace: Make stack walk callback consistent with generic code
  stacktrace: Remove reliable argument from arch_stack_walk() callback

Support for ASID pinning, which is required when sharing page-tables with
the SMMU.
(Jean-Philippe Brucker)
* for-next/svm:
  arm64: cpufeature: Export symbol read_sanitised_ftr_reg()
  arm64: mm: Pin down ASIDs for sharing mm with devices

Rely on firmware tables for establishing CPU topology.
(Valentin Schneider)
* for-next/topology:
  arm64: topology: Stop using MPIDR for topology information

Spelling fixes.
(Xiaoming Ni and Yanfei Xu)
* for-next/tpyos:
  arm64/numa: Fix a typo in comment of arm64_numa_init
  arm64: fix some spelling mistakes in the comments by codespell

vDSO cleanups.
(Will Deacon)
* for-next/vdso:
  arm64: vdso: Fix unusual formatting in *setup_additional_pages()
  arm64: vdso32: Remove a bunch of #ifdef CONFIG_COMPAT_VDSO guards
This commit is contained in:
Will Deacon 2020-10-02 12:01:41 +01:00
commit 57b8b1b435
105 changed files with 5573 additions and 785 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

65
Documentation/admin-guide/perf/arm-cmn.rst Normal file
View File

@ -0,0 +1,65 @@
=============================
Arm Coherent Mesh Network PMU
=============================
CMN-600 is a configurable mesh interconnect consisting of a rectangular
grid of crosspoints (XPs), with each crosspoint supporting up to two
device ports to which various AMBA CHI agents are attached.
CMN implements a distributed PMU design as part of its debug and trace
functionality. This consists of a local monitor (DTM) at every XP, which
counts up to 4 event signals from the connected device nodes and/or the
XP itself. Overflow from these local counters is accumulated in up to 8
global counters implemented by the main controller (DTC), which provides
overall PMU control and interrupts for global counter overflow.
PMU events
----------
The PMU driver registers a single PMU device for the whole interconnect,
see /sys/bus/event_source/devices/arm_cmn. Multi-chip systems may link
more than one CMN together via external CCIX links - in this situation,
each mesh counts its own events entirely independently, and additional
PMU devices will be named arm_cmn_{1..n}.
Most events are specified in a format based directly on the TRM
definitions - "type" selects the respective node type, and "eventid" the
event number. Some events require an additional occupancy ID, which is
specified by "occupid".
* Since RN-D nodes do not have any distinct events from RN-I nodes, they
are treated as the same type (0xa), and the common event templates are
named "rnid_*".
* The cycle counter is treated as a synthetic event belonging to the DTC
node ("type" == 0x3, "eventid" is ignored).
* XP events also encode the port and channel in the "eventid" field, to
match the underlying pmu_event0_id encoding for the pmu_event_sel
register. The event templates are named with prefixes to cover all
permutations.
By default each event provides an aggregate count over all nodes of the
given type. To target a specific node, "bynodeid" must be set to 1 and
"nodeid" to the appropriate value derived from the CMN configuration
(as defined in the "Node ID Mapping" section of the TRM).
Watchpoints
-----------
The PMU can also count watchpoint events to monitor specific flit
traffic. Watchpoints are treated as a synthetic event type, and like PMU
events can be global or targeted with a particular XP's "nodeid" value.
Since the watchpoint direction is otherwise implicit in the underlying
register selection, separate events are provided for flit uploads and
downloads.
The flit match value and mask are passed in config1 and config2 ("val"
and "mask" respectively). "wp_dev_sel", "wp_chn_sel", "wp_grp" and
"wp_exclusive" are specified per the TRM definitions for dtm_wp_config0.
Where a watchpoint needs to match fields from both match groups on the
REQ or SNP channel, it can be specified as two events - one for each
group - with the same nonzero "combine" value. The count for such a
pair of combined events will be attributed to the primary match.
Watchpoint events with a "combine" value of 0 are considered independent
and will count individually.

1
Documentation/admin-guide/perf/index.rst
View File

@ -12,6 +12,7 @@ Performance monitor support
qcom_l2_pmu
qcom_l3_pmu
arm-ccn
arm-cmn
xgene-pmu
arm_dsu_pmu
thunderx2-pmu

57
Documentation/devicetree/bindings/perf/arm,cmn.yaml Normal file
View File

@ -0,0 +1,57 @@
# SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
# Copyright 2020 Arm Ltd.
%YAML 1.2
---
$id: http://devicetree.org/schemas/perf/arm,cmn.yaml#
$schema: http://devicetree.org/meta-schemas/core.yaml#
title: Arm CMN (Coherent Mesh Network) Performance Monitors
maintainers:
- Robin Murphy <robin.murphy@arm.com>
properties:
compatible:
const: arm,cmn-600
reg:
items:
- description: Physical address of the base (PERIPHBASE) and
size (up to 64MB) of the configuration address space.
interrupts:
minItems: 1
maxItems: 4
items:
- description: Overflow interrupt for DTC0
- description: Overflow interrupt for DTC1
- description: Overflow interrupt for DTC2
- description: Overflow interrupt for DTC3
description: One interrupt for each DTC domain implemented must
be specified, in order. DTC0 is always present.
arm,root-node:
$ref: /schemas/types.yaml#/definitions/uint32
description: Offset from PERIPHBASE of the configuration
discovery node (see TRM definition of ROOTNODEBASE).
required:
- compatible
- reg
- interrupts
- arm,root-node
additionalProperties: false
examples:
- |
#include <dt-bindings/interrupt-controller/arm-gic.h>
#include <dt-bindings/interrupt-controller/irq.h>
pmu@50000000 {
compatible = "arm,cmn-600";
reg = <0x50000000 0x4000000>;
/* 4x2 mesh with one DTC, and CFG node at 0,1,1,0 */
interrupts = <GIC_SPI 46 IRQ_TYPE_LEVEL_HIGH>;
arm,root-node = <0x104000>;
};
...

6
Documentation/virt/kvm/arm/hyp-abi.rst
View File

@ -54,9 +54,9 @@ these functions (see arch/arm{,64}/include/asm/virt.h):
x3 = x1's value when entering the next payload (arm64)
x4 = x2's value when entering the next payload (arm64)
Mask all exceptions, disable the MMU, move the arguments into place
(arm64 only), and jump to the restart address while at HYP/EL2. This
hypercall is not expected to return to its caller.
Mask all exceptions, disable the MMU, clear I+D bits, move the arguments
into place (arm64 only), and jump to the restart address while at HYP/EL2.
This hypercall is not expected to return to its caller.
Any other value of r0/x0 triggers a hypervisor-specific handling,
which is not documented here.

13
arch/arm64/Kconfig
View File

@ -29,6 +29,7 @@ config ARM64
select ARCH_HAS_SETUP_DMA_OPS
select ARCH_HAS_SET_DIRECT_MAP
select ARCH_HAS_SET_MEMORY
select ARCH_STACKWALK
select ARCH_HAS_STRICT_KERNEL_RWX
select ARCH_HAS_STRICT_MODULE_RWX
select ARCH_HAS_SYNC_DMA_FOR_DEVICE
@ -211,12 +212,18 @@ config ARM64_PAGE_SHIFT
default 14 if ARM64_16K_PAGES
default 12
config ARM64_CONT_SHIFT
config ARM64_CONT_PTE_SHIFT
int
default 5 if ARM64_64K_PAGES
default 7 if ARM64_16K_PAGES
default 4
config ARM64_CONT_PMD_SHIFT
int
default 5 if ARM64_64K_PAGES
default 5 if ARM64_16K_PAGES
default 4
config ARCH_MMAP_RND_BITS_MIN
default 14 if ARM64_64K_PAGES
default 16 if ARM64_16K_PAGES
@ -1876,6 +1883,10 @@ config ARCH_ENABLE_HUGEPAGE_MIGRATION
def_bool y
depends on HUGETLB_PAGE && MIGRATION
config ARCH_ENABLE_THP_MIGRATION
def_bool y
depends on TRANSPARENT_HUGEPAGE
menu "Power management options"
source "kernel/power/Kconfig"

6
arch/arm64/Makefile
View File

@ -11,7 +11,6 @@
# Copyright (C) 1995-2001 by Russell King
LDFLAGS_vmlinux :=--no-undefined -X
CPPFLAGS_vmlinux.lds = -DTEXT_OFFSET=$(TEXT_OFFSET)
ifeq ($(CONFIG_RELOCATABLE), y)
# Pass --no-apply-dynamic-relocs to restore pre-binutils-2.27 behaviour
@ -132,9 +131,6 @@ endif
# Default value
head-y := arch/arm64/kernel/head.o
# The byte offset of the kernel image in RAM from the start of RAM.
TEXT_OFFSET := 0x0
ifeq ($(CONFIG_KASAN_SW_TAGS), y)
KASAN_SHADOW_SCALE_SHIFT := 4
else
@ -145,8 +141,6 @@ KBUILD_CFLAGS += -DKASAN_SHADOW_SCALE_SHIFT=$(KASAN_SHADOW_SCALE_SHIFT)
KBUILD_CPPFLAGS += -DKASAN_SHADOW_SCALE_SHIFT=$(KASAN_SHADOW_SCALE_SHIFT)
KBUILD_AFLAGS += -DKASAN_SHADOW_SCALE_SHIFT=$(KASAN_SHADOW_SCALE_SHIFT)
export TEXT_OFFSET
core-y += arch/arm64/
libs-y := arch/arm64/lib/ $(libs-y)
libs-$(CONFIG_EFI_STUB) += $(objtree)/drivers/firmware/efi/libstub/lib.a

3
arch/arm64/include/asm/boot.h
View File

@ -13,8 +13,7 @@
#define MAX_FDT_SIZE SZ_2M
/*
* arm64 requires the kernel image to placed
* TEXT_OFFSET bytes beyond a 2 MB aligned base
* arm64 requires the kernel image to placed at a 2 MB aligned base address
*/
#define MIN_KIMG_ALIGN SZ_2M

2
arch/arm64/include/asm/cpu_ops.h
View File

@ -21,7 +21,7 @@
* mechanism for doing so, tests whether it is possible to boot
* the given CPU.
* @cpu_boot: Boots a cpu into the kernel.
* @cpu_postboot: Optionally, perform any post-boot cleanup or necesary
* @cpu_postboot: Optionally, perform any post-boot cleanup or necessary
* synchronisation. Called from the cpu being booted.
* @cpu_can_disable: Determines whether a CPU can be disabled based on
* mechanism-specific information.

2
arch/arm64/include/asm/cpufeature.h
View File

@ -358,7 +358,7 @@ static inline int cpucap_default_scope(const struct arm64_cpu_capabilities *cap)
}
/*
* Generic helper for handling capabilties with multiple (match,enable) pairs
* Generic helper for handling capabilities with multiple (match,enable) pairs
* of call backs, sharing the same capability bit.
* Iterate over each entry to see if at least one matches.
*/

4
arch/arm64/include/asm/esr.h
View File

@ -35,7 +35,9 @@
#define ESR_ELx_EC_SYS64 (0x18)
#define ESR_ELx_EC_SVE (0x19)
#define ESR_ELx_EC_ERET (0x1a) /* EL2 only */
/* Unallocated EC: 0x1b - 0x1E */
/* Unallocated EC: 0x1B */
#define ESR_ELx_EC_FPAC (0x1C) /* EL1 and above */
/* Unallocated EC: 0x1D - 0x1E */
#define ESR_ELx_EC_IMP_DEF (0x1f) /* EL3 only */
#define ESR_ELx_EC_IABT_LOW (0x20)
#define ESR_ELx_EC_IABT_CUR (0x21)

1
arch/arm64/include/asm/exception.h
View File

@ -47,4 +47,5 @@ void bad_el0_sync(struct pt_regs *regs, int reason, unsigned int esr);
void do_cp15instr(unsigned int esr, struct pt_regs *regs);
void do_el0_svc(struct pt_regs *regs);
void do_el0_svc_compat(struct pt_regs *regs);
void do_ptrauth_fault(struct pt_regs *regs, unsigned int esr);
#endif /* __ASM_EXCEPTION_H */

9
arch/arm64/include/asm/extable.h
View File

@ -22,6 +22,15 @@ struct exception_table_entry
#define ARCH_HAS_RELATIVE_EXTABLE
static inline bool in_bpf_jit(struct pt_regs *regs)
{
if (!IS_ENABLED(CONFIG_BPF_JIT))
return false;
return regs->pc >= BPF_JIT_REGION_START &&
regs->pc < BPF_JIT_REGION_END;
}
#ifdef CONFIG_BPF_JIT
int arm64_bpf_fixup_exception(const struct exception_table_entry *ex,
struct pt_regs *regs);

3
arch/arm64/include/asm/fpsimd.h
View File

@ -69,6 +69,9 @@ static inline void *sve_pffr(struct thread_struct *thread)
extern void sve_save_state(void *state, u32 *pfpsr);
extern void sve_load_state(void const *state, u32 const *pfpsr,
unsigned long vq_minus_1);
extern void sve_flush_live(void);
extern void sve_load_from_fpsimd_state(struct user_fpsimd_state const *state,
unsigned long vq_minus_1);
extern unsigned int sve_get_vl(void);
struct arm64_cpu_capabilities;

48
arch/arm64/include/asm/fpsimdmacros.h
View File

@ -164,25 +164,59 @@
| ((\np) << 5)
.endm
/* PFALSE P\np.B */
.macro _sve_pfalse np
_sve_check_preg \np
.inst 0x2518e400 \
| (\np)
.endm
.macro __for from:req, to:req
.if (\from) == (\to)
_for__body \from
_for__body %\from
.else
__for \from, (\from) + ((\to) - (\from)) / 2
__for (\from) + ((\to) - (\from)) / 2 + 1, \to
__for %\from, %((\from) + ((\to) - (\from)) / 2)
__for %((\from) + ((\to) - (\from)) / 2 + 1), %\to
.endif
.endm
.macro _for var:req, from:req, to:req, insn:vararg
.macro _for__body \var:req
.noaltmacro
\insn
.altmacro
.endm
.altmacro
__for \from, \to
.noaltmacro
.purgem _for__body
.endm
/* Update ZCR_EL1.LEN with the new VQ */
.macro sve_load_vq xvqminus1, xtmp, xtmp2
mrs_s \xtmp, SYS_ZCR_EL1
bic \xtmp2, \xtmp, ZCR_ELx_LEN_MASK
orr \xtmp2, \xtmp2, \xvqminus1
cmp \xtmp2, \xtmp
b.eq 921f
msr_s SYS_ZCR_EL1, \xtmp2 //self-synchronising
921:
.endm
/* Preserve the first 128-bits of Znz and zero the rest. */
.macro _sve_flush_z nz
_sve_check_zreg \nz
mov v\nz\().16b, v\nz\().16b
.endm
.macro sve_flush
_for n, 0, 31, _sve_flush_z \n
_for n, 0, 15, _sve_pfalse \n
_sve_wrffr 0
.endm
.macro sve_save nxbase, xpfpsr, nxtmp
_for n, 0, 31, _sve_str_v \n, \nxbase, \n - 34
_for n, 0, 15, _sve_str_p \n, \nxbase, \n - 16
@ -197,13 +231,7 @@
.endm
.macro sve_load nxbase, xpfpsr, xvqminus1, nxtmp, xtmp2
mrs_s x\nxtmp, SYS_ZCR_EL1
bic \xtmp2, x\nxtmp, ZCR_ELx_LEN_MASK
orr \xtmp2, \xtmp2, \xvqminus1
cmp \xtmp2, x\nxtmp
b.eq 921f
msr_s SYS_ZCR_EL1, \xtmp2 // self-synchronising
921:
sve_load_vq \xvqminus1, x\nxtmp, \xtmp2
_for n, 0, 31, _sve_ldr_v \n, \nxbase, \n - 34
_sve_ldr_p 0, \nxbase
_sve_wrffr 0

9
arch/arm64/include/asm/hwcap.h
View File

@ -8,18 +8,27 @@
#include <uapi/asm/hwcap.h>
#include <asm/cpufeature.h>
#define COMPAT_HWCAP_SWP (1 << 0)
#define COMPAT_HWCAP_HALF (1 << 1)
#define COMPAT_HWCAP_THUMB (1 << 2)
#define COMPAT_HWCAP_26BIT (1 << 3)
#define COMPAT_HWCAP_FAST_MULT (1 << 4)
#define COMPAT_HWCAP_FPA (1 << 5)
#define COMPAT_HWCAP_VFP (1 << 6)
#define COMPAT_HWCAP_EDSP (1 << 7)
#define COMPAT_HWCAP_JAVA (1 << 8)
#define COMPAT_HWCAP_IWMMXT (1 << 9)
#define COMPAT_HWCAP_CRUNCH (1 << 10)
#define COMPAT_HWCAP_THUMBEE (1 << 11)
#define COMPAT_HWCAP_NEON (1 << 12)
#define COMPAT_HWCAP_VFPv3 (1 << 13)
#define COMPAT_HWCAP_VFPV3D16 (1 << 14)
#define COMPAT_HWCAP_TLS (1 << 15)
#define COMPAT_HWCAP_VFPv4 (1 << 16)
#define COMPAT_HWCAP_IDIVA (1 << 17)
#define COMPAT_HWCAP_IDIVT (1 << 18)
#define COMPAT_HWCAP_IDIV (COMPAT_HWCAP_IDIVA|COMPAT_HWCAP_IDIVT)
#define COMPAT_HWCAP_VFPD32 (1 << 19)
#define COMPAT_HWCAP_LPAE (1 << 20)
#define COMPAT_HWCAP_EVTSTRM (1 << 21)

4
arch/arm64/include/asm/insn.h
View File

@ -359,9 +359,13 @@ __AARCH64_INSN_FUNCS(brk, 0xFFE0001F, 0xD4200000)
__AARCH64_INSN_FUNCS(exception, 0xFF000000, 0xD4000000)
__AARCH64_INSN_FUNCS(hint, 0xFFFFF01F, 0xD503201F)
__AARCH64_INSN_FUNCS(br, 0xFFFFFC1F, 0xD61F0000)
__AARCH64_INSN_FUNCS(br_auth, 0xFEFFF800, 0xD61F0800)
__AARCH64_INSN_FUNCS(blr, 0xFFFFFC1F, 0xD63F0000)
__AARCH64_INSN_FUNCS(blr_auth, 0xFEFFF800, 0xD63F0800)
__AARCH64_INSN_FUNCS(ret, 0xFFFFFC1F, 0xD65F0000)
__AARCH64_INSN_FUNCS(ret_auth, 0xFFFFFBFF, 0xD65F0BFF)
__AARCH64_INSN_FUNCS(eret, 0xFFFFFFFF, 0xD69F03E0)
__AARCH64_INSN_FUNCS(eret_auth, 0xFFFFFBFF, 0xD69F0BFF)
__AARCH64_INSN_FUNCS(mrs, 0xFFF00000, 0xD5300000)
__AARCH64_INSN_FUNCS(msr_imm, 0xFFF8F01F, 0xD500401F)
__AARCH64_INSN_FUNCS(msr_reg, 0xFFF00000, 0xD5100000)

2
arch/arm64/include/asm/kernel-pgtable.h
View File

@ -86,7 +86,7 @@
+ EARLY_PGDS((vstart), (vend)) /* each PGDIR needs a next level page table */ \
+ EARLY_PUDS((vstart), (vend)) /* each PUD needs a next level page table */ \
+ EARLY_PMDS((vstart), (vend))) /* each PMD needs a next level page table */
#define INIT_DIR_SIZE (PAGE_SIZE * EARLY_PAGES(KIMAGE_VADDR + TEXT_OFFSET, _end))
#define INIT_DIR_SIZE (PAGE_SIZE * EARLY_PAGES(KIMAGE_VADDR, _end))
#define IDMAP_DIR_SIZE (IDMAP_PGTABLE_LEVELS * PAGE_SIZE)
#ifdef CONFIG_ARM64_SW_TTBR0_PAN

2
arch/arm64/include/asm/kvm_arm.h
View File

@ -66,7 +66,7 @@
* TWI: Trap WFI
* TIDCP: Trap L2CTLR/L2ECTLR
* BSU_IS: Upgrade barriers to the inner shareable domain
* FB: Force broadcast of all maintainance operations
* FB: Force broadcast of all maintenance operations
* AMO: Override CPSR.A and enable signaling with VA
* IMO: Override CPSR.I and enable signaling with VI
* FMO: Override CPSR.F and enable signaling with VF

2
arch/arm64/include/asm/memory.h
View File

@ -169,7 +169,7 @@ extern s64 memstart_addr;
/* PHYS_OFFSET - the physical address of the start of memory. */
#define PHYS_OFFSET ({ VM_BUG_ON(memstart_addr & 1); memstart_addr; })
/* the virtual base of the kernel image (minus TEXT_OFFSET) */
/* the virtual base of the kernel image */
extern u64 kimage_vaddr;
/* the offset between the kernel virtual and physical mappings */

3
arch/arm64/include/asm/mmu.h
View File

@ -17,11 +17,14 @@
#ifndef __ASSEMBLY__
#include <linux/refcount.h>
typedef struct {
atomic64_t id;
#ifdef CONFIG_COMPAT
void *sigpage;
#endif
refcount_t pinned;
void *vdso;
unsigned long flags;
} mm_context_t;

11
arch/arm64/include/asm/mmu_context.h
View File

@ -177,7 +177,13 @@ static inline void cpu_replace_ttbr1(pgd_t *pgdp)
#define destroy_context(mm) do { } while(0)
void check_and_switch_context(struct mm_struct *mm);
#define init_new_context(tsk,mm) ({ atomic64_set(&(mm)->context.id, 0); 0; })
static inline int
init_new_context(struct task_struct *tsk, struct mm_struct *mm)
{
atomic64_set(&mm->context.id, 0);
refcount_set(&mm->context.pinned, 0);
return 0;
}
#ifdef CONFIG_ARM64_SW_TTBR0_PAN
static inline void update_saved_ttbr0(struct task_struct *tsk,
@ -248,6 +254,9 @@ switch_mm(struct mm_struct *prev, struct mm_struct *next,
void verify_cpu_asid_bits(void);
void post_ttbr_update_workaround(void);
unsigned long arm64_mm_context_get(struct mm_struct *mm);
void arm64_mm_context_put(struct mm_struct *mm);
#endif /* !__ASSEMBLY__ */
#endif /* !__ASM_MMU_CONTEXT_H */

3
arch/arm64/include/asm/numa.h
View File

@ -25,6 +25,9 @@ const struct cpumask *cpumask_of_node(int node);
/* Returns a pointer to the cpumask of CPUs on Node 'node'. */
static inline const struct cpumask *cpumask_of_node(int node)
{
if (node == NUMA_NO_NODE)
return cpu_all_mask;
return node_to_cpumask_map[node];
}
#endif

5
arch/arm64/include/asm/page-def.h
View File

@ -11,13 +11,8 @@
#include <linux/const.h>
/* PAGE_SHIFT determines the page size */
/* CONT_SHIFT determines the number of pages which can be tracked together */
#define PAGE_SHIFT CONFIG_ARM64_PAGE_SHIFT
#define CONT_SHIFT CONFIG_ARM64_CONT_SHIFT
#define PAGE_SIZE (_AC(1, UL) << PAGE_SHIFT)
#define PAGE_MASK (~(PAGE_SIZE-1))
#define CONT_SIZE (_AC(1, UL) << (CONT_SHIFT + PAGE_SHIFT))
#define CONT_MASK (~(CONT_SIZE-1))
#endif /* __ASM_PAGE_DEF_H */

1
arch/arm64/include/asm/pci.h
View File

@ -17,6 +17,7 @@
#define pcibios_assign_all_busses() \
(pci_has_flag(PCI_REASSIGN_ALL_BUS))
#define arch_can_pci_mmap_wc() 1
#define ARCH_GENERIC_PCI_MMAP_RESOURCE 1
extern int isa_dma_bridge_buggy;

3
arch/arm64/include/asm/perf_event.h
View File

@ -236,6 +236,9 @@
#define ARMV8_PMU_USERENR_CR (1 << 2) /* Cycle counter can be read at EL0 */
#define ARMV8_PMU_USERENR_ER (1 << 3) /* Event counter can be read at EL0 */
/* PMMIR_EL1.SLOTS mask */
#define ARMV8_PMU_SLOTS_MASK 0xff
#ifdef CONFIG_PERF_EVENTS
struct pt_regs;
extern unsigned long perf_instruction_pointer(struct pt_regs *regs);

16
arch/arm64/include/asm/pgtable-hwdef.h
View File

@ -81,25 +81,15 @@
/*
* Contiguous page definitions.
*/
#ifdef CONFIG_ARM64_64K_PAGES
#define CONT_PTE_SHIFT (5 + PAGE_SHIFT)
#define CONT_PMD_SHIFT (5 + PMD_SHIFT)
#elif defined(CONFIG_ARM64_16K_PAGES)
#define CONT_PTE_SHIFT (7 + PAGE_SHIFT)
#define CONT_PMD_SHIFT (5 + PMD_SHIFT)
#else
#define CONT_PTE_SHIFT (4 + PAGE_SHIFT)
#define CONT_PMD_SHIFT (4 + PMD_SHIFT)
#endif
#define CONT_PTE_SHIFT (CONFIG_ARM64_CONT_PTE_SHIFT + PAGE_SHIFT)
#define CONT_PTES (1 << (CONT_PTE_SHIFT - PAGE_SHIFT))
#define CONT_PTE_SIZE (CONT_PTES * PAGE_SIZE)
#define CONT_PTE_MASK (~(CONT_PTE_SIZE - 1))
#define CONT_PMD_SHIFT (CONFIG_ARM64_CONT_PMD_SHIFT + PMD_SHIFT)
#define CONT_PMDS (1 << (CONT_PMD_SHIFT - PMD_SHIFT))
#define CONT_PMD_SIZE (CONT_PMDS * PMD_SIZE)
#define CONT_PMD_MASK (~(CONT_PMD_SIZE - 1))
/* the numerical offset of the PTE within a range of CONT_PTES */
#define CONT_RANGE_OFFSET(addr) (((addr)>>PAGE_SHIFT)&(CONT_PTES-1))
/*
* Hardware page table definitions.

7
arch/arm64/include/asm/pgtable-prot.h
View File

@ -19,6 +19,13 @@
#define PTE_DEVMAP (_AT(pteval_t, 1) << 57)
#define PTE_PROT_NONE (_AT(pteval_t, 1) << 58) /* only when !PTE_VALID */
/*
* This bit indicates that the entry is present i.e. pmd_page()
* still points to a valid huge page in memory even if the pmd
* has been invalidated.
*/
#define PMD_PRESENT_INVALID (_AT(pteval_t, 1) << 59) /* only when !PMD_SECT_VALID */
#ifndef __ASSEMBLY__
#include <asm/cpufeature.h>

64
arch/arm64/include/asm/pgtable.h
View File

@ -35,11 +35,6 @@
extern struct page *vmemmap;
extern void __pte_error(const char *file, int line, unsigned long val);
extern void __pmd_error(const char *file, int line, unsigned long val);
extern void __pud_error(const char *file, int line, unsigned long val);
extern void __pgd_error(const char *file, int line, unsigned long val);
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
#define __HAVE_ARCH_FLUSH_PMD_TLB_RANGE
@ -50,6 +45,14 @@ extern void __pgd_error(const char *file, int line, unsigned long val);
__flush_tlb_range(vma, addr, end, PUD_SIZE, false, 1)
#endif /* CONFIG_TRANSPARENT_HUGEPAGE */
/*
* Outside of a few very special situations (e.g. hibernation), we always
* use broadcast TLB invalidation instructions, therefore a spurious page
* fault on one CPU which has been handled concurrently by another CPU
* does not need to perform additional invalidation.
*/
#define flush_tlb_fix_spurious_fault(vma, address) do { } while (0)
/*
* ZERO_PAGE is a global shared page that is always zero: used
* for zero-mapped memory areas etc..
@ -57,7 +60,8 @@ extern void __pgd_error(const char *file, int line, unsigned long val);
extern unsigned long empty_zero_page[PAGE_SIZE / sizeof(unsigned long)];
#define ZERO_PAGE(vaddr) phys_to_page(__pa_symbol(empty_zero_page))
#define pte_ERROR(pte) __pte_error(__FILE__, __LINE__, pte_val(pte))
#define pte_ERROR(e) \
pr_err("%s:%d: bad pte %016llx.\n", __FILE__, __LINE__, pte_val(e))
/*
* Macros to convert between a physical address and its placement in a
@ -145,6 +149,18 @@ static inline pte_t set_pte_bit(pte_t pte, pgprot_t prot)
return pte;
}
static inline pmd_t clear_pmd_bit(pmd_t pmd, pgprot_t prot)
{
pmd_val(pmd) &= ~pgprot_val(prot);
return pmd;
}
static inline pmd_t set_pmd_bit(pmd_t pmd, pgprot_t prot)
{
pmd_val(pmd) |= pgprot_val(prot);
return pmd;
}
static inline pte_t pte_wrprotect(pte_t pte)
{
pte = clear_pte_bit(pte, __pgprot(PTE_WRITE));
@ -363,15 +379,24 @@ static inline int pmd_protnone(pmd_t pmd)
}
#endif
#define pmd_present_invalid(pmd) (!!(pmd_val(pmd) & PMD_PRESENT_INVALID))
static inline int pmd_present(pmd_t pmd)
{
return pte_present(pmd_pte(pmd)) || pmd_present_invalid(pmd);
}
/*
* THP definitions.
*/
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
#define pmd_trans_huge(pmd) (pmd_val(pmd) && !(pmd_val(pmd) & PMD_TABLE_BIT))
static inline int pmd_trans_huge(pmd_t pmd)
{
return pmd_val(pmd) && pmd_present(pmd) && !(pmd_val(pmd) & PMD_TABLE_BIT);
}
#endif /* CONFIG_TRANSPARENT_HUGEPAGE */
#define pmd_present(pmd) pte_present(pmd_pte(pmd))
#define pmd_dirty(pmd) pte_dirty(pmd_pte(pmd))
#define pmd_young(pmd) pte_young(pmd_pte(pmd))
#define pmd_valid(pmd) pte_valid(pmd_pte(pmd))
@ -381,7 +406,14 @@ static inline int pmd_protnone(pmd_t pmd)
#define pmd_mkclean(pmd) pte_pmd(pte_mkclean(pmd_pte(pmd)))
#define pmd_mkdirty(pmd) pte_pmd(pte_mkdirty(pmd_pte(pmd)))
#define pmd_mkyoung(pmd) pte_pmd(pte_mkyoung(pmd_pte(pmd)))
#define pmd_mkinvalid(pmd) (__pmd(pmd_val(pmd) & ~PMD_SECT_VALID))
static inline pmd_t pmd_mkinvalid(pmd_t pmd)
{
pmd = set_pmd_bit(pmd, __pgprot(PMD_PRESENT_INVALID));
pmd = clear_pmd_bit(pmd, __pgprot(PMD_SECT_VALID));
return pmd;
}
#define pmd_thp_or_huge(pmd) (pmd_huge(pmd) || pmd_trans_huge(pmd))
@ -541,7 +573,8 @@ static inline unsigned long pmd_page_vaddr(pmd_t pmd)
#if CONFIG_PGTABLE_LEVELS > 2
#define pmd_ERROR(pmd) __pmd_error(__FILE__, __LINE__, pmd_val(pmd))
#define pmd_ERROR(e) \
pr_err("%s:%d: bad pmd %016llx.\n", __FILE__, __LINE__, pmd_val(e))
#define pud_none(pud) (!pud_val(pud))
#define pud_bad(pud) (!(pud_val(pud) & PUD_TABLE_BIT))
@ -608,7 +641,8 @@ static inline unsigned long pud_page_vaddr(pud_t pud)
#if CONFIG_PGTABLE_LEVELS > 3
#define pud_ERROR(pud) __pud_error(__FILE__, __LINE__, pud_val(pud))
#define pud_ERROR(e) \
pr_err("%s:%d: bad pud %016llx.\n", __FILE__, __LINE__, pud_val(e))
#define p4d_none(p4d) (!p4d_val(p4d))
#define p4d_bad(p4d) (!(p4d_val(p4d) & 2))
@ -667,7 +701,8 @@ static inline unsigned long p4d_page_vaddr(p4d_t p4d)
#endif /* CONFIG_PGTABLE_LEVELS > 3 */
#define pgd_ERROR(pgd) __pgd_error(__FILE__, __LINE__, pgd_val(pgd))
#define pgd_ERROR(e) \
pr_err("%s:%d: bad pgd %016llx.\n", __FILE__, __LINE__, pgd_val(e))
#define pgd_set_fixmap(addr) ((pgd_t *)set_fixmap_offset(FIX_PGD, addr))
#define pgd_clear_fixmap() clear_fixmap(FIX_PGD)
@ -847,6 +882,11 @@ static inline pmd_t pmdp_establish(struct vm_area_struct *vma,
#define __pte_to_swp_entry(pte) ((swp_entry_t) { pte_val(pte) })
#define __swp_entry_to_pte(swp) ((pte_t) { (swp).val })
#ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION
#define __pmd_to_swp_entry(pmd) ((swp_entry_t) { pmd_val(pmd) })
#define __swp_entry_to_pmd(swp) __pmd((swp).val)
#endif /* CONFIG_ARCH_ENABLE_THP_MIGRATION */
/*
* Ensure that there are not more swap files than can be encoded in the kernel
* PTEs.

2
arch/arm64/include/asm/stacktrace.h
View File

@ -63,7 +63,7 @@ struct stackframe {
extern int unwind_frame(struct task_struct *tsk, struct stackframe *frame);
extern void walk_stackframe(struct task_struct *tsk, struct stackframe *frame,
int (*fn)(struct stackframe *, void *), void *data);
bool (*fn)(void *, unsigned long), void *data);
extern void dump_backtrace(struct pt_regs *regs, struct task_struct *tsk,
const char *loglvl);

26
arch/arm64/include/asm/sysreg.h
View File

@ -321,6 +321,8 @@
#define SYS_PMINTENSET_EL1 sys_reg(3, 0, 9, 14, 1)
#define SYS_PMINTENCLR_EL1 sys_reg(3, 0, 9, 14, 2)
#define SYS_PMMIR_EL1 sys_reg(3, 0, 9, 14, 6)
#define SYS_MAIR_EL1 sys_reg(3, 0, 10, 2, 0)
#define SYS_AMAIR_EL1 sys_reg(3, 0, 10, 3, 0)
@ -636,14 +638,22 @@
#define ID_AA64ISAR1_APA_SHIFT 4
#define ID_AA64ISAR1_DPB_SHIFT 0
#define ID_AA64ISAR1_APA_NI 0x0
#define ID_AA64ISAR1_APA_ARCHITECTED 0x1
#define ID_AA64ISAR1_API_NI 0x0
#define ID_AA64ISAR1_API_IMP_DEF 0x1
#define ID_AA64ISAR1_GPA_NI 0x0
#define ID_AA64ISAR1_GPA_ARCHITECTED 0x1
#define ID_AA64ISAR1_GPI_NI 0x0
#define ID_AA64ISAR1_GPI_IMP_DEF 0x1
#define ID_AA64ISAR1_APA_NI 0x0
#define ID_AA64ISAR1_APA_ARCHITECTED 0x1
#define ID_AA64ISAR1_APA_ARCH_EPAC 0x2
#define ID_AA64ISAR1_APA_ARCH_EPAC2 0x3
#define ID_AA64ISAR1_APA_ARCH_EPAC2_FPAC 0x4
#define ID_AA64ISAR1_APA_ARCH_EPAC2_FPAC_CMB 0x5
#define ID_AA64ISAR1_API_NI 0x0
#define ID_AA64ISAR1_API_IMP_DEF 0x1
#define ID_AA64ISAR1_API_IMP_DEF_EPAC 0x2
#define ID_AA64ISAR1_API_IMP_DEF_EPAC2 0x3
#define ID_AA64ISAR1_API_IMP_DEF_EPAC2_FPAC 0x4
#define ID_AA64ISAR1_API_IMP_DEF_EPAC2_FPAC_CMB 0x5
#define ID_AA64ISAR1_GPA_NI 0x0
#define ID_AA64ISAR1_GPA_ARCHITECTED 0x1
#define ID_AA64ISAR1_GPI_NI 0x0
#define ID_AA64ISAR1_GPI_IMP_DEF 0x1
/* id_aa64pfr0 */
#define ID_AA64PFR0_CSV3_SHIFT 60

2
arch/arm64/include/asm/traps.h
View File

@ -24,7 +24,7 @@ struct undef_hook {
void register_undef_hook(struct undef_hook *hook);
void unregister_undef_hook(struct undef_hook *hook);
void force_signal_inject(int signal, int code, unsigned long address);
void force_signal_inject(int signal, int code, unsigned long address, unsigned int err);
void arm64_notify_segfault(unsigned long addr);
void arm64_force_sig_fault(int signo, int code, void __user *addr, const char *str);
void arm64_force_sig_mceerr(int code, void __user *addr, short lsb, const char *str);

2
arch/arm64/kernel/Makefile
View File

@ -3,8 +3,6 @@
# Makefile for the linux kernel.
#
CPPFLAGS_vmlinux.lds := -DTEXT_OFFSET=$(TEXT_OFFSET)
AFLAGS_head.o := -DTEXT_OFFSET=$(TEXT_OFFSET)
CFLAGS_armv8_deprecated.o := -I$(src)
CFLAGS_REMOVE_ftrace.o = $(CC_FLAGS_FTRACE)

4
arch/arm64/kernel/cpu-reset.S
View File

@ -35,6 +35,10 @@ SYM_CODE_START(__cpu_soft_restart)
mov_q x13, SCTLR_ELx_FLAGS
bic x12, x12, x13
pre_disable_mmu_workaround
/*
* either disable EL1&0 translation regime or disable EL2&0 translation
* regime if HCR_EL2.E2H == 1
*/
msr sctlr_el1, x12
isb

2
arch/arm64/kernel/cpu_errata.c
View File

@ -169,8 +169,6 @@ static void install_bp_hardening_cb(bp_hardening_cb_t fn,
}
#endif /* CONFIG_KVM_INDIRECT_VECTORS */
#include <linux/arm-smccc.h>
static void __maybe_unused call_smc_arch_workaround_1(void)
{
arm_smccc_1_1_smc(ARM_SMCCC_ARCH_WORKAROUND_1, NULL);

46
arch/arm64/kernel/cpufeature.c
View File

@ -197,9 +197,9 @@ static const struct arm64_ftr_bits ftr_id_aa64isar1[] = {
ARM64_FTR_BITS(FTR_VISIBLE, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64ISAR1_FCMA_SHIFT, 4, 0),
ARM64_FTR_BITS(FTR_VISIBLE, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64ISAR1_JSCVT_SHIFT, 4, 0),
ARM64_FTR_BITS(FTR_VISIBLE_IF_IS_ENABLED(CONFIG_ARM64_PTR_AUTH),
FTR_STRICT, FTR_LOWER_SAFE, ID_AA64ISAR1_API_SHIFT, 4, 0),
FTR_STRICT, FTR_EXACT, ID_AA64ISAR1_API_SHIFT, 4, 0),
ARM64_FTR_BITS(FTR_VISIBLE_IF_IS_ENABLED(CONFIG_ARM64_PTR_AUTH),
FTR_STRICT, FTR_LOWER_SAFE, ID_AA64ISAR1_APA_SHIFT, 4, 0),
FTR_STRICT, FTR_EXACT, ID_AA64ISAR1_APA_SHIFT, 4, 0),
ARM64_FTR_BITS(FTR_VISIBLE, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64ISAR1_DPB_SHIFT, 4, 0),
ARM64_FTR_END,
};
@ -1111,6 +1111,7 @@ u64 read_sanitised_ftr_reg(u32 id)
return 0;
return regp->sys_val;
}
EXPORT_SYMBOL_GPL(read_sanitised_ftr_reg);
#define read_sysreg_case(r) \
case r: return read_sysreg_s(r)
@ -1443,6 +1444,7 @@ static inline void __cpu_enable_hw_dbm(void)
write_sysreg(tcr, tcr_el1);
isb();
local_flush_tlb_all();
}
static bool cpu_has_broken_dbm(void)
@ -1648,11 +1650,37 @@ static void cpu_clear_disr(const struct arm64_cpu_capabilities *__unused)
#endif /* CONFIG_ARM64_RAS_EXTN */
#ifdef CONFIG_ARM64_PTR_AUTH
static bool has_address_auth(const struct arm64_cpu_capabilities *entry,
int __unused)
static bool has_address_auth_cpucap(const struct arm64_cpu_capabilities *entry, int scope)
{
return __system_matches_cap(ARM64_HAS_ADDRESS_AUTH_ARCH) ||
__system_matches_cap(ARM64_HAS_ADDRESS_AUTH_IMP_DEF);
int boot_val, sec_val;
/* We don't expect to be called with SCOPE_SYSTEM */
WARN_ON(scope == SCOPE_SYSTEM);
/*
* The ptr-auth feature levels are not intercompatible with lower
* levels. Hence we must match ptr-auth feature level of the secondary
* CPUs with that of the boot CPU. The level of boot cpu is fetched
* from the sanitised register whereas direct register read is done for
* the secondary CPUs.
* The sanitised feature state is guaranteed to match that of the
* boot CPU as a mismatched secondary CPU is parked before it gets
* a chance to update the state, with the capability.
*/
boot_val = cpuid_feature_extract_field(read_sanitised_ftr_reg(entry->sys_reg),
entry->field_pos, entry->sign);
if (scope & SCOPE_BOOT_CPU)
return boot_val >= entry->min_field_value;
/* Now check for the secondary CPUs with SCOPE_LOCAL_CPU scope */
sec_val = cpuid_feature_extract_field(__read_sysreg_by_encoding(entry->sys_reg),
entry->field_pos, entry->sign);
return sec_val == boot_val;
}
static bool has_address_auth_metacap(const struct arm64_cpu_capabilities *entry,
int scope)
{
return has_address_auth_cpucap(cpu_hwcaps_ptrs[ARM64_HAS_ADDRESS_AUTH_ARCH], scope) ||
has_address_auth_cpucap(cpu_hwcaps_ptrs[ARM64_HAS_ADDRESS_AUTH_IMP_DEF], scope);
}
static bool has_generic_auth(const struct arm64_cpu_capabilities *entry,
@ -2021,7 +2049,7 @@ static const struct arm64_cpu_capabilities arm64_features[] = {
.sign = FTR_UNSIGNED,
.field_pos = ID_AA64ISAR1_APA_SHIFT,
.min_field_value = ID_AA64ISAR1_APA_ARCHITECTED,
.matches = has_cpuid_feature,
.matches = has_address_auth_cpucap,
},
{
.desc = "Address authentication (IMP DEF algorithm)",
@ -2031,12 +2059,12 @@ static const struct arm64_cpu_capabilities arm64_features[] = {
.sign = FTR_UNSIGNED,
.field_pos = ID_AA64ISAR1_API_SHIFT,
.min_field_value = ID_AA64ISAR1_API_IMP_DEF,
.matches = has_cpuid_feature,
.matches = has_address_auth_cpucap,
},
{
.capability = ARM64_HAS_ADDRESS_AUTH,
.type = ARM64_CPUCAP_BOOT_CPU_FEATURE,
.matches = has_address_auth,
.matches = has_address_auth_metacap,
},
{
.desc = "Generic authentication (architected algorithm)",

179
arch/arm64/kernel/cpuinfo.c
View File

@ -43,94 +43,92 @@ static const char *icache_policy_str[] = {
unsigned long __icache_flags;
static const char *const hwcap_str[] = {
"fp",
"asimd",
"evtstrm",
"aes",
"pmull",
"sha1",
"sha2",
"crc32",
"atomics",
"fphp",
"asimdhp",
"cpuid",
"asimdrdm",
"jscvt",
"fcma",
"lrcpc",
"dcpop",
"sha3",
"sm3",
"sm4",
"asimddp",
"sha512",
"sve",
"asimdfhm",
"dit",
"uscat",
"ilrcpc",
"flagm",
"ssbs",
"sb",
"paca",
"pacg",
"dcpodp",
"sve2",
"sveaes",
"svepmull",
"svebitperm",
"svesha3",
"svesm4",
"flagm2",
"frint",
"svei8mm",
"svef32mm",
"svef64mm",
"svebf16",
"i8mm",
"bf16",
"dgh",
"rng",
"bti",
/* reserved for "mte" */
NULL
[KERNEL_HWCAP_FP] = "fp",
[KERNEL_HWCAP_ASIMD] = "asimd",
[KERNEL_HWCAP_EVTSTRM] = "evtstrm",
[KERNEL_HWCAP_AES] = "aes",
[KERNEL_HWCAP_PMULL] = "pmull",
[KERNEL_HWCAP_SHA1] = "sha1",
[KERNEL_HWCAP_SHA2] = "sha2",
[KERNEL_HWCAP_CRC32] = "crc32",
[KERNEL_HWCAP_ATOMICS] = "atomics",
[KERNEL_HWCAP_FPHP] = "fphp",
[KERNEL_HWCAP_ASIMDHP] = "asimdhp",
[KERNEL_HWCAP_CPUID] = "cpuid",
[KERNEL_HWCAP_ASIMDRDM] = "asimdrdm",
[KERNEL_HWCAP_JSCVT] = "jscvt",
[KERNEL_HWCAP_FCMA] = "fcma",
[KERNEL_HWCAP_LRCPC] = "lrcpc",
[KERNEL_HWCAP_DCPOP] = "dcpop",
[KERNEL_HWCAP_SHA3] = "sha3",
[KERNEL_HWCAP_SM3] = "sm3",
[KERNEL_HWCAP_SM4] = "sm4",
[KERNEL_HWCAP_ASIMDDP] = "asimddp",
[KERNEL_HWCAP_SHA512] = "sha512",
[KERNEL_HWCAP_SVE] = "sve",
[KERNEL_HWCAP_ASIMDFHM] = "asimdfhm",
[KERNEL_HWCAP_DIT] = "dit",
[KERNEL_HWCAP_USCAT] = "uscat",
[KERNEL_HWCAP_ILRCPC] = "ilrcpc",
[KERNEL_HWCAP_FLAGM] = "flagm",
[KERNEL_HWCAP_SSBS] = "ssbs",
[KERNEL_HWCAP_SB] = "sb",
[KERNEL_HWCAP_PACA] = "paca",
[KERNEL_HWCAP_PACG] = "pacg",
[KERNEL_HWCAP_DCPODP] = "dcpodp",
[KERNEL_HWCAP_SVE2] = "sve2",
[KERNEL_HWCAP_SVEAES] = "sveaes",
[KERNEL_HWCAP_SVEPMULL] = "svepmull",
[KERNEL_HWCAP_SVEBITPERM] = "svebitperm",
[KERNEL_HWCAP_SVESHA3] = "svesha3",
[KERNEL_HWCAP_SVESM4] = "svesm4",
[KERNEL_HWCAP_FLAGM2] = "flagm2",
[KERNEL_HWCAP_FRINT] = "frint",
[KERNEL_HWCAP_SVEI8MM] = "svei8mm",
[KERNEL_HWCAP_SVEF32MM] = "svef32mm",
[KERNEL_HWCAP_SVEF64MM] = "svef64mm",
[KERNEL_HWCAP_SVEBF16] = "svebf16",
[KERNEL_HWCAP_I8MM] = "i8mm",
[KERNEL_HWCAP_BF16] = "bf16",
[KERNEL_HWCAP_DGH] = "dgh",
[KERNEL_HWCAP_RNG] = "rng",
[KERNEL_HWCAP_BTI] = "bti",
};
#ifdef CONFIG_COMPAT
#define COMPAT_KERNEL_HWCAP(x) const_ilog2(COMPAT_HWCAP_ ## x)
static const char *const compat_hwcap_str[] = {
"swp",
"half",
"thumb",
"26bit",
"fastmult",
"fpa",
"vfp",
"edsp",
"java",
"iwmmxt",
"crunch",
"thumbee",
"neon",
"vfpv3",
"vfpv3d16",
"tls",
"vfpv4",
"idiva",
"idivt",
"vfpd32",
"lpae",
"evtstrm",
NULL
[COMPAT_KERNEL_HWCAP(SWP)] = "swp",
[COMPAT_KERNEL_HWCAP(HALF)] = "half",
[COMPAT_KERNEL_HWCAP(THUMB)] = "thumb",
[COMPAT_KERNEL_HWCAP(26BIT)] = NULL, /* Not possible on arm64 */
[COMPAT_KERNEL_HWCAP(FAST_MULT)] = "fastmult",
[COMPAT_KERNEL_HWCAP(FPA)] = NULL, /* Not possible on arm64 */
[COMPAT_KERNEL_HWCAP(VFP)] = "vfp",
[COMPAT_KERNEL_HWCAP(EDSP)] = "edsp",
[COMPAT_KERNEL_HWCAP(JAVA)] = NULL, /* Not possible on arm64 */
[COMPAT_KERNEL_HWCAP(IWMMXT)] = NULL, /* Not possible on arm64 */
[COMPAT_KERNEL_HWCAP(CRUNCH)] = NULL, /* Not possible on arm64 */
[COMPAT_KERNEL_HWCAP(THUMBEE)] = NULL, /* Not possible on arm64 */
[COMPAT_KERNEL_HWCAP(NEON)] = "neon",
[COMPAT_KERNEL_HWCAP(VFPv3)] = "vfpv3",
[COMPAT_KERNEL_HWCAP(VFPV3D16)] = NULL, /* Not possible on arm64 */
[COMPAT_KERNEL_HWCAP(TLS)] = "tls",
[COMPAT_KERNEL_HWCAP(VFPv4)] = "vfpv4",
[COMPAT_KERNEL_HWCAP(IDIVA)] = "idiva",
[COMPAT_KERNEL_HWCAP(IDIVT)] = "idivt",
[COMPAT_KERNEL_HWCAP(VFPD32)] = NULL, /* Not possible on arm64 */
[COMPAT_KERNEL_HWCAP(LPAE)] = "lpae",
[COMPAT_KERNEL_HWCAP(EVTSTRM)] = "evtstrm",
};
#define COMPAT_KERNEL_HWCAP2(x) const_ilog2(COMPAT_HWCAP2_ ## x)
static const char *const compat_hwcap2_str[] = {
"aes",
"pmull",
"sha1",
"sha2",
"crc32",
NULL
[COMPAT_KERNEL_HWCAP2(AES)] = "aes",
[COMPAT_KERNEL_HWCAP2(PMULL)] = "pmull",
[COMPAT_KERNEL_HWCAP2(SHA1)] = "sha1",
[COMPAT_KERNEL_HWCAP2(SHA2)] = "sha2",
[COMPAT_KERNEL_HWCAP2(CRC32)] = "crc32",
};
#endif /* CONFIG_COMPAT */
@ -166,16 +164,25 @@ static int c_show(struct seq_file *m, void *v)
seq_puts(m, "Features\t:");
if (compat) {
#ifdef CONFIG_COMPAT
for (j = 0; compat_hwcap_str[j]; j++)
if (compat_elf_hwcap & (1 << j))
seq_printf(m, " %s", compat_hwcap_str[j]);
for (j = 0; j < ARRAY_SIZE(compat_hwcap_str); j++) {
if (compat_elf_hwcap & (1 << j)) {
/*
* Warn once if any feature should not
* have been present on arm64 platform.
*/
if (WARN_ON_ONCE(!compat_hwcap_str[j]))
continue;
for (j = 0; compat_hwcap2_str[j]; j++)
seq_printf(m, " %s", compat_hwcap_str[j]);
}
}
for (j = 0; j < ARRAY_SIZE(compat_hwcap2_str); j++)
if (compat_elf_hwcap2 & (1 << j))
seq_printf(m, " %s", compat_hwcap2_str[j]);
#endif /* CONFIG_COMPAT */
} else {
for (j = 0; hwcap_str[j]; j++)
for (j = 0; j < ARRAY_SIZE(hwcap_str); j++)
if (cpu_have_feature(j))
seq_printf(m, " %s", hwcap_str[j]);
}

2
arch/arm64/kernel/debug-monitors.c
View File

@ -384,7 +384,7 @@ void __init debug_traps_init(void)
hook_debug_fault_code(DBG_ESR_EVT_HWSS, single_step_handler, SIGTRAP,
TRAP_TRACE, "single-step handler");
hook_debug_fault_code(DBG_ESR_EVT_BRK, brk_handler, SIGTRAP,
TRAP_BRKPT, "ptrace BRK handler");
TRAP_BRKPT, "BRK handler");
}
/* Re-enable single step for syscall restarting. */

21
arch/arm64/kernel/entry-common.c
View File

@ -66,6 +66,13 @@ static void notrace el1_dbg(struct pt_regs *regs, unsigned long esr)
}
NOKPROBE_SYMBOL(el1_dbg);
static void notrace el1_fpac(struct pt_regs *regs, unsigned long esr)
{
local_daif_inherit(regs);
do_ptrauth_fault(regs, esr);
}
NOKPROBE_SYMBOL(el1_fpac);
asmlinkage void notrace el1_sync_handler(struct pt_regs *regs)
{
unsigned long esr = read_sysreg(esr_el1);
@ -92,6 +99,9 @@ asmlinkage void notrace el1_sync_handler(struct pt_regs *regs)
case ESR_ELx_EC_BRK64:
el1_dbg(regs, esr);
break;
case ESR_ELx_EC_FPAC:
el1_fpac(regs, esr);
break;
default:
el1_inv(regs, esr);
}
@ -227,6 +237,14 @@ static void notrace el0_svc(struct pt_regs *regs)
}
NOKPROBE_SYMBOL(el0_svc);
static void notrace el0_fpac(struct pt_regs *regs, unsigned long esr)
{
user_exit_irqoff();
local_daif_restore(DAIF_PROCCTX);
do_ptrauth_fault(regs, esr);
}
NOKPROBE_SYMBOL(el0_fpac);
asmlinkage void notrace el0_sync_handler(struct pt_regs *regs)
{
unsigned long esr = read_sysreg(esr_el1);
@ -272,6 +290,9 @@ asmlinkage void notrace el0_sync_handler(struct pt_regs *regs)
case ESR_ELx_EC_BRK64:
el0_dbg(regs, esr);
break;
case ESR_ELx_EC_FPAC:
el0_fpac(regs, esr);
break;
default:
el0_inv(regs, esr);
}

25
arch/arm64/kernel/entry-fpsimd.S
View File

@ -32,6 +32,7 @@ SYM_FUNC_START(fpsimd_load_state)
SYM_FUNC_END(fpsimd_load_state)
#ifdef CONFIG_ARM64_SVE
SYM_FUNC_START(sve_save_state)
sve_save 0, x1, 2
ret
@ -46,4 +47,28 @@ SYM_FUNC_START(sve_get_vl)
_sve_rdvl 0, 1
ret
SYM_FUNC_END(sve_get_vl)
/*
* Load SVE state from FPSIMD state.
*
* x0 = pointer to struct fpsimd_state
* x1 = VQ - 1
*
* Each SVE vector will be loaded with the first 128-bits taken from FPSIMD
* and the rest zeroed. All the other SVE registers will be zeroed.
*/
SYM_FUNC_START(sve_load_from_fpsimd_state)
sve_load_vq x1, x2, x3
fpsimd_restore x0, 8
_for n, 0, 15, _sve_pfalse \n
_sve_wrffr 0
ret
SYM_FUNC_END(sve_load_from_fpsimd_state)
/* Zero all SVE registers but the first 128-bits of each vector */
SYM_FUNC_START(sve_flush_live)
sve_flush
ret
SYM_FUNC_END(sve_flush_live)
#endif /* CONFIG_ARM64_SVE */

8
arch/arm64/kernel/fpsimd.c
View File

@ -32,9 +32,11 @@
#include <linux/swab.h>
#include <asm/esr.h>
#include <asm/exception.h>
#include <asm/fpsimd.h>
#include <asm/cpufeature.h>
#include <asm/cputype.h>
#include <asm/neon.h>
#include <asm/processor.h>
#include <asm/simd.h>
#include <asm/sigcontext.h>
@ -312,7 +314,7 @@ static void fpsimd_save(void)
* re-enter user with corrupt state.
* There's no way to recover, so kill it:
*/
force_signal_inject(SIGKILL, SI_KERNEL, 0);
force_signal_inject(SIGKILL, SI_KERNEL, 0, 0);
return;
}
@ -928,7 +930,7 @@ void fpsimd_release_task(struct task_struct *dead_task)
* the SVE access trap will be disabled the next time this task
* reaches ret_to_user.
*
* TIF_SVE should be clear on entry: otherwise, task_fpsimd_load()
* TIF_SVE should be clear on entry: otherwise, fpsimd_restore_current_state()
* would have disabled the SVE access trap for userspace during
* ret_to_user, making an SVE access trap impossible in that case.
*/
@ -936,7 +938,7 @@ void do_sve_acc(unsigned int esr, struct pt_regs *regs)
{
/* Even if we chose not to use SVE, the hardware could still trap: */
if (unlikely(!system_supports_sve()) || WARN_ON(is_compat_task())) {
force_signal_inject(SIGILL, ILL_ILLOPC, regs->pc);
force_signal_inject(SIGILL, ILL_ILLOPC, regs->pc, 0);
return;
}

16
arch/arm64/kernel/head.S
View File

@ -36,14 +36,10 @@
#include "efi-header.S"
#define __PHYS_OFFSET (KERNEL_START - TEXT_OFFSET)
#define __PHYS_OFFSET KERNEL_START
#if (TEXT_OFFSET & 0xfff) != 0
#error TEXT_OFFSET must be at least 4KB aligned
#elif (PAGE_OFFSET & 0x1fffff) != 0
#if (PAGE_OFFSET & 0x1fffff) != 0
#error PAGE_OFFSET must be at least 2MB aligned
#elif TEXT_OFFSET > 0x1fffff
#error TEXT_OFFSET must be less than 2MB
#endif
/*
@ -55,7 +51,7 @@
* x0 = physical address to the FDT blob.
*
* This code is mostly position independent so you call this at
* __pa(PAGE_OFFSET + TEXT_OFFSET).
* __pa(PAGE_OFFSET).
*
* Note that the callee-saved registers are used for storing variables
* that are useful before the MMU is enabled. The allocations are described
@ -77,7 +73,7 @@ _head:
b primary_entry // branch to kernel start, magic
.long 0 // reserved
#endif
le64sym _kernel_offset_le // Image load offset from start of RAM, little-endian
.quad 0 // Image load offset from start of RAM, little-endian
le64sym _kernel_size_le // Effective size of kernel image, little-endian
le64sym _kernel_flags_le // Informative flags, little-endian
.quad 0 // reserved
@ -382,7 +378,7 @@ SYM_FUNC_START_LOCAL(__create_page_tables)
* Map the kernel image (starting with PHYS_OFFSET).
*/
adrp x0, init_pg_dir
mov_q x5, KIMAGE_VADDR + TEXT_OFFSET // compile time __va(_text)
mov_q x5, KIMAGE_VADDR // compile time __va(_text)
add x5, x5, x23 // add KASLR displacement
mov x4, PTRS_PER_PGD
adrp x6, _end // runtime __pa(_end)
@ -474,7 +470,7 @@ SYM_FUNC_END(__primary_switched)
.pushsection ".rodata", "a"
SYM_DATA_START(kimage_vaddr)
.quad _text - TEXT_OFFSET
.quad _text
SYM_DATA_END(kimage_vaddr)
EXPORT_SYMBOL(kimage_vaddr)
.popsection

1
arch/arm64/kernel/hibernate.c
View File

@ -21,7 +21,6 @@
#include <linux/sched.h>
#include <linux/suspend.h>
#include <linux/utsname.h>
#include <linux/version.h>
#include <asm/barrier.h>
#include <asm/cacheflush.h>

1
arch/arm64/kernel/image.h
View File

@ -62,7 +62,6 @@
*/
#define HEAD_SYMBOLS \
DEFINE_IMAGE_LE64(_kernel_size_le, _end - _text); \
DEFINE_IMAGE_LE64(_kernel_offset_le, TEXT_OFFSET); \
DEFINE_IMAGE_LE64(_kernel_flags_le, __HEAD_FLAGS);
#endif /* __ARM64_KERNEL_IMAGE_H */

11
arch/arm64/kernel/insn.c
View File

@ -60,16 +60,10 @@ bool __kprobes aarch64_insn_is_steppable_hint(u32 insn)
case AARCH64_INSN_HINT_XPACLRI:
case AARCH64_INSN_HINT_PACIA_1716:
case AARCH64_INSN_HINT_PACIB_1716:
case AARCH64_INSN_HINT_AUTIA_1716:
case AARCH64_INSN_HINT_AUTIB_1716:
case AARCH64_INSN_HINT_PACIAZ:
case AARCH64_INSN_HINT_PACIASP:
case AARCH64_INSN_HINT_PACIBZ:
case AARCH64_INSN_HINT_PACIBSP:
case AARCH64_INSN_HINT_AUTIAZ:
case AARCH64_INSN_HINT_AUTIASP:
case AARCH64_INSN_HINT_AUTIBZ:
case AARCH64_INSN_HINT_AUTIBSP:
case AARCH64_INSN_HINT_BTI:
case AARCH64_INSN_HINT_BTIC:
case AARCH64_INSN_HINT_BTIJ:
@ -176,7 +170,7 @@ bool __kprobes aarch64_insn_uses_literal(u32 insn)
bool __kprobes aarch64_insn_is_branch(u32 insn)
{
/* b, bl, cb*, tb*, b.cond, br, blr */
/* b, bl, cb*, tb*, ret*, b.cond, br*, blr* */
return aarch64_insn_is_b(insn) ||
aarch64_insn_is_bl(insn) ||
@ -185,8 +179,11 @@ bool __kprobes aarch64_insn_is_branch(u32 insn)
aarch64_insn_is_tbz(insn) ||
aarch64_insn_is_tbnz(insn) ||
aarch64_insn_is_ret(insn) ||
aarch64_insn_is_ret_auth(insn) ||
aarch64_insn_is_br(insn) ||
aarch64_insn_is_br_auth(insn) ||
aarch64_insn_is_blr(insn) ||
aarch64_insn_is_blr_auth(insn) ||
aarch64_insn_is_bcond(insn);
}

6
arch/arm64/kernel/perf_callchain.c
View File

@ -137,11 +137,11 @@ void perf_callchain_user(struct perf_callchain_entry_ctx *entry,
* whist unwinding the stackframe and is like a subroutine return so we use
* the PC.
*/
static int callchain_trace(struct stackframe *frame, void *data)
static bool callchain_trace(void *data, unsigned long pc)
{
struct perf_callchain_entry_ctx *entry = data;
perf_callchain_store(entry, frame->pc);
return 0;
perf_callchain_store(entry, pc);
return true;
}
void perf_callchain_kernel(struct perf_callchain_entry_ctx *entry,

272
arch/arm64/kernel/perf_event.c
View File

@ -69,6 +69,9 @@ static const unsigned armv8_pmuv3_perf_cache_map[PERF_COUNT_HW_CACHE_MAX]
[C(ITLB)][C(OP_READ)][C(RESULT_MISS)] = ARMV8_PMUV3_PERFCTR_L1I_TLB_REFILL,
[C(ITLB)][C(OP_READ)][C(RESULT_ACCESS)] = ARMV8_PMUV3_PERFCTR_L1I_TLB,
[C(LL)][C(OP_READ)][C(RESULT_MISS)] = ARMV8_PMUV3_PERFCTR_LL_CACHE_MISS_RD,
[C(LL)][C(OP_READ)][C(RESULT_ACCESS)] = ARMV8_PMUV3_PERFCTR_LL_CACHE_RD,
[C(BPU)][C(OP_READ)][C(RESULT_ACCESS)] = ARMV8_PMUV3_PERFCTR_BR_PRED,
[C(BPU)][C(OP_READ)][C(RESULT_MISS)] = ARMV8_PMUV3_PERFCTR_BR_MIS_PRED,
};
@ -302,13 +305,33 @@ static struct attribute_group armv8_pmuv3_format_attr_group = {
.attrs = armv8_pmuv3_format_attrs,
};
static ssize_t slots_show(struct device *dev, struct device_attribute *attr,
char *page)
{
struct pmu *pmu = dev_get_drvdata(dev);
struct arm_pmu *cpu_pmu = container_of(pmu, struct arm_pmu, pmu);
u32 slots = cpu_pmu->reg_pmmir & ARMV8_PMU_SLOTS_MASK;
return snprintf(page, PAGE_SIZE, "0x%08x\n", slots);
}
static DEVICE_ATTR_RO(slots);
static struct attribute *armv8_pmuv3_caps_attrs[] = {
&dev_attr_slots.attr,
NULL,
};
static struct attribute_group armv8_pmuv3_caps_attr_group = {
.name = "caps",
.attrs = armv8_pmuv3_caps_attrs,
};
/*
* Perf Events' indices
*/
#define ARMV8_IDX_CYCLE_COUNTER 0
#define ARMV8_IDX_COUNTER0 1
#define ARMV8_IDX_COUNTER_LAST(cpu_pmu) \
(ARMV8_IDX_CYCLE_COUNTER + cpu_pmu->num_events - 1)
/*
@ -348,6 +371,73 @@ static inline bool armv8pmu_event_is_chained(struct perf_event *event)
#define ARMV8_IDX_TO_COUNTER(x) \
(((x) - ARMV8_IDX_COUNTER0) & ARMV8_PMU_COUNTER_MASK)
/*
* This code is really good
*/
#define PMEVN_CASE(n, case_macro) \
case n: case_macro(n); break
#define PMEVN_SWITCH(x, case_macro) \
do { \
switch (x) { \
PMEVN_CASE(0, case_macro); \
PMEVN_CASE(1, case_macro); \
PMEVN_CASE(2, case_macro); \
PMEVN_CASE(3, case_macro); \
PMEVN_CASE(4, case_macro); \
PMEVN_CASE(5, case_macro); \
PMEVN_CASE(6, case_macro); \
PMEVN_CASE(7, case_macro); \
PMEVN_CASE(8, case_macro); \
PMEVN_CASE(9, case_macro); \
PMEVN_CASE(10, case_macro); \
PMEVN_CASE(11, case_macro); \
PMEVN_CASE(12, case_macro); \
PMEVN_CASE(13, case_macro); \
PMEVN_CASE(14, case_macro); \
PMEVN_CASE(15, case_macro); \
PMEVN_CASE(16, case_macro); \
PMEVN_CASE(17, case_macro); \
PMEVN_CASE(18, case_macro); \
PMEVN_CASE(19, case_macro); \
PMEVN_CASE(20, case_macro); \
PMEVN_CASE(21, case_macro); \
PMEVN_CASE(22, case_macro); \
PMEVN_CASE(23, case_macro); \
PMEVN_CASE(24, case_macro); \
PMEVN_CASE(25, case_macro); \
PMEVN_CASE(26, case_macro); \
PMEVN_CASE(27, case_macro); \
PMEVN_CASE(28, case_macro); \
PMEVN_CASE(29, case_macro); \
PMEVN_CASE(30, case_macro); \
default: WARN(1, "Invalid PMEV* index\n"); \
} \
} while (0)
#define RETURN_READ_PMEVCNTRN(n) \
return read_sysreg(pmevcntr##n##_el0)
static unsigned long read_pmevcntrn(int n)
{
PMEVN_SWITCH(n, RETURN_READ_PMEVCNTRN);
return 0;
}
#define WRITE_PMEVCNTRN(n) \
write_sysreg(val, pmevcntr##n##_el0)
static void write_pmevcntrn(int n, unsigned long val)
{
PMEVN_SWITCH(n, WRITE_PMEVCNTRN);
}
#define WRITE_PMEVTYPERN(n) \
write_sysreg(val, pmevtyper##n##_el0)
static void write_pmevtypern(int n, unsigned long val)
{
PMEVN_SWITCH(n, WRITE_PMEVTYPERN);
}
static inline u32 armv8pmu_pmcr_read(void)
{
return read_sysreg(pmcr_el0);
@ -365,28 +455,16 @@ static inline int armv8pmu_has_overflowed(u32 pmovsr)
return pmovsr & ARMV8_PMU_OVERFLOWED_MASK;
}
static inline int armv8pmu_counter_valid(struct arm_pmu *cpu_pmu, int idx)
{
return idx >= ARMV8_IDX_CYCLE_COUNTER &&
idx <= ARMV8_IDX_COUNTER_LAST(cpu_pmu);
}
static inline int armv8pmu_counter_has_overflowed(u32 pmnc, int idx)
{
return pmnc & BIT(ARMV8_IDX_TO_COUNTER(idx));
}
static inline void armv8pmu_select_counter(int idx)
static inline u32 armv8pmu_read_evcntr(int idx)
{
u32 counter = ARMV8_IDX_TO_COUNTER(idx);
write_sysreg(counter, pmselr_el0);
isb();
}
static inline u64 armv8pmu_read_evcntr(int idx)
{
armv8pmu_select_counter(idx);
return read_sysreg(pmxevcntr_el0);
return read_pmevcntrn(counter);
}
static inline u64 armv8pmu_read_hw_counter(struct perf_event *event)
@ -440,15 +518,11 @@ static u64 armv8pmu_unbias_long_counter(struct perf_event *event, u64 value)
static u64 armv8pmu_read_counter(struct perf_event *event)
{
struct arm_pmu *cpu_pmu = to_arm_pmu(event->pmu);
struct hw_perf_event *hwc = &event->hw;
int idx = hwc->idx;
u64 value = 0;
if (!armv8pmu_counter_valid(cpu_pmu, idx))
pr_err("CPU%u reading wrong counter %d\n",
smp_processor_id(), idx);
else if (idx == ARMV8_IDX_CYCLE_COUNTER)
if (idx == ARMV8_IDX_CYCLE_COUNTER)
value = read_sysreg(pmccntr_el0);
else
value = armv8pmu_read_hw_counter(event);
@ -458,8 +532,9 @@ static u64 armv8pmu_read_counter(struct perf_event *event)
static inline void armv8pmu_write_evcntr(int idx, u64 value)
{
armv8pmu_select_counter(idx);
write_sysreg(value, pmxevcntr_el0);
u32 counter = ARMV8_IDX_TO_COUNTER(idx);
write_pmevcntrn(counter, value);
}
static inline void armv8pmu_write_hw_counter(struct perf_event *event,
@ -477,16 +552,12 @@ static inline void armv8pmu_write_hw_counter(struct perf_event *event,
static void armv8pmu_write_counter(struct perf_event *event, u64 value)
{
struct arm_pmu *cpu_pmu = to_arm_pmu(event->pmu);
struct hw_perf_event *hwc = &event->hw;
int idx = hwc->idx;
value = armv8pmu_bias_long_counter(event, value);
if (!armv8pmu_counter_valid(cpu_pmu, idx))
pr_err("CPU%u writing wrong counter %d\n",
smp_processor_id(), idx);
else if (idx == ARMV8_IDX_CYCLE_COUNTER)
if (idx == ARMV8_IDX_CYCLE_COUNTER)
write_sysreg(value, pmccntr_el0);
else
armv8pmu_write_hw_counter(event, value);
@ -494,9 +565,10 @@ static void armv8pmu_write_counter(struct perf_event *event, u64 value)
static inline void armv8pmu_write_evtype(int idx, u32 val)
{
armv8pmu_select_counter(idx);
u32 counter = ARMV8_IDX_TO_COUNTER(idx);
val &= ARMV8_PMU_EVTYPE_MASK;
write_sysreg(val, pmxevtyper_el0);
write_pmevtypern(counter, val);
}
static inline void armv8pmu_write_event_type(struct perf_event *event)
@ -516,7 +588,10 @@ static inline void armv8pmu_write_event_type(struct perf_event *event)
armv8pmu_write_evtype(idx - 1, hwc->config_base);
armv8pmu_write_evtype(idx, chain_evt);
} else {
armv8pmu_write_evtype(idx, hwc->config_base);
if (idx == ARMV8_IDX_CYCLE_COUNTER)
write_sysreg(hwc->config_base, pmccfiltr_el0);
else
armv8pmu_write_evtype(idx, hwc->config_base);
}
}
@ -532,6 +607,11 @@ static u32 armv8pmu_event_cnten_mask(struct perf_event *event)
static inline void armv8pmu_enable_counter(u32 mask)
{
/*
* Make sure event configuration register writes are visible before we
* enable the counter.
* */
isb();
write_sysreg(mask, pmcntenset_el0);
}
@ -550,6 +630,11 @@ static inline void armv8pmu_enable_event_counter(struct perf_event *event)
static inline void armv8pmu_disable_counter(u32 mask)
{
write_sysreg(mask, pmcntenclr_el0);
/*
* Make sure the effects of disabling the counter are visible before we
* start configuring the event.
*/
isb();
}
static inline void armv8pmu_disable_event_counter(struct perf_event *event)
@ -606,15 +691,10 @@ static inline u32 armv8pmu_getreset_flags(void)
static void armv8pmu_enable_event(struct perf_event *event)
{
unsigned long flags;
struct arm_pmu *cpu_pmu = to_arm_pmu(event->pmu);
struct pmu_hw_events *events = this_cpu_ptr(cpu_pmu->hw_events);
/*
* Enable counter and interrupt, and set the counter to count
* the event that we're interested in.
*/
raw_spin_lock_irqsave(&events->pmu_lock, flags);
/*
* Disable counter
@ -622,7 +702,7 @@ static void armv8pmu_enable_event(struct perf_event *event)
armv8pmu_disable_event_counter(event);
/*
* Set event (if destined for PMNx counters).
* Set event.
*/
armv8pmu_write_event_type(event);
@ -635,21 +715,10 @@ static void armv8pmu_enable_event(struct perf_event *event)
* Enable counter
*/
armv8pmu_enable_event_counter(event);
raw_spin_unlock_irqrestore(&events->pmu_lock, flags);
}
static void armv8pmu_disable_event(struct perf_event *event)
{
unsigned long flags;
struct arm_pmu *cpu_pmu = to_arm_pmu(event->pmu);
struct pmu_hw_events *events = this_cpu_ptr(cpu_pmu->hw_events);
/*
* Disable counter and interrupt
*/
raw_spin_lock_irqsave(&events->pmu_lock, flags);
/*
* Disable counter
*/
@ -659,30 +728,18 @@ static void armv8pmu_disable_event(struct perf_event *event)
* Disable interrupt for this counter
*/
armv8pmu_disable_event_irq(event);
raw_spin_unlock_irqrestore(&events->pmu_lock, flags);
}
static void armv8pmu_start(struct arm_pmu *cpu_pmu)
{
unsigned long flags;
struct pmu_hw_events *events = this_cpu_ptr(cpu_pmu->hw_events);
raw_spin_lock_irqsave(&events->pmu_lock, flags);
/* Enable all counters */
armv8pmu_pmcr_write(armv8pmu_pmcr_read() | ARMV8_PMU_PMCR_E);
raw_spin_unlock_irqrestore(&events->pmu_lock, flags);
}
static void armv8pmu_stop(struct arm_pmu *cpu_pmu)
{
unsigned long flags;
struct pmu_hw_events *events = this_cpu_ptr(cpu_pmu->hw_events);
raw_spin_lock_irqsave(&events->pmu_lock, flags);
/* Disable all counters */
armv8pmu_pmcr_write(armv8pmu_pmcr_read() & ~ARMV8_PMU_PMCR_E);
raw_spin_unlock_irqrestore(&events->pmu_lock, flags);
}
static irqreturn_t armv8pmu_handle_irq(struct arm_pmu *cpu_pmu)
@ -735,20 +792,16 @@ static irqreturn_t armv8pmu_handle_irq(struct arm_pmu *cpu_pmu)
if (!armpmu_event_set_period(event))
continue;
/*
* Perf event overflow will queue the processing of the event as
* an irq_work which will be taken care of in the handling of
* IPI_IRQ_WORK.
*/
if (perf_event_overflow(event, &data, regs))
cpu_pmu->disable(event);
}
armv8pmu_start(cpu_pmu);
/*
* Handle the pending perf events.
*
* Note: this call *must* be run with interrupts disabled. For
* platforms that can have the PMU interrupts raised as an NMI, this
* will not work.
*/
irq_work_run();
return IRQ_HANDLED;
}
@ -997,6 +1050,12 @@ static void __armv8pmu_probe_pmu(void *info)
bitmap_from_arr32(cpu_pmu->pmceid_ext_bitmap,
pmceid, ARMV8_PMUV3_MAX_COMMON_EVENTS);
/* store PMMIR_EL1 register for sysfs */
if (pmuver >= ID_AA64DFR0_PMUVER_8_4 && (pmceid_raw[1] & BIT(31)))
cpu_pmu->reg_pmmir = read_cpuid(PMMIR_EL1);
else
cpu_pmu->reg_pmmir = 0;
}
static int armv8pmu_probe_pmu(struct arm_pmu *cpu_pmu)
@ -1019,7 +1078,8 @@ static int armv8pmu_probe_pmu(struct arm_pmu *cpu_pmu)
static int armv8_pmu_init(struct arm_pmu *cpu_pmu, char *name,
int (*map_event)(struct perf_event *event),
const struct attribute_group *events,
const struct attribute_group *format)
const struct attribute_group *format,
const struct attribute_group *caps)
{
int ret = armv8pmu_probe_pmu(cpu_pmu);
if (ret)
@ -1044,104 +1104,112 @@ static int armv8_pmu_init(struct arm_pmu *cpu_pmu, char *name,
events : &armv8_pmuv3_events_attr_group;
cpu_pmu->attr_groups[ARMPMU_ATTR_GROUP_FORMATS] = format ?
format : &armv8_pmuv3_format_attr_group;
cpu_pmu->attr_groups[ARMPMU_ATTR_GROUP_CAPS] = caps ?
caps : &armv8_pmuv3_caps_attr_group;
return 0;
}
static int armv8_pmu_init_nogroups(struct arm_pmu *cpu_pmu, char *name,
int (*map_event)(struct perf_event *event))
{
return armv8_pmu_init(cpu_pmu, name, map_event, NULL, NULL, NULL);
}
static int armv8_pmuv3_init(struct arm_pmu *cpu_pmu)
{
return armv8_pmu_init(cpu_pmu, "armv8_pmuv3",
armv8_pmuv3_map_event, NULL, NULL);
return armv8_pmu_init_nogroups(cpu_pmu, "armv8_pmuv3",
armv8_pmuv3_map_event);
}
static int armv8_a34_pmu_init(struct arm_pmu *cpu_pmu)
{
return armv8_pmu_init(cpu_pmu, "armv8_cortex_a34",
armv8_pmuv3_map_event, NULL, NULL);
return armv8_pmu_init_nogroups(cpu_pmu, "armv8_cortex_a34",
armv8_pmuv3_map_event);
}
static int armv8_a35_pmu_init(struct arm_pmu *cpu_pmu)
{
return armv8_pmu_init(cpu_pmu, "armv8_cortex_a35",
armv8_a53_map_event, NULL, NULL);
return armv8_pmu_init_nogroups(cpu_pmu, "armv8_cortex_a35",
armv8_a53_map_event);
}
static int armv8_a53_pmu_init(struct arm_pmu *cpu_pmu)
{
return armv8_pmu_init(cpu_pmu, "armv8_cortex_a53",
armv8_a53_map_event, NULL, NULL);
return armv8_pmu_init_nogroups(cpu_pmu, "armv8_cortex_a53",
armv8_a53_map_event);
}
static int armv8_a55_pmu_init(struct arm_pmu *cpu_pmu)
{
return armv8_pmu_init(cpu_pmu, "armv8_cortex_a55",
armv8_pmuv3_map_event, NULL, NULL);
return armv8_pmu_init_nogroups(cpu_pmu, "armv8_cortex_a55",
armv8_pmuv3_map_event);
}
static int armv8_a57_pmu_init(struct arm_pmu *cpu_pmu)
{
return armv8_pmu_init(cpu_pmu, "armv8_cortex_a57",
armv8_a57_map_event, NULL, NULL);
return armv8_pmu_init_nogroups(cpu_pmu, "armv8_cortex_a57",
armv8_a57_map_event);
}
static int armv8_a65_pmu_init(struct arm_pmu *cpu_pmu)
{
return armv8_pmu_init(cpu_pmu, "armv8_cortex_a65",
armv8_pmuv3_map_event, NULL, NULL);
return armv8_pmu_init_nogroups(cpu_pmu, "armv8_cortex_a65",
armv8_pmuv3_map_event);
}
static int armv8_a72_pmu_init(struct arm_pmu *cpu_pmu)
{
return armv8_pmu_init(cpu_pmu, "armv8_cortex_a72",
armv8_a57_map_event, NULL, NULL);
return armv8_pmu_init_nogroups(cpu_pmu, "armv8_cortex_a72",
armv8_a57_map_event);
}
static int armv8_a73_pmu_init(struct arm_pmu *cpu_pmu)
{
return armv8_pmu_init(cpu_pmu, "armv8_cortex_a73",
armv8_a73_map_event, NULL, NULL);
return armv8_pmu_init_nogroups(cpu_pmu, "armv8_cortex_a73",
armv8_a73_map_event);
}
static int armv8_a75_pmu_init(struct arm_pmu *cpu_pmu)
{
return armv8_pmu_init(cpu_pmu, "armv8_cortex_a75",
armv8_pmuv3_map_event, NULL, NULL);
return armv8_pmu_init_nogroups(cpu_pmu, "armv8_cortex_a75",
armv8_pmuv3_map_event);
}
static int armv8_a76_pmu_init(struct arm_pmu *cpu_pmu)
{
return armv8_pmu_init(cpu_pmu, "armv8_cortex_a76",
armv8_pmuv3_map_event, NULL, NULL);
return armv8_pmu_init_nogroups(cpu_pmu, "armv8_cortex_a76",
armv8_pmuv3_map_event);
}
static int armv8_a77_pmu_init(struct arm_pmu *cpu_pmu)
{
return armv8_pmu_init(cpu_pmu, "armv8_cortex_a77",
armv8_pmuv3_map_event, NULL, NULL);
return armv8_pmu_init_nogroups(cpu_pmu, "armv8_cortex_a77",
armv8_pmuv3_map_event);
}
static int armv8_e1_pmu_init(struct arm_pmu *cpu_pmu)
{
return armv8_pmu_init(cpu_pmu, "armv8_neoverse_e1",
armv8_pmuv3_map_event, NULL, NULL);
return armv8_pmu_init_nogroups(cpu_pmu, "armv8_neoverse_e1",
armv8_pmuv3_map_event);
}
static int armv8_n1_pmu_init(struct arm_pmu *cpu_pmu)
{
return armv8_pmu_init(cpu_pmu, "armv8_neoverse_n1",
armv8_pmuv3_map_event, NULL, NULL);
return armv8_pmu_init_nogroups(cpu_pmu, "armv8_neoverse_n1",
armv8_pmuv3_map_event);
}
static int armv8_thunder_pmu_init(struct arm_pmu *cpu_pmu)
{
return armv8_pmu_init(cpu_pmu, "armv8_cavium_thunder",
armv8_thunder_map_event, NULL, NULL);
return armv8_pmu_init_nogroups(cpu_pmu, "armv8_cavium_thunder",
armv8_thunder_map_event);
}
static int armv8_vulcan_pmu_init(struct arm_pmu *cpu_pmu)
{
return armv8_pmu_init(cpu_pmu, "armv8_brcm_vulcan",
armv8_vulcan_map_event, NULL, NULL);
return armv8_pmu_init_nogroups(cpu_pmu, "armv8_brcm_vulcan",
armv8_vulcan_map_event);
}
static const struct of_device_id armv8_pmu_of_device_ids[] = {

2
arch/arm64/kernel/perf_regs.c
View File

@ -16,7 +16,7 @@ u64 perf_reg_value(struct pt_regs *regs, int idx)
/*
* Our handling of compat tasks (PERF_SAMPLE_REGS_ABI_32) is weird, but
* we're stuck with it for ABI compatability reasons.
* we're stuck with it for ABI compatibility reasons.
*
* For a 32-bit consumer inspecting a 32-bit task, then it will look at
* the first 16 registers (see arch/arm/include/uapi/asm/perf_regs.h).

9
arch/arm64/kernel/probes/decode-insn.c
View File

@ -29,7 +29,8 @@ static bool __kprobes aarch64_insn_is_steppable(u32 insn)
aarch64_insn_is_msr_imm(insn) ||
aarch64_insn_is_msr_reg(insn) ||
aarch64_insn_is_exception(insn) ||
aarch64_insn_is_eret(insn))
aarch64_insn_is_eret(insn) ||
aarch64_insn_is_eret_auth(insn))
return false;
/*
@ -42,8 +43,10 @@ static bool __kprobes aarch64_insn_is_steppable(u32 insn)
!= AARCH64_INSN_SPCLREG_DAIF;
/*
* The HINT instruction is is problematic when single-stepping,
* except for the NOP case.
* The HINT instruction is steppable only if it is in whitelist
* and the rest of other such instructions are blocked for
* single stepping as they may cause exception or other
* unintended behaviour.
*/
if (aarch64_insn_is_hint(insn))
return aarch64_insn_is_steppable_hint(insn);

12
arch/arm64/kernel/relocate_kernel.S
View File

@ -36,18 +36,6 @@ SYM_CODE_START(arm64_relocate_new_kernel)
mov x14, xzr /* x14 = entry ptr */
mov x13, xzr /* x13 = copy dest */
/* Clear the sctlr_el2 flags. */
mrs x0, CurrentEL
cmp x0, #CurrentEL_EL2
b.ne 1f
mrs x0, sctlr_el2
mov_q x1, SCTLR_ELx_FLAGS
bic x0, x0, x1
pre_disable_mmu_workaround
msr sctlr_el2, x0
isb
1:
/* Check if the new image needs relocation. */
tbnz x16, IND_DONE_BIT, .Ldone

8
arch/arm64/kernel/return_address.c
View File

@ -18,16 +18,16 @@ struct return_address_data {
void *addr;
};
static int save_return_addr(struct stackframe *frame, void *d)
static bool save_return_addr(void *d, unsigned long pc)
{
struct return_address_data *data = d;
if (!data->level) {
data->addr = (void *)frame->pc;
return 1;
data->addr = (void *)pc;
return false;
} else {
--data->level;
return 0;
return true;
}
}
NOKPROBE_SYMBOL(save_return_addr);

3
arch/arm64/kernel/signal.c
View File

@ -244,7 +244,8 @@ static int preserve_sve_context(struct sve_context __user *ctx)
if (vq) {
/*
* This assumes that the SVE state has already been saved to
* the task struct by calling preserve_fpsimd_context().
* the task struct by calling the function
* fpsimd_signal_preserve_current_state().
*/
err |= __copy_to_user((char __user *)ctx + SVE_SIG_REGS_OFFSET,
current->thread.sve_state,

4
arch/arm64/kernel/smp_spin_table.c
View File

@ -83,9 +83,9 @@ static int smp_spin_table_cpu_prepare(unsigned int cpu)
/*
* We write the release address as LE regardless of the native
* endianess of the kernel. Therefore, any boot-loaders that
* endianness of the kernel. Therefore, any boot-loaders that
* read this address need to convert this address to the
* boot-loader's endianess before jumping. This is mandated by
* boot-loader's endianness before jumping. This is mandated by
* the boot protocol.
*/
writeq_relaxed(__pa_symbol(secondary_holding_pen), release_addr);

131
arch/arm64/kernel/stacktrace.c
View File

@ -118,12 +118,12 @@ int notrace unwind_frame(struct task_struct *tsk, struct stackframe *frame)
NOKPROBE_SYMBOL(unwind_frame);
void notrace walk_stackframe(struct task_struct *tsk, struct stackframe *frame,
int (*fn)(struct stackframe *, void *), void *data)
bool (*fn)(void *, unsigned long), void *data)
{
while (1) {
int ret;
if (fn(frame, data))
if (!fn(data, frame->pc))
break;
ret = unwind_frame(tsk, frame);
if (ret < 0)
@ -132,84 +132,89 @@ void notrace walk_stackframe(struct task_struct *tsk, struct stackframe *frame,
}
NOKPROBE_SYMBOL(walk_stackframe);
#ifdef CONFIG_STACKTRACE
struct stack_trace_data {
struct stack_trace *trace;
unsigned int no_sched_functions;
unsigned int skip;
};
static int save_trace(struct stackframe *frame, void *d)
static void dump_backtrace_entry(unsigned long where, const char *loglvl)
{
struct stack_trace_data *data = d;
struct stack_trace *trace = data->trace;
unsigned long addr = frame->pc;
printk("%s %pS\n", loglvl, (void *)where);
}
if (data->no_sched_functions && in_sched_functions(addr))
return 0;
if (data->skip) {
data->skip--;
return 0;
void dump_backtrace(struct pt_regs *regs, struct task_struct *tsk,
const char *loglvl)
{
struct stackframe frame;
int skip = 0;
pr_debug("%s(regs = %p tsk = %p)\n", __func__, regs, tsk);
if (regs) {
if (user_mode(regs))
return;
skip = 1;
}
trace->entries[trace->nr_entries++] = addr;
return trace->nr_entries >= trace->max_entries;
}
void save_stack_trace_regs(struct pt_regs *regs, struct stack_trace *trace)
{
struct stack_trace_data data;
struct stackframe frame;
data.trace = trace;
data.skip = trace->skip;
data.no_sched_functions = 0;
start_backtrace(&frame, regs->regs[29], regs->pc);
walk_stackframe(current, &frame, save_trace, &data);
}
EXPORT_SYMBOL_GPL(save_stack_trace_regs);
static noinline void __save_stack_trace(struct task_struct *tsk,
struct stack_trace *trace, unsigned int nosched)
{
struct stack_trace_data data;
struct stackframe frame;
if (!tsk)
tsk = current;
if (!try_get_task_stack(tsk))
return;
data.trace = trace;
data.skip = trace->skip;
data.no_sched_functions = nosched;
if (tsk != current) {
start_backtrace(&frame, thread_saved_fp(tsk),
thread_saved_pc(tsk));
} else {
/* We don't want this function nor the caller */
data.skip += 2;
if (tsk == current) {
start_backtrace(&frame,
(unsigned long)__builtin_frame_address(0),
(unsigned long)__save_stack_trace);
(unsigned long)dump_backtrace);
} else {
/*
* task blocked in __switch_to
*/
start_backtrace(&frame,
thread_saved_fp(tsk),
thread_saved_pc(tsk));
}
walk_stackframe(tsk, &frame, save_trace, &data);
printk("%sCall trace:\n", loglvl);
do {
/* skip until specified stack frame */
if (!skip) {
dump_backtrace_entry(frame.pc, loglvl);
} else if (frame.fp == regs->regs[29]) {
skip = 0;
/*
* Mostly, this is the case where this function is
* called in panic/abort. As exception handler's
* stack frame does not contain the corresponding pc
* at which an exception has taken place, use regs->pc
* instead.
*/
dump_backtrace_entry(regs->pc, loglvl);
}
} while (!unwind_frame(tsk, &frame));
put_task_stack(tsk);
}
void save_stack_trace_tsk(struct task_struct *tsk, struct stack_trace *trace)
void show_stack(struct task_struct *tsk, unsigned long *sp, const char *loglvl)
{
__save_stack_trace(tsk, trace, 1);
}
EXPORT_SYMBOL_GPL(save_stack_trace_tsk);
void save_stack_trace(struct stack_trace *trace)
{
__save_stack_trace(current, trace, 0);
dump_backtrace(NULL, tsk, loglvl);
barrier();
}
#ifdef CONFIG_STACKTRACE
void arch_stack_walk(stack_trace_consume_fn consume_entry, void *cookie,
struct task_struct *task, struct pt_regs *regs)
{
struct stackframe frame;
if (regs)
start_backtrace(&frame, regs->regs[29], regs->pc);
else if (task == current)
start_backtrace(&frame,
(unsigned long)__builtin_frame_address(0),
(unsigned long)arch_stack_walk);
else
start_backtrace(&frame, thread_saved_fp(task),
thread_saved_pc(task));
walk_stackframe(task, &frame, consume_entry, cookie);
}
EXPORT_SYMBOL_GPL(save_stack_trace);
#endif

32
arch/arm64/kernel/topology.c
View File

@ -36,21 +36,23 @@ void store_cpu_topology(unsigned int cpuid)
if (mpidr & MPIDR_UP_BITMASK)
return;
/* Create cpu topology mapping based on MPIDR. */
if (mpidr & MPIDR_MT_BITMASK) {
/* Multiprocessor system : Multi-threads per core */
cpuid_topo->thread_id = MPIDR_AFFINITY_LEVEL(mpidr, 0);
cpuid_topo->core_id = MPIDR_AFFINITY_LEVEL(mpidr, 1);
cpuid_topo->package_id = MPIDR_AFFINITY_LEVEL(mpidr, 2) |
MPIDR_AFFINITY_LEVEL(mpidr, 3) << 8;
} else {
/* Multiprocessor system : Single-thread per core */
cpuid_topo->thread_id = -1;
cpuid_topo->core_id = MPIDR_AFFINITY_LEVEL(mpidr, 0);
cpuid_topo->package_id = MPIDR_AFFINITY_LEVEL(mpidr, 1) |
MPIDR_AFFINITY_LEVEL(mpidr, 2) << 8 |
MPIDR_AFFINITY_LEVEL(mpidr, 3) << 16;
}
/*
* This would be the place to create cpu topology based on MPIDR.
*
* However, it cannot be trusted to depict the actual topology; some
* pieces of the architecture enforce an artificial cap on Aff0 values
* (e.g. GICv3's ICC_SGI1R_EL1 limits it to 15), leading to an
* artificial cycling of Aff1, Aff2 and Aff3 values. IOW, these end up
* having absolutely no relationship to the actual underlying system
* topology, and cannot be reasonably used as core / package ID.
*
* If the MT bit is set, Aff0 *could* be used to define a thread ID, but
* we still wouldn't be able to obtain a sane core ID. This means we
* need to entirely ignore MPIDR for any topology deduction.
*/
cpuid_topo->thread_id = -1;
cpuid_topo->core_id = cpuid;
cpuid_topo->package_id = cpu_to_node(cpuid);
pr_debug("CPU%u: cluster %d core %d thread %d mpidr %#016llx\n",
cpuid, cpuid_topo->package_id, cpuid_topo->core_id,

132
arch/arm64/kernel/traps.c
View File

@ -34,6 +34,7 @@
#include <asm/daifflags.h>
#include <asm/debug-monitors.h>
#include <asm/esr.h>
#include <asm/extable.h>
#include <asm/insn.h>
#include <asm/kprobes.h>
#include <asm/traps.h>
@ -53,11 +54,6 @@ static const char *handler[]= {
int show_unhandled_signals = 0;
static void dump_backtrace_entry(unsigned long where, const char *loglvl)
{
printk("%s %pS\n", loglvl, (void *)where);
}
static void dump_kernel_instr(const char *lvl, struct pt_regs *regs)
{
unsigned long addr = instruction_pointer(regs);
@ -83,66 +79,6 @@ static void dump_kernel_instr(const char *lvl, struct pt_regs *regs)
printk("%sCode: %s\n", lvl, str);
}
void dump_backtrace(struct pt_regs *regs, struct task_struct *tsk,
const char *loglvl)
{
struct stackframe frame;
int skip = 0;
pr_debug("%s(regs = %p tsk = %p)\n", __func__, regs, tsk);
if (regs) {
if (user_mode(regs))
return;
skip = 1;
}
if (!tsk)
tsk = current;
if (!try_get_task_stack(tsk))
return;
if (tsk == current) {
start_backtrace(&frame,
(unsigned long)__builtin_frame_address(0),
(unsigned long)dump_backtrace);
} else {
/*
* task blocked in __switch_to
*/
start_backtrace(&frame,
thread_saved_fp(tsk),
thread_saved_pc(tsk));
}
printk("%sCall trace:\n", loglvl);
do {
/* skip until specified stack frame */
if (!skip) {
dump_backtrace_entry(frame.pc, loglvl);
} else if (frame.fp == regs->regs[29]) {
skip = 0;
/*
* Mostly, this is the case where this function is
* called in panic/abort. As exception handler's
* stack frame does not contain the corresponding pc
* at which an exception has taken place, use regs->pc
* instead.
*/
dump_backtrace_entry(regs->pc, loglvl);
}
} while (!unwind_frame(tsk, &frame));
put_task_stack(tsk);
}
void show_stack(struct task_struct *tsk, unsigned long *sp, const char *loglvl)
{
dump_backtrace(NULL, tsk, loglvl);
barrier();
}
#ifdef CONFIG_PREEMPT
#define S_PREEMPT " PREEMPT"
#elif defined(CONFIG_PREEMPT_RT)
@ -200,9 +136,9 @@ void die(const char *str, struct pt_regs *regs, int err)
oops_exit();
if (in_interrupt())
panic("Fatal exception in interrupt");
panic("%s: Fatal exception in interrupt", str);
if (panic_on_oops)
panic("Fatal exception");
panic("%s: Fatal exception", str);
raw_spin_unlock_irqrestore(&die_lock, flags);
@ -412,7 +348,7 @@ exit:
return fn ? fn(regs, instr) : 1;
}
void force_signal_inject(int signal, int code, unsigned long address)
void force_signal_inject(int signal, int code, unsigned long address, unsigned int err)
{
const char *desc;
struct pt_regs *regs = current_pt_regs();
@ -438,7 +374,7 @@ void force_signal_inject(int signal, int code, unsigned long address)
signal = SIGKILL;
}
arm64_notify_die(desc, regs, signal, code, (void __user *)address, 0);
arm64_notify_die(desc, regs, signal, code, (void __user *)address, err);
}
/*
@ -455,7 +391,7 @@ void arm64_notify_segfault(unsigned long addr)
code = SEGV_ACCERR;
mmap_read_unlock(current->mm);
force_signal_inject(SIGSEGV, code, addr);
force_signal_inject(SIGSEGV, code, addr, 0);
}
void do_undefinstr(struct pt_regs *regs)
@ -468,17 +404,28 @@ void do_undefinstr(struct pt_regs *regs)
return;
BUG_ON(!user_mode(regs));
force_signal_inject(SIGILL, ILL_ILLOPC, regs->pc);
force_signal_inject(SIGILL, ILL_ILLOPC, regs->pc, 0);
}
NOKPROBE_SYMBOL(do_undefinstr);
void do_bti(struct pt_regs *regs)
{
BUG_ON(!user_mode(regs));
force_signal_inject(SIGILL, ILL_ILLOPC, regs->pc);
force_signal_inject(SIGILL, ILL_ILLOPC, regs->pc, 0);
}
NOKPROBE_SYMBOL(do_bti);
void do_ptrauth_fault(struct pt_regs *regs, unsigned int esr)
{
/*
* Unexpected FPAC exception or pointer authentication failure in
* the kernel: kill the task before it does any more harm.
*/
BUG_ON(!user_mode(regs));
force_signal_inject(SIGILL, ILL_ILLOPN, regs->pc, esr);
}
NOKPROBE_SYMBOL(do_ptrauth_fault);
#define __user_cache_maint(insn, address, res) \
if (address >= user_addr_max()) { \
res = -EFAULT; \
@ -528,7 +475,7 @@ static void user_cache_maint_handler(unsigned int esr, struct pt_regs *regs)
__user_cache_maint("ic ivau", address, ret);
break;
default:
force_signal_inject(SIGILL, ILL_ILLOPC, regs->pc);
force_signal_inject(SIGILL, ILL_ILLOPC, regs->pc, 0);
return;
}
@ -581,7 +528,7 @@ static void mrs_handler(unsigned int esr, struct pt_regs *regs)
sysreg = esr_sys64_to_sysreg(esr);
if (do_emulate_mrs(regs, sysreg, rt) != 0)
force_signal_inject(SIGILL, ILL_ILLOPC, regs->pc);
force_signal_inject(SIGILL, ILL_ILLOPC, regs->pc, 0);
}
static void wfi_handler(unsigned int esr, struct pt_regs *regs)
@ -775,6 +722,7 @@ static const char *esr_class_str[] = {
[ESR_ELx_EC_SYS64] = "MSR/MRS (AArch64)",
[ESR_ELx_EC_SVE] = "SVE",
[ESR_ELx_EC_ERET] = "ERET/ERETAA/ERETAB",
[ESR_ELx_EC_FPAC] = "FPAC",
[ESR_ELx_EC_IMP_DEF] = "EL3 IMP DEF",
[ESR_ELx_EC_IABT_LOW] = "IABT (lower EL)",
[ESR_ELx_EC_IABT_CUR] = "IABT (current EL)",
@ -935,26 +883,6 @@ asmlinkage void enter_from_user_mode(void)
}
NOKPROBE_SYMBOL(enter_from_user_mode);
void __pte_error(const char *file, int line, unsigned long val)
{
pr_err("%s:%d: bad pte %016lx.\n", file, line, val);
}
void __pmd_error(const char *file, int line, unsigned long val)
{
pr_err("%s:%d: bad pmd %016lx.\n", file, line, val);
}
void __pud_error(const char *file, int line, unsigned long val)
{
pr_err("%s:%d: bad pud %016lx.\n", file, line, val);
}
void __pgd_error(const char *file, int line, unsigned long val)
{
pr_err("%s:%d: bad pgd %016lx.\n", file, line, val);
}
/* GENERIC_BUG traps */
int is_valid_bugaddr(unsigned long addr)
@ -994,6 +922,21 @@ static struct break_hook bug_break_hook = {
.imm = BUG_BRK_IMM,
};
static int reserved_fault_handler(struct pt_regs *regs, unsigned int esr)
{
pr_err("%s generated an invalid instruction at %pS!\n",
in_bpf_jit(regs) ? "BPF JIT" : "Kernel text patching",
(void *)instruction_pointer(regs));
/* We cannot handle this */
return DBG_HOOK_ERROR;
}
static struct break_hook fault_break_hook = {
.fn = reserved_fault_handler,
.imm = FAULT_BRK_IMM,
};
#ifdef CONFIG_KASAN_SW_TAGS
#define KASAN_ESR_RECOVER 0x20
@ -1059,6 +1002,7 @@ int __init early_brk64(unsigned long addr, unsigned int esr,
void __init trap_init(void)
{
register_kernel_break_hook(&bug_break_hook);
register_kernel_break_hook(&fault_break_hook);
#ifdef CONFIG_KASAN_SW_TAGS
register_kernel_break_hook(&kasan_break_hook);
#endif

51
arch/arm64/kernel/vdso.c
View File

@ -30,15 +30,11 @@
#include <asm/vdso.h>
extern char vdso_start[], vdso_end[];
#ifdef CONFIG_COMPAT_VDSO
extern char vdso32_start[], vdso32_end[];
#endif /* CONFIG_COMPAT_VDSO */
enum vdso_abi {
VDSO_ABI_AA64,
#ifdef CONFIG_COMPAT_VDSO
VDSO_ABI_AA32,
#endif /* CONFIG_COMPAT_VDSO */
};
enum vvar_pages {
@ -284,21 +280,17 @@ up_fail:
/*
* Create and map the vectors page for AArch32 tasks.
*/
#ifdef CONFIG_COMPAT_VDSO
static int aarch32_vdso_mremap(const struct vm_special_mapping *sm,
struct vm_area_struct *new_vma)
{
return __vdso_remap(VDSO_ABI_AA32, sm, new_vma);
}
#endif /* CONFIG_COMPAT_VDSO */
enum aarch32_map {
AA32_MAP_VECTORS, /* kuser helpers */
#ifdef CONFIG_COMPAT_VDSO
AA32_MAP_SIGPAGE,
AA32_MAP_VVAR,
AA32_MAP_VDSO,
#endif
AA32_MAP_SIGPAGE
};
static struct page *aarch32_vectors_page __ro_after_init;
@ -309,7 +301,10 @@ static struct vm_special_mapping aarch32_vdso_maps[] = {
.name = "[vectors]", /* ABI */
.pages = &aarch32_vectors_page,
},
#ifdef CONFIG_COMPAT_VDSO
[AA32_MAP_SIGPAGE] = {
.name = "[sigpage]", /* ABI */
.pages = &aarch32_sig_page,
},
[AA32_MAP_VVAR] = {
.name = "[vvar]",
.fault = vvar_fault,
@ -319,11 +314,6 @@ static struct vm_special_mapping aarch32_vdso_maps[] = {
.name = "[vdso]",
.mremap = aarch32_vdso_mremap,
},
#endif /* CONFIG_COMPAT_VDSO */
[AA32_MAP_SIGPAGE] = {
.name = "[sigpage]", /* ABI */
.pages = &aarch32_sig_page,
},
};
static int aarch32_alloc_kuser_vdso_page(void)
@ -362,25 +352,25 @@ static int aarch32_alloc_sigpage(void)
return 0;
}
#ifdef CONFIG_COMPAT_VDSO
static int __aarch32_alloc_vdso_pages(void)
{
if (!IS_ENABLED(CONFIG_COMPAT_VDSO))
return 0;
vdso_info[VDSO_ABI_AA32].dm = &aarch32_vdso_maps[AA32_MAP_VVAR];
vdso_info[VDSO_ABI_AA32].cm = &aarch32_vdso_maps[AA32_MAP_VDSO];
return __vdso_init(VDSO_ABI_AA32);
}
#endif /* CONFIG_COMPAT_VDSO */
static int __init aarch32_alloc_vdso_pages(void)
{
int ret;
#ifdef CONFIG_COMPAT_VDSO
ret = __aarch32_alloc_vdso_pages();
if (ret)
return ret;
#endif
ret = aarch32_alloc_sigpage();
if (ret)
@ -449,14 +439,12 @@ int aarch32_setup_additional_pages(struct linux_binprm *bprm, int uses_interp)
if (ret)
goto out;
#ifdef CONFIG_COMPAT_VDSO
ret = __setup_additional_pages(VDSO_ABI_AA32,
mm,
bprm,
uses_interp);
if (ret)
goto out;
#endif /* CONFIG_COMPAT_VDSO */
if (IS_ENABLED(CONFIG_COMPAT_VDSO)) {
ret = __setup_additional_pages(VDSO_ABI_AA32, mm, bprm,
uses_interp);
if (ret)
goto out;
}
ret = aarch32_sigreturn_setup(mm);
out:
@ -497,8 +485,7 @@ static int __init vdso_init(void)
}
arch_initcall(vdso_init);
int arch_setup_additional_pages(struct linux_binprm *bprm,
int uses_interp)
int arch_setup_additional_pages(struct linux_binprm *bprm, int uses_interp)
{
struct mm_struct *mm = current->mm;
int ret;
@ -506,11 +493,7 @@ int arch_setup_additional_pages(struct linux_binprm *bprm,
if (mmap_write_lock_killable(mm))
return -EINTR;
ret = __setup_additional_pages(VDSO_ABI_AA64,
mm,
bprm,
uses_interp);
ret = __setup_additional_pages(VDSO_ABI_AA64, mm, bprm, uses_interp);
mmap_write_unlock(mm);
return ret;

4
arch/arm64/kernel/vmlinux.lds.S
View File

@ -105,7 +105,7 @@ SECTIONS
*(.eh_frame)
}
. = KIMAGE_VADDR + TEXT_OFFSET;
. = KIMAGE_VADDR;
.head.text : {
_text = .;
@ -274,4 +274,4 @@ ASSERT((__entry_tramp_text_end - __entry_tramp_text_start) == PAGE_SIZE,
/*
* If padding is applied before .head.text, virt<->phys conversions will fail.
*/
ASSERT(_text == (KIMAGE_VADDR + TEXT_OFFSET), "HEAD is misaligned")
ASSERT(_text == KIMAGE_VADDR, "HEAD is misaligned")

26
arch/arm64/kvm/pmu-emul.c
View File

@ -269,6 +269,7 @@ void kvm_pmu_vcpu_destroy(struct kvm_vcpu *vcpu)
for (i = 0; i < ARMV8_PMU_MAX_COUNTERS; i++)
kvm_pmu_release_perf_event(&pmu->pmc[i]);
irq_work_sync(&vcpu->arch.pmu.overflow_work);
}
u64 kvm_pmu_valid_counter_mask(struct kvm_vcpu *vcpu)
@ -433,6 +434,22 @@ void kvm_pmu_sync_hwstate(struct kvm_vcpu *vcpu)
kvm_pmu_update_state(vcpu);
}
/**
* When perf interrupt is an NMI, we cannot safely notify the vcpu corresponding
* to the event.
* This is why we need a callback to do it once outside of the NMI context.
*/
static void kvm_pmu_perf_overflow_notify_vcpu(struct irq_work *work)
{
struct kvm_vcpu *vcpu;
struct kvm_pmu *pmu;
pmu = container_of(work, struct kvm_pmu, overflow_work);
vcpu = kvm_pmc_to_vcpu(pmu->pmc);
kvm_vcpu_kick(vcpu);
}
/**
* When the perf event overflows, set the overflow status and inform the vcpu.
*/
@ -465,7 +482,11 @@ static void kvm_pmu_perf_overflow(struct perf_event *perf_event,
if (kvm_pmu_overflow_status(vcpu)) {
kvm_make_request(KVM_REQ_IRQ_PENDING, vcpu);
kvm_vcpu_kick(vcpu);
if (!in_nmi())
kvm_vcpu_kick(vcpu);
else
irq_work_queue(&vcpu->arch.pmu.overflow_work);
}
cpu_pmu->pmu.start(perf_event, PERF_EF_RELOAD);
@ -764,6 +785,9 @@ static int kvm_arm_pmu_v3_init(struct kvm_vcpu *vcpu)
return ret;
}
init_irq_work(&vcpu->arch.pmu.overflow_work,
kvm_pmu_perf_overflow_notify_vcpu);
vcpu->arch.pmu.created = true;
return 0;
}

4
arch/arm64/kvm/vgic/vgic-mmio-v3.c
View File

@ -1001,8 +1001,8 @@ void vgic_v3_dispatch_sgi(struct kvm_vcpu *vcpu, u64 reg, bool allow_group1)
raw_spin_lock_irqsave(&irq->irq_lock, flags);
/*
* An access targetting Group0 SGIs can only generate
* those, while an access targetting Group1 SGIs can
* An access targeting Group0 SGIs can only generate
* those, while an access targeting Group1 SGIs can
* generate interrupts of either group.
*/
if (!irq->group || allow_group1) {

2
arch/arm64/mm/Makefile
View File

@ -4,7 +4,7 @@ obj-y := dma-mapping.o extable.o fault.o init.o \
ioremap.o mmap.o pgd.o mmu.o \
context.o proc.o pageattr.o
obj-$(CONFIG_HUGETLB_PAGE) += hugetlbpage.o
obj-$(CONFIG_PTDUMP_CORE) += dump.o
obj-$(CONFIG_PTDUMP_CORE) += ptdump.o
obj-$(CONFIG_PTDUMP_DEBUGFS) += ptdump_debugfs.o
obj-$(CONFIG_NUMA) += numa.o
obj-$(CONFIG_DEBUG_VIRTUAL) += physaddr.o

105
arch/arm64/mm/context.c
View File

@ -27,6 +27,10 @@ static DEFINE_PER_CPU(atomic64_t, active_asids);
static DEFINE_PER_CPU(u64, reserved_asids);
static cpumask_t tlb_flush_pending;
static unsigned long max_pinned_asids;
static unsigned long nr_pinned_asids;
static unsigned long *pinned_asid_map;
#define ASID_MASK (~GENMASK(asid_bits - 1, 0))
#define ASID_FIRST_VERSION (1UL << asid_bits)
@ -72,7 +76,7 @@ void verify_cpu_asid_bits(void)
}
}
static void set_kpti_asid_bits(void)
static void set_kpti_asid_bits(unsigned long *map)
{
unsigned int len = BITS_TO_LONGS(NUM_USER_ASIDS) * sizeof(unsigned long);
/*
@ -81,13 +85,15 @@ static void set_kpti_asid_bits(void)
* is set, then the ASID will map only userspace. Thus
* mark even as reserved for kernel.
*/
memset(asid_map, 0xaa, len);
memset(map, 0xaa, len);
}
static void set_reserved_asid_bits(void)
{
if (arm64_kernel_unmapped_at_el0())
set_kpti_asid_bits();
if (pinned_asid_map)
bitmap_copy(asid_map, pinned_asid_map, NUM_USER_ASIDS);
else if (arm64_kernel_unmapped_at_el0())
set_kpti_asid_bits(asid_map);
else
bitmap_clear(asid_map, 0, NUM_USER_ASIDS);
}
@ -165,6 +171,14 @@ static u64 new_context(struct mm_struct *mm)
if (check_update_reserved_asid(asid, newasid))
return newasid;
/*
* If it is pinned, we can keep using it. Note that reserved
* takes priority, because even if it is also pinned, we need to
* update the generation into the reserved_asids.
*/
if (refcount_read(&mm->context.pinned))
return newasid;
/*
* We had a valid ASID in a previous life, so try to re-use
* it if possible.
@ -256,6 +270,71 @@ switch_mm_fastpath:
cpu_switch_mm(mm->pgd, mm);
}
unsigned long arm64_mm_context_get(struct mm_struct *mm)
{
unsigned long flags;
u64 asid;
if (!pinned_asid_map)
return 0;
raw_spin_lock_irqsave(&cpu_asid_lock, flags);
asid = atomic64_read(&mm->context.id);
if (refcount_inc_not_zero(&mm->context.pinned))
goto out_unlock;
if (nr_pinned_asids >= max_pinned_asids) {
asid = 0;
goto out_unlock;
}
if (!asid_gen_match(asid)) {
/*
* We went through one or more rollover since that ASID was
* used. Ensure that it is still valid, or generate a new one.
*/
asid = new_context(mm);
atomic64_set(&mm->context.id, asid);
}
nr_pinned_asids++;
__set_bit(asid2idx(asid), pinned_asid_map);
refcount_set(&mm->context.pinned, 1);
out_unlock:
raw_spin_unlock_irqrestore(&cpu_asid_lock, flags);
asid &= ~ASID_MASK;
/* Set the equivalent of USER_ASID_BIT */
if (asid && arm64_kernel_unmapped_at_el0())
asid |= 1;
return asid;
}
EXPORT_SYMBOL_GPL(arm64_mm_context_get);
void arm64_mm_context_put(struct mm_struct *mm)
{
unsigned long flags;
u64 asid = atomic64_read(&mm->context.id);
if (!pinned_asid_map)
return;
raw_spin_lock_irqsave(&cpu_asid_lock, flags);
if (refcount_dec_and_test(&mm->context.pinned)) {
__clear_bit(asid2idx(asid), pinned_asid_map);
nr_pinned_asids--;
}
raw_spin_unlock_irqrestore(&cpu_asid_lock, flags);
}
EXPORT_SYMBOL_GPL(arm64_mm_context_put);
/* Errata workaround post TTBRx_EL1 update. */
asmlinkage void post_ttbr_update_workaround(void)
{
@ -296,8 +375,11 @@ static int asids_update_limit(void)
{
unsigned long num_available_asids = NUM_USER_ASIDS;
if (arm64_kernel_unmapped_at_el0())
if (arm64_kernel_unmapped_at_el0()) {
num_available_asids /= 2;
if (pinned_asid_map)
set_kpti_asid_bits(pinned_asid_map);
}
/*
* Expect allocation after rollover to fail if we don't have at least
* one more ASID than CPUs. ASID #0 is reserved for init_mm.
@ -305,6 +387,13 @@ static int asids_update_limit(void)
WARN_ON(num_available_asids - 1 <= num_possible_cpus());
pr_info("ASID allocator initialised with %lu entries\n",
num_available_asids);
/*
* There must always be an ASID available after rollover. Ensure that,
* even if all CPUs have a reserved ASID and the maximum number of ASIDs
* are pinned, there still is at least one empty slot in the ASID map.
*/
max_pinned_asids = num_available_asids - num_possible_cpus() - 2;
return 0;
}
arch_initcall(asids_update_limit);
@ -319,13 +408,17 @@ static int asids_init(void)
panic("Failed to allocate bitmap for %lu ASIDs\n",
NUM_USER_ASIDS);
pinned_asid_map = kcalloc(BITS_TO_LONGS(NUM_USER_ASIDS),
sizeof(*pinned_asid_map), GFP_KERNEL);
nr_pinned_asids = 0;
/*
* We cannot call set_reserved_asid_bits() here because CPU
* caps are not finalized yet, so it is safer to assume KPTI
* and reserve kernel ASID's from beginning.
*/
if (IS_ENABLED(CONFIG_UNMAP_KERNEL_AT_EL0))
set_kpti_asid_bits();
set_kpti_asid_bits(asid_map);
return 0;
}
early_initcall(asids_init);

4
arch/arm64/mm/extable.c
View File

@ -14,9 +14,7 @@ int fixup_exception(struct pt_regs *regs)
if (!fixup)
return 0;
if (IS_ENABLED(CONFIG_BPF_JIT) &&
regs->pc >= BPF_JIT_REGION_START &&
regs->pc < BPF_JIT_REGION_END)
if (in_bpf_jit(regs))
return arm64_bpf_fixup_exception(fixup, regs);
regs->pc = (unsigned long)&fixup->fixup + fixup->fixup;

4
arch/arm64/mm/fault.c
View File

@ -218,7 +218,9 @@ int ptep_set_access_flags(struct vm_area_struct *vma,
pteval = cmpxchg_relaxed(&pte_val(*ptep), old_pteval, pteval);
} while (pteval != old_pteval);
flush_tlb_fix_spurious_fault(vma, address);
/* Invalidate a stale read-only entry */
if (dirty)
flush_tlb_page(vma, address);
return 1;
}

8
arch/arm64/mm/numa.c
View File

@ -46,7 +46,11 @@ EXPORT_SYMBOL(node_to_cpumask_map);
*/
const struct cpumask *cpumask_of_node(int node)
{
if (WARN_ON(node >= nr_node_ids))
if (node == NUMA_NO_NODE)
return cpu_all_mask;
if (WARN_ON(node < 0 || node >= nr_node_ids))
return cpu_none_mask;
if (WARN_ON(node_to_cpumask_map[node] == NULL))
@ -448,7 +452,7 @@ static int __init dummy_numa_init(void)
* arm64_numa_init() - Initialize NUMA
*
* Try each configured NUMA initialization method until one succeeds. The
* last fallback is dummy single node config encomapssing whole memory.
* last fallback is dummy single node config encompassing whole memory.
*/
void __init arm64_numa_init(void)
{

1
arch/arm64/mm/pageattr.c
View File

@ -8,6 +8,7 @@
#include <linux/sched.h>
#include <linux/vmalloc.h>
#include <asm/cacheflush.h>
#include <asm/set_memory.h>
#include <asm/tlbflush.h>

2
arch/arm64/mm/dump.c → arch/arm64/mm/ptdump.c
View File

@ -41,6 +41,8 @@ static struct addr_marker address_markers[] = {
{ 0 /* KASAN_SHADOW_START */, "Kasan shadow start" },
{ KASAN_SHADOW_END, "Kasan shadow end" },
#endif
{ BPF_JIT_REGION_START, "BPF start" },
{ BPF_JIT_REGION_END, "BPF end" },
{ MODULES_VADDR, "Modules start" },
{ MODULES_END, "Modules end" },
{ VMALLOC_START, "vmalloc() area" },

4
arch/s390/kernel/stacktrace.c
View File

@ -19,7 +19,7 @@ void arch_stack_walk(stack_trace_consume_fn consume_entry, void *cookie,
unwind_for_each_frame(&state, task, regs, 0) {
addr = unwind_get_return_address(&state);
if (!addr || !consume_entry(cookie, addr, false))
if (!addr || !consume_entry(cookie, addr))
break;
}
}
@ -56,7 +56,7 @@ int arch_stack_walk_reliable(stack_trace_consume_fn consume_entry,
return -EINVAL;
#endif
if (!consume_entry(cookie, addr, false))
if (!consume_entry(cookie, addr))
return -EINVAL;
}

10
arch/x86/kernel/stacktrace.c
View File

@ -18,13 +18,13 @@ void arch_stack_walk(stack_trace_consume_fn consume_entry, void *cookie,
struct unwind_state state;
unsigned long addr;
if (regs && !consume_entry(cookie, regs->ip, false))
if (regs && !consume_entry(cookie, regs->ip))
return;
for (unwind_start(&state, task, regs, NULL); !unwind_done(&state);
unwind_next_frame(&state)) {
addr = unwind_get_return_address(&state);
if (!addr || !consume_entry(cookie, addr, false))
if (!addr || !consume_entry(cookie, addr))
break;
}
}
@ -72,7 +72,7 @@ int arch_stack_walk_reliable(stack_trace_consume_fn consume_entry,
if (!addr)
return -EINVAL;
if (!consume_entry(cookie, addr, false))
if (!consume_entry(cookie, addr))
return -EINVAL;
}
@ -114,7 +114,7 @@ void arch_stack_walk_user(stack_trace_consume_fn consume_entry, void *cookie,
{
const void __user *fp = (const void __user *)regs->bp;
if (!consume_entry(cookie, regs->ip, false))
if (!consume_entry(cookie, regs->ip))
return;
while (1) {
@ -128,7 +128,7 @@ void arch_stack_walk_user(stack_trace_consume_fn consume_entry, void *cookie,
break;
if (!frame.ret_addr)
break;
if (!consume_entry(cookie, frame.ret_addr, false))
if (!consume_entry(cookie, frame.ret_addr))
break;
fp = frame.next_fp;
}

10
drivers/acpi/arm64/iort.c
View File

@ -811,8 +811,7 @@ static inline const struct iommu_ops *iort_fwspec_iommu_ops(struct device *dev)
return (fwspec && fwspec->ops) ? fwspec->ops : NULL;
}
static inline int iort_add_device_replay(const struct iommu_ops *ops,
struct device *dev)
static inline int iort_add_device_replay(struct device *dev)
{
int err = 0;
@ -1072,7 +1071,7 @@ const struct iommu_ops *iort_iommu_configure_id(struct device *dev,
*/
if (!err) {
ops = iort_fwspec_iommu_ops(dev);
err = iort_add_device_replay(ops, dev);
err = iort_add_device_replay(dev);
}
/* Ignore all other errors apart from EPROBE_DEFER */
@ -1087,11 +1086,6 @@ const struct iommu_ops *iort_iommu_configure_id(struct device *dev,
}
#else
static inline const struct iommu_ops *iort_fwspec_iommu_ops(struct device *dev)
{ return NULL; }
static inline int iort_add_device_replay(const struct iommu_ops *ops,
struct device *dev)
{ return 0; }
int iort_iommu_msi_get_resv_regions(struct device *dev, struct list_head *head)
{ return 0; }
const struct iommu_ops *iort_iommu_configure_id(struct device *dev,

301
drivers/firmware/arm_sdei.c
View File

@ -78,11 +78,26 @@ struct sdei_crosscall_args {
int first_error;
};
#define CROSSCALL_INIT(arg, event) (arg.event = event, \
arg.first_error = 0, \
atomic_set(&arg.errors, 0))
#define CROSSCALL_INIT(arg, event) \
do { \
arg.event = event; \
arg.first_error = 0; \
atomic_set(&arg.errors, 0); \
} while (0)
static inline int sdei_do_cross_call(void *fn, struct sdei_event * event)
static inline int sdei_do_local_call(smp_call_func_t fn,
struct sdei_event *event)
{
struct sdei_crosscall_args arg;
CROSSCALL_INIT(arg, event);
fn(&arg);
return arg.first_error;
}
static inline int sdei_do_cross_call(smp_call_func_t fn,
struct sdei_event *event)
{
struct sdei_crosscall_args arg;
@ -114,26 +129,7 @@ static int sdei_to_linux_errno(unsigned long sdei_err)
return -ENOMEM;
}
/* Not an error value ... */
return sdei_err;
}
/*
* If x0 is any of these values, then the call failed, use sdei_to_linux_errno()
* to translate.
*/
static int sdei_is_err(struct arm_smccc_res *res)
{
switch (res->a0) {
case SDEI_NOT_SUPPORTED:
case SDEI_INVALID_PARAMETERS:
case SDEI_DENIED:
case SDEI_PENDING:
case SDEI_OUT_OF_RESOURCE:
return true;
}
return false;
return 0;
}
static int invoke_sdei_fn(unsigned long function_id, unsigned long arg0,
@ -141,14 +137,13 @@ static int invoke_sdei_fn(unsigned long function_id, unsigned long arg0,
unsigned long arg3, unsigned long arg4,
u64 *result)
{
int err = 0;
int err;
struct arm_smccc_res res;
if (sdei_firmware_call) {
sdei_firmware_call(function_id, arg0, arg1, arg2, arg3, arg4,
&res);
if (sdei_is_err(&res))
err = sdei_to_linux_errno(res.a0);
err = sdei_to_linux_errno(res.a0);
} else {
/*
* !sdei_firmware_call means we failed to probe or called
@ -210,36 +205,34 @@ static struct sdei_event *sdei_event_create(u32 event_num,
lockdep_assert_held(&sdei_events_lock);
event = kzalloc(sizeof(*event), GFP_KERNEL);
if (!event)
return ERR_PTR(-ENOMEM);
if (!event) {
err = -ENOMEM;
goto fail;
}
INIT_LIST_HEAD(&event->list);
event->event_num = event_num;
err = sdei_api_event_get_info(event_num, SDEI_EVENT_INFO_EV_PRIORITY,
&result);
if (err) {
kfree(event);
return ERR_PTR(err);
}
if (err)
goto fail;
event->priority = result;
err = sdei_api_event_get_info(event_num, SDEI_EVENT_INFO_EV_TYPE,
&result);
if (err) {
kfree(event);
return ERR_PTR(err);
}
if (err)
goto fail;
event->type = result;
if (event->type == SDEI_EVENT_TYPE_SHARED) {
reg = kzalloc(sizeof(*reg), GFP_KERNEL);
if (!reg) {
kfree(event);
return ERR_PTR(-ENOMEM);
err = -ENOMEM;
goto fail;
}
reg->event_num = event_num;
reg->event_num = event->event_num;
reg->priority = event->priority;
reg->callback = cb;
@ -251,8 +244,8 @@ static struct sdei_event *sdei_event_create(u32 event_num,
regs = alloc_percpu(struct sdei_registered_event);
if (!regs) {
kfree(event);
return ERR_PTR(-ENOMEM);
err = -ENOMEM;
goto fail;
}
for_each_possible_cpu(cpu) {
@ -272,6 +265,10 @@ static struct sdei_event *sdei_event_create(u32 event_num,
spin_unlock(&sdei_list_lock);
return event;
fail:
kfree(event);
return ERR_PTR(err);
}
static void sdei_event_destroy_llocked(struct sdei_event *event)
@ -490,16 +487,6 @@ static void _local_event_unregister(void *data)
sdei_cross_call_return(arg, err);
}
static int _sdei_event_unregister(struct sdei_event *event)
{
lockdep_assert_held(&sdei_events_lock);
if (event->type == SDEI_EVENT_TYPE_SHARED)
return sdei_api_event_unregister(event->event_num);
return sdei_do_cross_call(_local_event_unregister, event);
}
int sdei_event_unregister(u32 event_num)
{
int err;
@ -509,24 +496,27 @@ int sdei_event_unregister(u32 event_num)
mutex_lock(&sdei_events_lock);
event = sdei_event_find(event_num);
do {
if (!event) {
pr_warn("Event %u not registered\n", event_num);
err = -ENOENT;
break;
}
if (!event) {
pr_warn("Event %u not registered\n", event_num);
err = -ENOENT;
goto unlock;
}
spin_lock(&sdei_list_lock);
event->reregister = false;
event->reenable = false;
spin_unlock(&sdei_list_lock);
spin_lock(&sdei_list_lock);
event->reregister = false;
event->reenable = false;
spin_unlock(&sdei_list_lock);
err = _sdei_event_unregister(event);
if (err)
break;
if (event->type == SDEI_EVENT_TYPE_SHARED)
err = sdei_api_event_unregister(event->event_num);
else
err = sdei_do_cross_call(_local_event_unregister, event);
sdei_event_destroy(event);
} while (0);
if (err)
goto unlock;
sdei_event_destroy(event);
unlock:
mutex_unlock(&sdei_events_lock);
return err;
@ -547,7 +537,7 @@ static int sdei_unregister_shared(void)
if (event->type != SDEI_EVENT_TYPE_SHARED)
continue;
err = _sdei_event_unregister(event);
err = sdei_api_event_unregister(event->event_num);
if (err)
break;
}
@ -581,25 +571,6 @@ static void _local_event_register(void *data)
sdei_cross_call_return(arg, err);
}
static int _sdei_event_register(struct sdei_event *event)
{
int err;
lockdep_assert_held(&sdei_events_lock);
if (event->type == SDEI_EVENT_TYPE_SHARED)
return sdei_api_event_register(event->event_num,
sdei_entry_point,
event->registered,
SDEI_EVENT_REGISTER_RM_ANY, 0);
err = sdei_do_cross_call(_local_event_register, event);
if (err)
sdei_do_cross_call(_local_event_unregister, event);
return err;
}
int sdei_event_register(u32 event_num, sdei_event_callback *cb, void *arg)
{
int err;
@ -608,63 +579,44 @@ int sdei_event_register(u32 event_num, sdei_event_callback *cb, void *arg)
WARN_ON(in_nmi());
mutex_lock(&sdei_events_lock);
do {
if (sdei_event_find(event_num)) {
pr_warn("Event %u already registered\n", event_num);
err = -EBUSY;
break;
}
if (sdei_event_find(event_num)) {
pr_warn("Event %u already registered\n", event_num);
err = -EBUSY;
goto unlock;
}
event = sdei_event_create(event_num, cb, arg);
if (IS_ERR(event)) {
err = PTR_ERR(event);
pr_warn("Failed to create event %u: %d\n", event_num,
err);
break;
}
event = sdei_event_create(event_num, cb, arg);
if (IS_ERR(event)) {
err = PTR_ERR(event);
pr_warn("Failed to create event %u: %d\n", event_num, err);
goto unlock;
}
cpus_read_lock();
err = _sdei_event_register(event);
if (err) {
sdei_event_destroy(event);
pr_warn("Failed to register event %u: %d\n", event_num,
err);
} else {
spin_lock(&sdei_list_lock);
event->reregister = true;
spin_unlock(&sdei_list_lock);
}
cpus_read_unlock();
} while (0);
mutex_unlock(&sdei_events_lock);
cpus_read_lock();
if (event->type == SDEI_EVENT_TYPE_SHARED) {
err = sdei_api_event_register(event->event_num,
sdei_entry_point,
event->registered,
SDEI_EVENT_REGISTER_RM_ANY, 0);
} else {
err = sdei_do_cross_call(_local_event_register, event);
if (err)
sdei_do_cross_call(_local_event_unregister, event);
}
return err;
}
static int sdei_reregister_event_llocked(struct sdei_event *event)
{
int err;
lockdep_assert_held(&sdei_events_lock);
lockdep_assert_held(&sdei_list_lock);
err = _sdei_event_register(event);
if (err) {
pr_err("Failed to re-register event %u\n", event->event_num);
sdei_event_destroy_llocked(event);
return err;
sdei_event_destroy(event);
pr_warn("Failed to register event %u: %d\n", event_num, err);
goto cpu_unlock;
}
if (event->reenable) {
if (event->type == SDEI_EVENT_TYPE_SHARED)
err = sdei_api_event_enable(event->event_num);
else
err = sdei_do_cross_call(_local_event_enable, event);
}
if (err)
pr_err("Failed to re-enable event %u\n", event->event_num);
spin_lock(&sdei_list_lock);
event->reregister = true;
spin_unlock(&sdei_list_lock);
cpu_unlock:
cpus_read_unlock();
unlock:
mutex_unlock(&sdei_events_lock);
return err;
}
@ -680,9 +632,24 @@ static int sdei_reregister_shared(void)
continue;
if (event->reregister) {
err = sdei_reregister_event_llocked(event);
if (err)
err = sdei_api_event_register(event->event_num,
sdei_entry_point, event->registered,
SDEI_EVENT_REGISTER_RM_ANY, 0);
if (err) {
pr_err("Failed to re-register event %u\n",
event->event_num);
sdei_event_destroy_llocked(event);
break;
}
}
if (event->reenable) {
err = sdei_api_event_enable(event->event_num);
if (err) {
pr_err("Failed to re-enable event %u\n",
event->event_num);
break;
}
}
}
spin_unlock(&sdei_list_lock);
@ -694,7 +661,7 @@ static int sdei_reregister_shared(void)
static int sdei_cpuhp_down(unsigned int cpu)
{
struct sdei_event *event;
struct sdei_crosscall_args arg;
int err;
/* un-register private events */
spin_lock(&sdei_list_lock);
@ -702,12 +669,11 @@ static int sdei_cpuhp_down(unsigned int cpu)
if (event->type == SDEI_EVENT_TYPE_SHARED)
continue;
CROSSCALL_INIT(arg, event);
/* call the cross-call function locally... */
_local_event_unregister(&arg);
if (arg.first_error)
err = sdei_do_local_call(_local_event_unregister, event);
if (err) {
pr_err("Failed to unregister event %u: %d\n",
event->event_num, arg.first_error);
event->event_num, err);
}
}
spin_unlock(&sdei_list_lock);
@ -717,7 +683,7 @@ static int sdei_cpuhp_down(unsigned int cpu)
static int sdei_cpuhp_up(unsigned int cpu)
{
struct sdei_event *event;
struct sdei_crosscall_args arg;
int err;
/* re-register/enable private events */
spin_lock(&sdei_list_lock);
@ -726,20 +692,19 @@ static int sdei_cpuhp_up(unsigned int cpu)
continue;
if (event->reregister) {
CROSSCALL_INIT(arg, event);
/* call the cross-call function locally... */
_local_event_register(&arg);
if (arg.first_error)
err = sdei_do_local_call(_local_event_register, event);
if (err) {
pr_err("Failed to re-register event %u: %d\n",
event->event_num, arg.first_error);
event->event_num, err);
}
}
if (event->reenable) {
CROSSCALL_INIT(arg, event);
_local_event_enable(&arg);
if (arg.first_error)
err = sdei_do_local_call(_local_event_enable, event);
if (err) {
pr_err("Failed to re-enable event %u: %d\n",
event->event_num, arg.first_error);
event->event_num, err);
}
}
}
spin_unlock(&sdei_list_lock);
@ -976,7 +941,7 @@ static int sdei_get_conduit(struct platform_device *pdev)
}
pr_warn("invalid \"method\" property: %s\n", method);
} else if (IS_ENABLED(CONFIG_ACPI) && !acpi_disabled) {
} else if (!acpi_disabled) {
if (acpi_psci_use_hvc()) {
sdei_firmware_call = &sdei_smccc_hvc;
return SMCCC_CONDUIT_HVC;
@ -1000,8 +965,6 @@ static int sdei_probe(struct platform_device *pdev)
return 0;
err = sdei_api_get_version(&ver);
if (err == -EOPNOTSUPP)
pr_err("advertised but not implemented in platform firmware\n");
if (err) {
pr_err("Failed to get SDEI version: %d\n", err);
sdei_mark_interface_broken();
@ -1099,16 +1062,20 @@ static bool __init sdei_present_acpi(void)
static int __init sdei_init(void)
{
int ret = platform_driver_register(&sdei_driver);
struct platform_device *pdev;
int ret;
if (!ret && sdei_present_acpi()) {
struct platform_device *pdev;
ret = platform_driver_register(&sdei_driver);
if (ret || !sdei_present_acpi())
return ret;
pdev = platform_device_register_simple(sdei_driver.driver.name,
0, NULL, 0);
if (IS_ERR(pdev))
pr_info("Failed to register ACPI:SDEI platform device %ld\n",
PTR_ERR(pdev));
pdev = platform_device_register_simple(sdei_driver.driver.name,
0, NULL, 0);
if (IS_ERR(pdev)) {
ret = PTR_ERR(pdev);
platform_driver_unregister(&sdei_driver);
pr_info("Failed to register ACPI:SDEI platform device %d\n",
ret);
}
return ret;

1
drivers/firmware/efi/libstub/Makefile
View File

@ -64,7 +64,6 @@ lib-$(CONFIG_ARM) += arm32-stub.o
lib-$(CONFIG_ARM64) += arm64-stub.o
lib-$(CONFIG_X86) += x86-stub.o
CFLAGS_arm32-stub.o := -DTEXT_OFFSET=$(TEXT_OFFSET)
CFLAGS_arm64-stub.o := -DTEXT_OFFSET=$(TEXT_OFFSET)
#
# For x86, bootloaders like systemd-boot or grub-efi do not zero-initialize the

6
drivers/firmware/efi/libstub/arm64-stub.c
View File

@ -77,7 +77,7 @@ efi_status_t handle_kernel_image(unsigned long *image_addr,
kernel_size = _edata - _text;
kernel_memsize = kernel_size + (_end - _edata);
*reserve_size = kernel_memsize + TEXT_OFFSET % min_kimg_align();
*reserve_size = kernel_memsize;
if (IS_ENABLED(CONFIG_RANDOMIZE_BASE) && phys_seed != 0) {
/*
@ -91,7 +91,7 @@ efi_status_t handle_kernel_image(unsigned long *image_addr,
}
if (status != EFI_SUCCESS) {
if (IS_ALIGNED((u64)_text - TEXT_OFFSET, min_kimg_align())) {
if (IS_ALIGNED((u64)_text, min_kimg_align())) {
/*
* Just execute from wherever we were loaded by the
* UEFI PE/COFF loader if the alignment is suitable.
@ -111,7 +111,7 @@ efi_status_t handle_kernel_image(unsigned long *image_addr,
}
}
*image_addr = *reserve_addr + TEXT_OFFSET % min_kimg_align();
*image_addr = *reserve_addr;
memcpy((void *)*image_addr, _text, kernel_size);
return EFI_SUCCESS;

7
drivers/perf/Kconfig
View File

@ -41,6 +41,13 @@ config ARM_CCN
PMU (perf) driver supporting the ARM CCN (Cache Coherent Network)
interconnect.
config ARM_CMN
tristate "Arm CMN-600 PMU support"
depends on ARM64 || (COMPILE_TEST && 64BIT)
help
Support for PMU events monitoring on the Arm CMN-600 Coherent Mesh
Network interconnect.
config ARM_PMU
depends on ARM || ARM64
bool "ARM PMU framework"

1
drivers/perf/Makefile
View File

@ -1,6 +1,7 @@
# SPDX-License-Identifier: GPL-2.0
obj-$(CONFIG_ARM_CCI_PMU) += arm-cci.o
obj-$(CONFIG_ARM_CCN) += arm-ccn.o
obj-$(CONFIG_ARM_CMN) += arm-cmn.o
obj-$(CONFIG_ARM_DSU_PMU) += arm_dsu_pmu.o
obj-$(CONFIG_ARM_PMU) += arm_pmu.o arm_pmu_platform.o
obj-$(CONFIG_ARM_PMU_ACPI) += arm_pmu_acpi.o

1641
drivers/perf/arm-cmn.c Normal file
View File

File diff suppressed because it is too large Load Diff

63
drivers/perf/arm_dsu_pmu.c
View File

@ -11,6 +11,7 @@
#define DRVNAME PMUNAME "_pmu"
#define pr_fmt(fmt) DRVNAME ": " fmt
#include <linux/acpi.h>
#include <linux/bitmap.h>
#include <linux/bitops.h>
#include <linux/bug.h>
@ -603,18 +604,19 @@ static struct dsu_pmu *dsu_pmu_alloc(struct platform_device *pdev)
}
/**
* dsu_pmu_dt_get_cpus: Get the list of CPUs in the cluster.
* dsu_pmu_dt_get_cpus: Get the list of CPUs in the cluster
* from device tree.
*/
static int dsu_pmu_dt_get_cpus(struct device_node *dev, cpumask_t *mask)
static int dsu_pmu_dt_get_cpus(struct device *dev, cpumask_t *mask)
{
int i = 0, n, cpu;
struct device_node *cpu_node;
n = of_count_phandle_with_args(dev, "cpus", NULL);
n = of_count_phandle_with_args(dev->of_node, "cpus", NULL);
if (n <= 0)
return -ENODEV;
for (; i < n; i++) {
cpu_node = of_parse_phandle(dev, "cpus", i);
cpu_node = of_parse_phandle(dev->of_node, "cpus", i);
if (!cpu_node)
break;
cpu = of_cpu_node_to_id(cpu_node);
@ -631,6 +633,36 @@ static int dsu_pmu_dt_get_cpus(struct device_node *dev, cpumask_t *mask)
return 0;
}
/**
* dsu_pmu_acpi_get_cpus: Get the list of CPUs in the cluster
* from ACPI.
*/
static int dsu_pmu_acpi_get_cpus(struct device *dev, cpumask_t *mask)
{
#ifdef CONFIG_ACPI
int cpu;
/*
* A dsu pmu node is inside a cluster parent node along with cpu nodes.
* We need to find out all cpus that have the same parent with this pmu.
*/
for_each_possible_cpu(cpu) {
struct acpi_device *acpi_dev;
struct device *cpu_dev = get_cpu_device(cpu);
if (!cpu_dev)
continue;
acpi_dev = ACPI_COMPANION(cpu_dev);
if (acpi_dev &&
acpi_dev->parent == ACPI_COMPANION(dev)->parent)
cpumask_set_cpu(cpu, mask);
}
#endif
return 0;
}
/*
* dsu_pmu_probe_pmu: Probe the PMU details on a CPU in the cluster.
*/
@ -676,6 +708,7 @@ static int dsu_pmu_device_probe(struct platform_device *pdev)
{
int irq, rc;
struct dsu_pmu *dsu_pmu;
struct fwnode_handle *fwnode = dev_fwnode(&pdev->dev);
char *name;
static atomic_t pmu_idx = ATOMIC_INIT(-1);
@ -683,7 +716,16 @@ static int dsu_pmu_device_probe(struct platform_device *pdev)
if (IS_ERR(dsu_pmu))
return PTR_ERR(dsu_pmu);
rc = dsu_pmu_dt_get_cpus(pdev->dev.of_node, &dsu_pmu->associated_cpus);
if (IS_ERR_OR_NULL(fwnode))
return -ENOENT;
if (is_of_node(fwnode))
rc = dsu_pmu_dt_get_cpus(&pdev->dev, &dsu_pmu->associated_cpus);
else if (is_acpi_device_node(fwnode))
rc = dsu_pmu_acpi_get_cpus(&pdev->dev, &dsu_pmu->associated_cpus);
else
return -ENOENT;
if (rc) {
dev_warn(&pdev->dev, "Failed to parse the CPUs\n");
return rc;
@ -752,11 +794,21 @@ static const struct of_device_id dsu_pmu_of_match[] = {
{ .compatible = "arm,dsu-pmu", },
{},
};
MODULE_DEVICE_TABLE(of, dsu_pmu_of_match);
#ifdef CONFIG_ACPI
static const struct acpi_device_id dsu_pmu_acpi_match[] = {
{ "ARMHD500", 0},
{},
};
MODULE_DEVICE_TABLE(acpi, dsu_pmu_acpi_match);
#endif
static struct platform_driver dsu_pmu_driver = {
.driver = {
.name = DRVNAME,
.of_match_table = of_match_ptr(dsu_pmu_of_match),
.acpi_match_table = ACPI_PTR(dsu_pmu_acpi_match),
.suppress_bind_attrs = true,
},
.probe = dsu_pmu_device_probe,
@ -826,7 +878,6 @@ static void __exit dsu_pmu_exit(void)
module_init(dsu_pmu_init);
module_exit(dsu_pmu_exit);
MODULE_DEVICE_TABLE(of, dsu_pmu_of_match);
MODULE_DESCRIPTION("Perf driver for ARM DynamIQ Shared Unit");
MODULE_AUTHOR("Suzuki K Poulose <suzuki.poulose@arm.com>");
MODULE_LICENSE("GPL v2");

155
drivers/perf/arm_pmu.c
View File

@ -26,8 +26,84 @@
#include <asm/irq_regs.h>
static int armpmu_count_irq_users(const int irq);
struct pmu_irq_ops {
void (*enable_pmuirq)(unsigned int irq);
void (*disable_pmuirq)(unsigned int irq);
void (*free_pmuirq)(unsigned int irq, int cpu, void __percpu *devid);
};
static void armpmu_free_pmuirq(unsigned int irq, int cpu, void __percpu *devid)
{
free_irq(irq, per_cpu_ptr(devid, cpu));
}
static const struct pmu_irq_ops pmuirq_ops = {
.enable_pmuirq = enable_irq,
.disable_pmuirq = disable_irq_nosync,
.free_pmuirq = armpmu_free_pmuirq
};
static void armpmu_free_pmunmi(unsigned int irq, int cpu, void __percpu *devid)
{
free_nmi(irq, per_cpu_ptr(devid, cpu));
}
static const struct pmu_irq_ops pmunmi_ops = {
.enable_pmuirq = enable_nmi,
.disable_pmuirq = disable_nmi_nosync,
.free_pmuirq = armpmu_free_pmunmi
};
static void armpmu_enable_percpu_pmuirq(unsigned int irq)
{
enable_percpu_irq(irq, IRQ_TYPE_NONE);
}
static void armpmu_free_percpu_pmuirq(unsigned int irq, int cpu,
void __percpu *devid)
{
if (armpmu_count_irq_users(irq) == 1)
free_percpu_irq(irq, devid);
}
static const struct pmu_irq_ops percpu_pmuirq_ops = {
.enable_pmuirq = armpmu_enable_percpu_pmuirq,
.disable_pmuirq = disable_percpu_irq,
.free_pmuirq = armpmu_free_percpu_pmuirq
};
static void armpmu_enable_percpu_pmunmi(unsigned int irq)
{
if (!prepare_percpu_nmi(irq))
enable_percpu_nmi(irq, IRQ_TYPE_NONE);
}
static void armpmu_disable_percpu_pmunmi(unsigned int irq)
{
disable_percpu_nmi(irq);
teardown_percpu_nmi(irq);
}
static void armpmu_free_percpu_pmunmi(unsigned int irq, int cpu,
void __percpu *devid)
{
if (armpmu_count_irq_users(irq) == 1)
free_percpu_nmi(irq, devid);
}
static const struct pmu_irq_ops percpu_pmunmi_ops = {
.enable_pmuirq = armpmu_enable_percpu_pmunmi,
.disable_pmuirq = armpmu_disable_percpu_pmunmi,
.free_pmuirq = armpmu_free_percpu_pmunmi
};
static DEFINE_PER_CPU(struct arm_pmu *, cpu_armpmu);
static DEFINE_PER_CPU(int, cpu_irq);
static DEFINE_PER_CPU(const struct pmu_irq_ops *, cpu_irq_ops);
static bool has_nmi;
static inline u64 arm_pmu_event_max_period(struct perf_event *event)
{
@ -544,6 +620,23 @@ static int armpmu_count_irq_users(const int irq)
return count;
}
static const struct pmu_irq_ops *armpmu_find_irq_ops(int irq)
{
const struct pmu_irq_ops *ops = NULL;
int cpu;
for_each_possible_cpu(cpu) {
if (per_cpu(cpu_irq, cpu) != irq)
continue;
ops = per_cpu(cpu_irq_ops, cpu);
if (ops)
break;
}
return ops;
}
void armpmu_free_irq(int irq, int cpu)
{
if (per_cpu(cpu_irq, cpu) == 0)
@ -551,18 +644,18 @@ void armpmu_free_irq(int irq, int cpu)
if (WARN_ON(irq != per_cpu(cpu_irq, cpu)))
return;
if (!irq_is_percpu_devid(irq))
free_irq(irq, per_cpu_ptr(&cpu_armpmu, cpu));
else if (armpmu_count_irq_users(irq) == 1)
free_percpu_irq(irq, &cpu_armpmu);
per_cpu(cpu_irq_ops, cpu)->free_pmuirq(irq, cpu, &cpu_armpmu);
per_cpu(cpu_irq, cpu) = 0;
per_cpu(cpu_irq_ops, cpu) = NULL;
}
int armpmu_request_irq(int irq, int cpu)
{
int err = 0;
const irq_handler_t handler = armpmu_dispatch_irq;
const struct pmu_irq_ops *irq_ops;
if (!irq)
return 0;
@ -582,17 +675,44 @@ int armpmu_request_irq(int irq, int cpu)
IRQF_NO_THREAD;
irq_set_status_flags(irq, IRQ_NOAUTOEN);
err = request_irq(irq, handler, irq_flags, "arm-pmu",
err = request_nmi(irq, handler, irq_flags, "arm-pmu",
per_cpu_ptr(&cpu_armpmu, cpu));
/* If cannot get an NMI, get a normal interrupt */
if (err) {
err = request_irq(irq, handler, irq_flags, "arm-pmu",
per_cpu_ptr(&cpu_armpmu, cpu));
irq_ops = &pmuirq_ops;
} else {
has_nmi = true;
irq_ops = &pmunmi_ops;
}
} else if (armpmu_count_irq_users(irq) == 0) {
err = request_percpu_irq(irq, handler, "arm-pmu",
&cpu_armpmu);
err = request_percpu_nmi(irq, handler, "arm-pmu", &cpu_armpmu);
/* If cannot get an NMI, get a normal interrupt */
if (err) {
err = request_percpu_irq(irq, handler, "arm-pmu",
&cpu_armpmu);
irq_ops = &percpu_pmuirq_ops;
} else {
has_nmi= true;
irq_ops = &percpu_pmunmi_ops;
}
} else {
/* Per cpudevid irq was already requested by another CPU */
irq_ops = armpmu_find_irq_ops(irq);
if (WARN_ON(!irq_ops))
err = -EINVAL;
}
if (err)
goto err_out;
per_cpu(cpu_irq, cpu) = irq;
per_cpu(cpu_irq_ops, cpu) = irq_ops;
return 0;
err_out:
@ -625,12 +745,8 @@ static int arm_perf_starting_cpu(unsigned int cpu, struct hlist_node *node)
per_cpu(cpu_armpmu, cpu) = pmu;
irq = armpmu_get_cpu_irq(pmu, cpu);
if (irq) {
if (irq_is_percpu_devid(irq))
enable_percpu_irq(irq, IRQ_TYPE_NONE);
else
enable_irq(irq);
}
if (irq)
per_cpu(cpu_irq_ops, cpu)->enable_pmuirq(irq);
return 0;
}
@ -644,12 +760,8 @@ static int arm_perf_teardown_cpu(unsigned int cpu, struct hlist_node *node)
return 0;
irq = armpmu_get_cpu_irq(pmu, cpu);
if (irq) {
if (irq_is_percpu_devid(irq))
disable_percpu_irq(irq);
else
disable_irq_nosync(irq);
}
if (irq)
per_cpu(cpu_irq_ops, cpu)->disable_pmuirq(irq);
per_cpu(cpu_armpmu, cpu) = NULL;
@ -870,8 +982,9 @@ int armpmu_register(struct arm_pmu *pmu)
if (!__oprofile_cpu_pmu)
__oprofile_cpu_pmu = pmu;
pr_info("enabled with %s PMU driver, %d counters available\n",
pmu->name, pmu->num_events);
pr_info("enabled with %s PMU driver, %d counters available%s\n",
pmu->name, pmu->num_events,
has_nmi ? ", using NMIs" : "");
return 0;

1
drivers/perf/hisilicon/hisi_uncore_pmu.h
View File

@ -14,6 +14,7 @@
#include <linux/cpumask.h>
#include <linux/device.h>
#include <linux/kernel.h>
#include <linux/module.h>
#include <linux/perf_event.h>
#include <linux/types.h>

11
drivers/perf/thunderx2_pmu.c
View File

@ -805,14 +805,17 @@ static struct tx2_uncore_pmu *tx2_uncore_pmu_init_dev(struct device *dev,
list_for_each_entry(rentry, &list, node) {
if (resource_type(rentry->res) == IORESOURCE_MEM) {
res = *rentry->res;
rentry = NULL;
break;
}
}
if (!rentry->res)
return NULL;
acpi_dev_free_resource_list(&list);
if (rentry) {
dev_err(dev, "PMU type %d: Fail to find resource\n", type);
return NULL;
}
base = devm_ioremap_resource(dev, &res);
if (IS_ERR(base)) {
dev_err(dev, "PMU type %d: Fail to map resource\n", type);

32
drivers/perf/xgene_pmu.c
View File

@ -1453,17 +1453,6 @@ static char *xgene_pmu_dev_name(struct device *dev, u32 type, int id)
}
#if defined(CONFIG_ACPI)
static int acpi_pmu_dev_add_resource(struct acpi_resource *ares, void *data)
{
struct resource *res = data;
if (ares->type == ACPI_RESOURCE_TYPE_FIXED_MEMORY32)
acpi_dev_resource_memory(ares, res);
/* Always tell the ACPI core to skip this resource */
return 1;
}
static struct
xgene_pmu_dev_ctx *acpi_get_pmu_hw_inf(struct xgene_pmu *xgene_pmu,
struct acpi_device *adev, u32 type)
@ -1475,6 +1464,7 @@ xgene_pmu_dev_ctx *acpi_get_pmu_hw_inf(struct xgene_pmu *xgene_pmu,
struct hw_pmu_info *inf;
void __iomem *dev_csr;
struct resource res;
struct resource_entry *rentry;
int enable_bit;
int rc;
@ -1483,11 +1473,23 @@ xgene_pmu_dev_ctx *acpi_get_pmu_hw_inf(struct xgene_pmu *xgene_pmu,
return NULL;
INIT_LIST_HEAD(&resource_list);
rc = acpi_dev_get_resources(adev, &resource_list,
acpi_pmu_dev_add_resource, &res);
rc = acpi_dev_get_resources(adev, &resource_list, NULL, NULL);
if (rc <= 0) {
dev_err(dev, "PMU type %d: No resources found\n", type);
return NULL;
}
list_for_each_entry(rentry, &resource_list, node) {
if (resource_type(rentry->res) == IORESOURCE_MEM) {
res = *rentry->res;
rentry = NULL;
break;
}
}
acpi_dev_free_resource_list(&resource_list);
if (rc < 0) {
dev_err(dev, "PMU type %d: No resource address found\n", type);
if (rentry) {
dev_err(dev, "PMU type %d: No memory resource found\n", type);
return NULL;
}

1
include/kvm/arm_pmu.h
View File

@ -27,6 +27,7 @@ struct kvm_pmu {
bool ready;
bool created;
bool irq_level;
struct irq_work overflow_work;
};
#define kvm_arm_pmu_v3_ready(v) ((v)->arch.pmu.ready)

3
include/linux/perf/arm_pmu.h
View File

@ -73,6 +73,7 @@ enum armpmu_attr_groups {
ARMPMU_ATTR_GROUP_COMMON,
ARMPMU_ATTR_GROUP_EVENTS,
ARMPMU_ATTR_GROUP_FORMATS,
ARMPMU_ATTR_GROUP_CAPS,
ARMPMU_NR_ATTR_GROUPS
};
@ -109,6 +110,8 @@ struct arm_pmu {
struct notifier_block cpu_pm_nb;
/* the attr_groups array must be NULL-terminated */
const struct attribute_group *attr_groups[ARMPMU_NR_ATTR_GROUPS + 1];
/* store the PMMIR_EL1 to expose slots */
u64 reg_pmmir;
/* Only to be used by ACPI probing code */
unsigned long acpi_cpuid;

5
include/linux/stacktrace.h
View File

@ -29,14 +29,11 @@ unsigned int stack_trace_save_user(unsigned long *store, unsigned int size);
* stack_trace_consume_fn - Callback for arch_stack_walk()
* @cookie: Caller supplied pointer handed back by arch_stack_walk()
* @addr: The stack entry address to consume
* @reliable: True when the stack entry is reliable. Required by
* some printk based consumers.
*
* Return: True, if the entry was consumed or skipped
* False, if there is no space left to store
*/
typedef bool (*stack_trace_consume_fn)(void *cookie, unsigned long addr,
bool reliable);
typedef bool (*stack_trace_consume_fn)(void *cookie, unsigned long addr);
/**
* arch_stack_walk - Architecture specific function to walk the stack
* @consume_entry: Callback which is invoked by the architecture code for

8
kernel/stacktrace.c
View File

@ -78,8 +78,7 @@ struct stacktrace_cookie {
unsigned int len;
};
static bool stack_trace_consume_entry(void *cookie, unsigned long addr,
bool reliable)
static bool stack_trace_consume_entry(void *cookie, unsigned long addr)
{
struct stacktrace_cookie *c = cookie;
@ -94,12 +93,11 @@ static bool stack_trace_consume_entry(void *cookie, unsigned long addr,
return c->len < c->size;
}
static bool stack_trace_consume_entry_nosched(void *cookie, unsigned long addr,
bool reliable)
static bool stack_trace_consume_entry_nosched(void *cookie, unsigned long addr)
{
if (in_sched_functions(addr))
return true;
return stack_trace_consume_entry(cookie, addr, reliable);
return stack_trace_consume_entry(cookie, addr);
}
/**

2
tools/testing/selftests/arm64/Makefile
View File

@ -4,7 +4,7 @@
ARCH ?= $(shell uname -m 2>/dev/null || echo not)
ifneq (,$(filter $(ARCH),aarch64 arm64))
ARM64_SUBTARGETS ?= tags signal
ARM64_SUBTARGETS ?= tags signal pauth fp
else
ARM64_SUBTARGETS :=
endif

5
tools/testing/selftests/arm64/fp/.gitignore vendored Normal file
View File

@ -0,0 +1,5 @@
fpsimd-test
sve-probe-vls
sve-ptrace
sve-test
vlset

17
tools/testing/selftests/arm64/fp/Makefile Normal file
View File

@ -0,0 +1,17 @@
# SPDX-License-Identifier: GPL-2.0
CFLAGS += -I../../../../../usr/include/
TEST_GEN_PROGS := sve-ptrace sve-probe-vls
TEST_PROGS_EXTENDED := fpsimd-test fpsimd-stress sve-test sve-stress vlset
all: $(TEST_GEN_PROGS) $(TEST_PROGS_EXTENDED)
fpsimd-test: fpsimd-test.o
$(CC) -nostdlib $^ -o $@
sve-ptrace: sve-ptrace.o sve-ptrace-asm.o
sve-probe-vls: sve-probe-vls.o
sve-test: sve-test.o
$(CC) -nostdlib $^ -o $@
vlset: vlset.o
include ../../lib.mk

100
tools/testing/selftests/arm64/fp/README Normal file
View File

@ -0,0 +1,100 @@
This directory contains a mix of tests integrated with kselftest and
standalone stress tests.
kselftest tests
===============
sve-probe-vls - Checks the SVE vector length enumeration interface
sve-ptrace - Checks the SVE ptrace interface
Running the non-kselftest tests
===============================
sve-stress performs an SVE context switch stress test, as described
below.
(The fpsimd-stress test works the same way; just substitute "fpsimd" for
"sve" in the following commands.)
The test runs until killed by the user.
If no context switch error was detected, you will see output such as
the following:
$ ./sve-stress
(wait for some time)
^C
Vector length: 512 bits
PID: 1573
Terminated by signal 15, no error, iterations=9467, signals=1014
Vector length: 512 bits
PID: 1575
Terminated by signal 15, no error, iterations=9448, signals=1028
Vector length: 512 bits
PID: 1577
Terminated by signal 15, no error, iterations=9436, signals=1039
Vector length: 512 bits
PID: 1579
Terminated by signal 15, no error, iterations=9421, signals=1039
Vector length: 512 bits
PID: 1581
Terminated by signal 15, no error, iterations=9403, signals=1039
Vector length: 512 bits
PID: 1583
Terminated by signal 15, no error, iterations=9385, signals=1036
Vector length: 512 bits
PID: 1585
Terminated by signal 15, no error, iterations=9376, signals=1039
Vector length: 512 bits
PID: 1587
Terminated by signal 15, no error, iterations=9361, signals=1039
Vector length: 512 bits
PID: 1589
Terminated by signal 15, no error, iterations=9350, signals=1039
If an error was detected, details of the mismatch will be printed
instead of "no error".
Ideally, the test should be allowed to run for many minutes or hours
to maximise test coverage.
KVM stress testing
==================
To try to reproduce the bugs that we have been observing, sve-stress
should be run in parallel in two KVM guests, while simultaneously
running on the host.
1) Start 2 guests, using the following command for each:
$ lkvm run --console=virtio -pconsole=hvc0 --sve Image
(Depending on the hardware GIC implementation, you may also need
--irqchip=gicv3. New kvmtool defaults to that if appropriate, but I
can't remember whether my branch is new enough for that. Try without
the option first.)
Kvmtool occupies the terminal until you kill it (Ctrl+A x),
or until the guest terminates. It is therefore recommended to run
each instance in separate terminal (use screen or ssh etc.) This
allows multiple guests to be run in parallel while running other
commands on the host.
Within the guest, the host filesystem is accessible, mounted on /host.
2) Run the sve-stress on *each* guest with the Vector-Length set to 32:
guest$ ./vlset --inherit 32 ./sve-stress
3) Run the sve-stress on the host with the maximum Vector-Length:
host$ ./vlset --inherit --max ./sve-stress
Again, the test should be allowed to run for many minutes or hours to
maximise test coverage.
If no error is detected, you will see output from each sve-stress
instance similar to that illustrated above; otherwise details of the
observed mismatches will be printed.

11
tools/testing/selftests/arm64/fp/asm-offsets.h Normal file
View File

@ -0,0 +1,11 @@
#define sa_sz 32
#define sa_flags 8
#define sa_handler 0
#define sa_mask_sz 8
#define SIGUSR1 10
#define SIGTERM 15
#define SIGINT 2
#define SIGABRT 6
#define SA_NODEFER 1073741824
#define SA_SIGINFO 4
#define ucontext_regs 184

57
tools/testing/selftests/arm64/fp/assembler.h Normal file
View File

@ -0,0 +1,57 @@
// SPDX-License-Identifier: GPL-2.0-only
// Copyright (C) 2015-2019 ARM Limited.
// Original author: Dave Martin <Dave.Martin@arm.com>
#ifndef ASSEMBLER_H
#define ASSEMBLER_H
.macro __for from:req, to:req
.if (\from) == (\to)
_for__body %\from
.else
__for \from, %(\from) + ((\to) - (\from)) / 2
__for %(\from) + ((\to) - (\from)) / 2 + 1, \to
.endif
.endm
.macro _for var:req, from:req, to:req, insn:vararg
.macro _for__body \var:req
.noaltmacro
\insn
.altmacro
.endm
.altmacro
__for \from, \to
.noaltmacro
.purgem _for__body
.endm
.macro function name
.macro endfunction
.type \name, @function
.purgem endfunction
.endm
\name:
.endm
.macro define_accessor name, num, insn
.macro \name\()_entry n
\insn \n, 1
ret
.endm
function \name
adr x2, .L__accessor_tbl\@
add x2, x2, x0, lsl #3
br x2
.L__accessor_tbl\@:
_for x, 0, (\num) - 1, \name\()_entry \x
endfunction
.purgem \name\()_entry
.endm
#endif /* ! ASSEMBLER_H */

60
tools/testing/selftests/arm64/fp/fpsimd-stress Executable file
View File

@ -0,0 +1,60 @@
#!/bin/bash
# SPDX-License-Identifier: GPL-2.0-only
# Copyright (C) 2015-2019 ARM Limited.
# Original author: Dave Martin <Dave.Martin@arm.com>
set -ue
NR_CPUS=`nproc`
pids=
logs=
cleanup () {
trap - INT TERM CHLD
set +e
if [ -n "$pids" ]; then
kill $pids
wait $pids
pids=
fi
if [ -n "$logs" ]; then
cat $logs
rm $logs
logs=
fi
}
interrupt () {
cleanup
exit 0
}
child_died () {
cleanup
exit 1
}
trap interrupt INT TERM EXIT
trap child_died CHLD
for x in `seq 0 $((NR_CPUS * 4))`; do
log=`mktemp`
logs=$logs\ $log
./fpsimd-test >$log &
pids=$pids\ $!
done
# Wait for all child processes to be created:
sleep 10
while :; do
kill -USR1 $pids
done &
pids=$pids\ $!
wait
exit 1

482
tools/testing/selftests/arm64/fp/fpsimd-test.S Normal file
View File

@ -0,0 +1,482 @@
// SPDX-License-Identifier: GPL-2.0-only
// Copyright (C) 2015-2019 ARM Limited.
// Original author: Dave Martin <Dave.Martin@arm.com>
//
// Simple FPSIMD context switch test
// Repeatedly writes unique test patterns into each FPSIMD register
// and reads them back to verify integrity.
//
// for x in `seq 1 NR_CPUS`; do fpsimd-test & pids=$pids\ $! ; done
// (leave it running for as long as you want...)
// kill $pids
#include <asm/unistd.h>
#include "assembler.h"
#include "asm-offsets.h"
#define NVR 32
#define MAXVL_B (128 / 8)
.macro _vldr Vn:req, Xt:req
ld1 {v\Vn\().2d}, [x\Xt]
.endm
.macro _vstr Vn:req, Xt:req
st1 {v\Vn\().2d}, [x\Xt]
.endm
// Generate accessor functions to read/write programmatically selected
// FPSIMD registers.
// x0 is the register index to access
// x1 is the memory address to read from (getv,setp) or store to (setv,setp)
// All clobber x0-x2
define_accessor setv, NVR, _vldr
define_accessor getv, NVR, _vstr
// Print a single character x0 to stdout
// Clobbers x0-x2,x8
function putc
str x0, [sp, #-16]!
mov x0, #1 // STDOUT_FILENO
mov x1, sp
mov x2, #1
mov x8, #__NR_write
svc #0
add sp, sp, #16
ret
endfunction
// Print a NUL-terminated string starting at address x0 to stdout
// Clobbers x0-x3,x8
function puts
mov x1, x0
mov x2, #0
0: ldrb w3, [x0], #1
cbz w3, 1f
add x2, x2, #1
b 0b
1: mov w0, #1 // STDOUT_FILENO
mov x8, #__NR_write
svc #0
ret
endfunction
// Utility macro to print a literal string
// Clobbers x0-x4,x8
.macro puts string
.pushsection .rodata.str1.1, "aMS", 1
.L__puts_literal\@: .string "\string"
.popsection
ldr x0, =.L__puts_literal\@
bl puts
.endm
// Print an unsigned decimal number x0 to stdout
// Clobbers x0-x4,x8
function putdec
mov x1, sp
str x30, [sp, #-32]! // Result can't be > 20 digits
mov x2, #0
strb w2, [x1, #-1]! // Write the NUL terminator
mov x2, #10
0: udiv x3, x0, x2 // div-mod loop to generate the digits
msub x0, x3, x2, x0
add w0, w0, #'0'
strb w0, [x1, #-1]!
mov x0, x3
cbnz x3, 0b
ldrb w0, [x1]
cbnz w0, 1f
mov w0, #'0' // Print "0" for 0, not ""
strb w0, [x1, #-1]!
1: mov x0, x1
bl puts
ldr x30, [sp], #32
ret
endfunction
// Print an unsigned decimal number x0 to stdout, followed by a newline
// Clobbers x0-x5,x8
function putdecn
mov x5, x30
bl putdec
mov x0, #'\n'
bl putc
ret x5
endfunction
// Clobbers x0-x3,x8
function puthexb
str x30, [sp, #-0x10]!
mov w3, w0
lsr w0, w0, #4
bl puthexnibble
mov w0, w3
ldr x30, [sp], #0x10
// fall through to puthexnibble
endfunction
// Clobbers x0-x2,x8
function puthexnibble
and w0, w0, #0xf
cmp w0, #10
blo 1f
add w0, w0, #'a' - ('9' + 1)
1: add w0, w0, #'0'
b putc
endfunction
// x0=data in, x1=size in, clobbers x0-x5,x8
function dumphex
str x30, [sp, #-0x10]!
mov x4, x0
mov x5, x1
0: subs x5, x5, #1
b.lo 1f
ldrb w0, [x4], #1
bl puthexb
b 0b
1: ldr x30, [sp], #0x10
ret
endfunction
// Declare some storate space to shadow the SVE register contents:
.pushsection .text
.data
.align 4
vref:
.space MAXVL_B * NVR
scratch:
.space MAXVL_B
.popsection
// Trivial memory copy: copy x2 bytes, starting at address x1, to address x0.
// Clobbers x0-x3
function memcpy
cmp x2, #0
b.eq 1f
0: ldrb w3, [x1], #1
strb w3, [x0], #1
subs x2, x2, #1
b.ne 0b
1: ret
endfunction
// Generate a test pattern for storage in SVE registers
// x0: pid (16 bits)
// x1: register number (6 bits)
// x2: generation (4 bits)
function pattern
orr w1, w0, w1, lsl #16
orr w2, w1, w2, lsl #28
ldr x0, =scratch
mov w1, #MAXVL_B / 4
0: str w2, [x0], #4
add w2, w2, #(1 << 22)
subs w1, w1, #1
bne 0b
ret
endfunction
// Get the address of shadow data for FPSIMD V-register V<xn>
.macro _adrv xd, xn, nrtmp
ldr \xd, =vref
mov x\nrtmp, #16
madd \xd, x\nrtmp, \xn, \xd
.endm
// Set up test pattern in a FPSIMD V-register
// x0: pid
// x1: register number
// x2: generation
function setup_vreg
mov x4, x30
mov x6, x1
bl pattern
_adrv x0, x6, 2
mov x5, x0
ldr x1, =scratch
bl memcpy
mov x0, x6
mov x1, x5
bl setv
ret x4
endfunction
// Fill x1 bytes starting at x0 with 0xae (for canary purposes)
// Clobbers x1, x2.
function memfill_ae
mov w2, #0xae
b memfill
endfunction
// Fill x1 bytes starting at x0 with 0.
// Clobbers x1, x2.
function memclr
mov w2, #0
endfunction
// fall through to memfill
// Trivial memory fill: fill x1 bytes starting at address x0 with byte w2
// Clobbers x1
function memfill
cmp x1, #0
b.eq 1f
0: strb w2, [x0], #1
subs x1, x1, #1
b.ne 0b
1: ret
endfunction
// Trivial memory compare: compare x2 bytes starting at address x0 with
// bytes starting at address x1.
// Returns only if all bytes match; otherwise, the program is aborted.
// Clobbers x0-x5.
function memcmp
cbz x2, 1f
mov x5, #0
0: ldrb w3, [x0, x5]
ldrb w4, [x1, x5]
add x5, x5, #1
cmp w3, w4
b.ne barf
subs x2, x2, #1
b.ne 0b
1: ret
endfunction
// Verify that a FPSIMD V-register matches its shadow in memory, else abort
// x0: reg number
// Clobbers x0-x5.
function check_vreg
mov x3, x30
_adrv x5, x0, 6
mov x4, x0
ldr x7, =scratch
mov x0, x7
mov x1, x6
bl memfill_ae
mov x0, x4
mov x1, x7
bl getv
mov x0, x5
mov x1, x7
mov x2, x6
mov x30, x3
b memcmp
endfunction
// Any SVE register modified here can cause corruption in the main
// thread -- but *only* the registers modified here.
function irritator_handler
// Increment the irritation signal count (x23):
ldr x0, [x2, #ucontext_regs + 8 * 23]
add x0, x0, #1
str x0, [x2, #ucontext_regs + 8 * 23]
// Corrupt some random V-regs
adr x0, .text + (irritator_handler - .text) / 16 * 16
movi v0.8b, #7
movi v9.16b, #9
movi v31.8b, #31
ret
endfunction
function terminate_handler
mov w21, w0
mov x20, x2
puts "Terminated by signal "
mov w0, w21
bl putdec
puts ", no error, iterations="
ldr x0, [x20, #ucontext_regs + 8 * 22]
bl putdec
puts ", signals="
ldr x0, [x20, #ucontext_regs + 8 * 23]
bl putdecn
mov x0, #0
mov x8, #__NR_exit
svc #0
endfunction
// w0: signal number
// x1: sa_action
// w2: sa_flags
// Clobbers x0-x6,x8
function setsignal
str x30, [sp, #-((sa_sz + 15) / 16 * 16 + 16)]!
mov w4, w0
mov x5, x1
mov w6, w2
add x0, sp, #16
mov x1, #sa_sz
bl memclr
mov w0, w4
add x1, sp, #16
str w6, [x1, #sa_flags]
str x5, [x1, #sa_handler]
mov x2, #0
mov x3, #sa_mask_sz
mov x8, #__NR_rt_sigaction
svc #0
cbz w0, 1f
puts "sigaction failure\n"
b .Labort
1: ldr x30, [sp], #((sa_sz + 15) / 16 * 16 + 16)
ret
endfunction
// Main program entry point
.globl _start
function _start
_start:
// Sanity-check and report the vector length
mov x19, #128
cmp x19, #128
b.lo 1f
cmp x19, #2048
b.hi 1f
tst x19, #(8 - 1)
b.eq 2f
1: puts "Bad vector length: "
mov x0, x19
bl putdecn
b .Labort
2: puts "Vector length:\t"
mov x0, x19
bl putdec
puts " bits\n"
// Obtain our PID, to ensure test pattern uniqueness between processes
mov x8, #__NR_getpid
svc #0
mov x20, x0
puts "PID:\t"
mov x0, x20
bl putdecn
mov x23, #0 // Irritation signal count
mov w0, #SIGINT
adr x1, terminate_handler
mov w2, #SA_SIGINFO
bl setsignal
mov w0, #SIGTERM
adr x1, terminate_handler
mov w2, #SA_SIGINFO
bl setsignal
mov w0, #SIGUSR1
adr x1, irritator_handler
mov w2, #SA_SIGINFO
orr w2, w2, #SA_NODEFER
bl setsignal
mov x22, #0 // generation number, increments per iteration
.Ltest_loop:
mov x21, #0 // Set up V-regs & shadow with test pattern
0: mov x0, x20
mov x1, x21
and x2, x22, #0xf
bl setup_vreg
add x21, x21, #1
cmp x21, #NVR
b.lo 0b
// Can't do this when SVE state is volatile across SVC:
mov x8, #__NR_sched_yield // Encourage preemption
svc #0
mov x21, #0
0: mov x0, x21
bl check_vreg
add x21, x21, #1
cmp x21, #NVR
b.lo 0b
add x22, x22, #1
b .Ltest_loop
.Labort:
mov x0, #0
mov x1, #SIGABRT
mov x8, #__NR_kill
svc #0
endfunction
function barf
mov x10, x0 // expected data
mov x11, x1 // actual data
mov x12, x2 // data size
puts "Mistatch: PID="
mov x0, x20
bl putdec
puts ", iteration="
mov x0, x22
bl putdec
puts ", reg="
mov x0, x21
bl putdecn
puts "\tExpected ["
mov x0, x10
mov x1, x12
bl dumphex
puts "]\n\tGot ["
mov x0, x11
mov x1, x12
bl dumphex
puts "]\n"
mov x8, #__NR_exit
mov x1, #1
svc #0
endfunction

58
tools/testing/selftests/arm64/fp/sve-probe-vls.c Normal file
View File

@ -0,0 +1,58 @@
// SPDX-License-Identifier: GPL-2.0-only
/*
* Copyright (C) 2015-2020 ARM Limited.
* Original author: Dave Martin <Dave.Martin@arm.com>
*/
#include <assert.h>
#include <errno.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/auxv.h>
#include <sys/prctl.h>
#include <asm/sigcontext.h>
#include "../../kselftest.h"
int main(int argc, char **argv)
{
unsigned int vq;
int vl;
static unsigned int vqs[SVE_VQ_MAX];
unsigned int nvqs = 0;
ksft_print_header();
ksft_set_plan(2);
if (!(getauxval(AT_HWCAP) & HWCAP_SVE))
ksft_exit_skip("SVE not available");
/*
* Enumerate up to SVE_VQ_MAX vector lengths
*/
for (vq = SVE_VQ_MAX; vq > 0; --vq) {
vl = prctl(PR_SVE_SET_VL, vq * 16);
if (vl == -1)
ksft_exit_fail_msg("PR_SVE_SET_VL failed: %s (%d)\n",
strerror(errno), errno);
vl &= PR_SVE_VL_LEN_MASK;
if (!sve_vl_valid(vl))
ksft_exit_fail_msg("VL %d invalid\n", vl);
vq = sve_vq_from_vl(vl);
if (!(nvqs < SVE_VQ_MAX))
ksft_exit_fail_msg("Too many VLs %u >= SVE_VQ_MAX\n",
nvqs);
vqs[nvqs++] = vq;
}
ksft_test_result_pass("Enumerated %d vector lengths\n", nvqs);
ksft_test_result_pass("All vector lengths valid\n");
/* Print out the vector lengths in ascending order: */
while (nvqs--)
ksft_print_msg("%u\n", 16 * vqs[nvqs]);
ksft_exit_pass();
}

33
tools/testing/selftests/arm64/fp/sve-ptrace-asm.S Normal file
View File

@ -0,0 +1,33 @@
// SPDX-License-Identifier: GPL-2.0-only
// Copyright (C) 2015-2019 ARM Limited.
// Original author: Dave Martin <Dave.Martin@arm.com>
#include <asm/unistd.h>
.arch_extension sve
.globl sve_store_patterns
sve_store_patterns:
mov x1, x0
index z0.b, #0, #1
str q0, [x1]
mov w8, #__NR_getpid
svc #0
str q0, [x1, #0x10]
mov z1.d, z0.d
str q0, [x1, #0x20]
mov w8, #__NR_getpid
svc #0
str q0, [x1, #0x30]
mov z1.d, z0.d
str q0, [x1, #0x40]
ret
.size sve_store_patterns, . - sve_store_patterns
.type sve_store_patterns, @function

336
tools/testing/selftests/arm64/fp/sve-ptrace.c Normal file
View File

@ -0,0 +1,336 @@
// SPDX-License-Identifier: GPL-2.0-only
/*
* Copyright (C) 2015-2020 ARM Limited.
* Original author: Dave Martin <Dave.Martin@arm.com>
*/
#include <errno.h>
#include <stddef.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <sys/auxv.h>
#include <sys/ptrace.h>
#include <sys/types.h>
#include <sys/uio.h>
#include <sys/wait.h>
#include <asm/sigcontext.h>
#include <asm/ptrace.h>
#include "../../kselftest.h"
/* <linux/elf.h> and <sys/auxv.h> don't like each other, so: */
#ifndef NT_ARM_SVE
#define NT_ARM_SVE 0x405
#endif
/* Number of registers filled in by sve_store_patterns */
#define NR_VREGS 5
void sve_store_patterns(__uint128_t v[NR_VREGS]);
static void dump(const void *buf, size_t size)
{
size_t i;
const unsigned char *p = buf;
for (i = 0; i < size; ++i)
printf(" %.2x", *p++);
}
static int check_vregs(const __uint128_t vregs[NR_VREGS])
{
int i;
int ok = 1;
for (i = 0; i < NR_VREGS; ++i) {
printf("# v[%d]:", i);
dump(&vregs[i], sizeof vregs[i]);
putchar('\n');
if (vregs[i] != vregs[0])
ok = 0;
}
return ok;
}
static int do_child(void)
{
if (ptrace(PTRACE_TRACEME, -1, NULL, NULL))
ksft_exit_fail_msg("PTRACE_TRACEME", strerror(errno));
if (raise(SIGSTOP))
ksft_exit_fail_msg("raise(SIGSTOP)", strerror(errno));
return EXIT_SUCCESS;
}
static struct user_sve_header *get_sve(pid_t pid, void **buf, size_t *size)
{
struct user_sve_header *sve;
void *p;
size_t sz = sizeof *sve;
struct iovec iov;
while (1) {
if (*size < sz) {
p = realloc(*buf, sz);
if (!p) {
errno = ENOMEM;
goto error;
}
*buf = p;
*size = sz;
}
iov.iov_base = *buf;
iov.iov_len = sz;
if (ptrace(PTRACE_GETREGSET, pid, NT_ARM_SVE, &iov))
goto error;
sve = *buf;
if (sve->size <= sz)
break;
sz = sve->size;
}
return sve;
error:
return NULL;
}
static int set_sve(pid_t pid, const struct user_sve_header *sve)
{
struct iovec iov;
iov.iov_base = (void *)sve;
iov.iov_len = sve->size;
return ptrace(PTRACE_SETREGSET, pid, NT_ARM_SVE, &iov);
}
static void dump_sve_regs(const struct user_sve_header *sve, unsigned int num,
unsigned int vlmax)
{
unsigned int vq;
unsigned int i;
if ((sve->flags & SVE_PT_REGS_MASK) != SVE_PT_REGS_SVE)
ksft_exit_fail_msg("Dumping non-SVE register\n");
if (vlmax > sve->vl)
vlmax = sve->vl;
vq = sve_vq_from_vl(sve->vl);
for (i = 0; i < num; ++i) {
printf("# z%u:", i);
dump((const char *)sve + SVE_PT_SVE_ZREG_OFFSET(vq, i),
vlmax);
printf("%s\n", vlmax == sve->vl ? "" : " ...");
}
}
static int do_parent(pid_t child)
{
int ret = EXIT_FAILURE;
pid_t pid;
int status;
siginfo_t si;
void *svebuf = NULL, *newsvebuf;
size_t svebufsz = 0, newsvebufsz;
struct user_sve_header *sve, *new_sve;
struct user_fpsimd_state *fpsimd;
unsigned int i, j;
unsigned char *p;
unsigned int vq;
/* Attach to the child */
while (1) {
int sig;
pid = wait(&status);
if (pid == -1) {
perror("wait");
goto error;
}
/*
* This should never happen but it's hard to flag in
* the framework.
*/
if (pid != child)
continue;
if (WIFEXITED(status) || WIFSIGNALED(status))
ksft_exit_fail_msg("Child died unexpectedly\n");
ksft_test_result(WIFSTOPPED(status), "WIFSTOPPED(%d)\n",
status);
if (!WIFSTOPPED(status))
goto error;
sig = WSTOPSIG(status);
if (ptrace(PTRACE_GETSIGINFO, pid, NULL, &si)) {
if (errno == ESRCH)
goto disappeared;
if (errno == EINVAL) {
sig = 0; /* bust group-stop */
goto cont;
}
ksft_test_result_fail("PTRACE_GETSIGINFO: %s\n",
strerror(errno));
goto error;
}
if (sig == SIGSTOP && si.si_code == SI_TKILL &&
si.si_pid == pid)
break;
cont:
if (ptrace(PTRACE_CONT, pid, NULL, sig)) {
if (errno == ESRCH)
goto disappeared;
ksft_test_result_fail("PTRACE_CONT: %s\n",
strerror(errno));
goto error;
}
}
sve = get_sve(pid, &svebuf, &svebufsz);
if (!sve) {
int e = errno;
ksft_test_result_fail("get_sve: %s\n", strerror(errno));
if (e == ESRCH)
goto disappeared;
goto error;
} else {
ksft_test_result_pass("get_sve\n");
}
ksft_test_result((sve->flags & SVE_PT_REGS_MASK) == SVE_PT_REGS_FPSIMD,
"FPSIMD registers\n");
if ((sve->flags & SVE_PT_REGS_MASK) != SVE_PT_REGS_FPSIMD)
goto error;
fpsimd = (struct user_fpsimd_state *)((char *)sve +
SVE_PT_FPSIMD_OFFSET);
for (i = 0; i < 32; ++i) {
p = (unsigned char *)&fpsimd->vregs[i];
for (j = 0; j < sizeof fpsimd->vregs[i]; ++j)
p[j] = j;
}
if (set_sve(pid, sve)) {
int e = errno;
ksft_test_result_fail("set_sve(FPSIMD): %s\n",
strerror(errno));
if (e == ESRCH)
goto disappeared;
goto error;
}
vq = sve_vq_from_vl(sve->vl);
newsvebufsz = SVE_PT_SVE_ZREG_OFFSET(vq, 1);
new_sve = newsvebuf = malloc(newsvebufsz);
if (!new_sve) {
errno = ENOMEM;
perror(NULL);
goto error;
}
*new_sve = *sve;
new_sve->flags &= ~SVE_PT_REGS_MASK;
new_sve->flags |= SVE_PT_REGS_SVE;
memset((char *)new_sve + SVE_PT_SVE_ZREG_OFFSET(vq, 0),
0, SVE_PT_SVE_ZREG_SIZE(vq));
new_sve->size = SVE_PT_SVE_ZREG_OFFSET(vq, 1);
if (set_sve(pid, new_sve)) {
int e = errno;
ksft_test_result_fail("set_sve(ZREG): %s\n", strerror(errno));
if (e == ESRCH)
goto disappeared;
goto error;
}
new_sve = get_sve(pid, &newsvebuf, &newsvebufsz);
if (!new_sve) {
int e = errno;
ksft_test_result_fail("get_sve(ZREG): %s\n", strerror(errno));
if (e == ESRCH)
goto disappeared;
goto error;
}
ksft_test_result((new_sve->flags & SVE_PT_REGS_MASK) == SVE_PT_REGS_SVE,
"SVE registers\n");
if ((new_sve->flags & SVE_PT_REGS_MASK) != SVE_PT_REGS_SVE)
goto error;
dump_sve_regs(new_sve, 3, sizeof fpsimd->vregs[0]);
p = (unsigned char *)new_sve + SVE_PT_SVE_ZREG_OFFSET(vq, 1);
for (i = 0; i < sizeof fpsimd->vregs[0]; ++i) {
unsigned char expected = i;
if (__BYTE_ORDER == __BIG_ENDIAN)
expected = sizeof fpsimd->vregs[0] - 1 - expected;
ksft_test_result(p[i] == expected, "p[%d] == expected\n", i);
if (p[i] != expected)
goto error;
}
ret = EXIT_SUCCESS;
error:
kill(child, SIGKILL);
disappeared:
return ret;
}
int main(void)
{
int ret = EXIT_SUCCESS;
__uint128_t v[NR_VREGS];
pid_t child;
ksft_print_header();
ksft_set_plan(20);
if (!(getauxval(AT_HWCAP) & HWCAP_SVE))
ksft_exit_skip("SVE not available\n");
sve_store_patterns(v);
if (!check_vregs(v))
ksft_exit_fail_msg("Initial check_vregs() failed\n");
child = fork();
if (!child)
return do_child();
if (do_parent(child))
ret = EXIT_FAILURE;
ksft_print_cnts();
return 0;
}

59
tools/testing/selftests/arm64/fp/sve-stress Executable file
View File

@ -0,0 +1,59 @@
#!/bin/bash
# SPDX-License-Identifier: GPL-2.0-only
# Copyright (C) 2015-2019 ARM Limited.
# Original author: Dave Martin <Dave.Martin@arm.com>
set -ue
NR_CPUS=`nproc`
pids=
logs=
cleanup () {
trap - INT TERM CHLD
set +e
if [ -n "$pids" ]; then
kill $pids
wait $pids
pids=
fi
if [ -n "$logs" ]; then
cat $logs
rm $logs
logs=
fi
}
interrupt () {
cleanup
exit 0
}
child_died () {
cleanup
exit 1
}
trap interrupt INT TERM EXIT
for x in `seq 0 $((NR_CPUS * 4))`; do
log=`mktemp`
logs=$logs\ $log
./sve-test >$log &
pids=$pids\ $!
done
# Wait for all child processes to be created:
sleep 10
while :; do
kill -USR1 $pids
done &
pids=$pids\ $!
wait
exit 1

672
tools/testing/selftests/arm64/fp/sve-test.S Normal file
View File

@ -0,0 +1,672 @@
// SPDX-License-Identifier: GPL-2.0-only
// Copyright (C) 2015-2019 ARM Limited.
// Original author: Dave Martin <Dave.Martin@arm.com>
//
// Simple Scalable Vector Extension context switch test
// Repeatedly writes unique test patterns into each SVE register
// and reads them back to verify integrity.
//
// for x in `seq 1 NR_CPUS`; do sve-test & pids=$pids\ $! ; done
// (leave it running for as long as you want...)
// kill $pids
#include <asm/unistd.h>
#include "assembler.h"
#include "asm-offsets.h"
#define NZR 32
#define NPR 16
#define MAXVL_B (2048 / 8)
.arch_extension sve
.macro _sve_ldr_v zt, xn
ldr z\zt, [x\xn]
.endm
.macro _sve_str_v zt, xn
str z\zt, [x\xn]
.endm
.macro _sve_ldr_p pt, xn
ldr p\pt, [x\xn]
.endm
.macro _sve_str_p pt, xn
str p\pt, [x\xn]
.endm
// Generate accessor functions to read/write programmatically selected
// SVE registers.
// x0 is the register index to access
// x1 is the memory address to read from (getz,setp) or store to (setz,setp)
// All clobber x0-x2
define_accessor setz, NZR, _sve_ldr_v
define_accessor getz, NZR, _sve_str_v
define_accessor setp, NPR, _sve_ldr_p
define_accessor getp, NPR, _sve_str_p
// Print a single character x0 to stdout
// Clobbers x0-x2,x8
function putc
str x0, [sp, #-16]!
mov x0, #1 // STDOUT_FILENO
mov x1, sp
mov x2, #1
mov x8, #__NR_write
svc #0
add sp, sp, #16
ret
endfunction
// Print a NUL-terminated string starting at address x0 to stdout
// Clobbers x0-x3,x8
function puts
mov x1, x0
mov x2, #0
0: ldrb w3, [x0], #1
cbz w3, 1f
add x2, x2, #1
b 0b
1: mov w0, #1 // STDOUT_FILENO
mov x8, #__NR_write
svc #0
ret
endfunction
// Utility macro to print a literal string
// Clobbers x0-x4,x8
.macro puts string
.pushsection .rodata.str1.1, "aMS", 1
.L__puts_literal\@: .string "\string"
.popsection
ldr x0, =.L__puts_literal\@
bl puts
.endm
// Print an unsigned decimal number x0 to stdout
// Clobbers x0-x4,x8
function putdec
mov x1, sp
str x30, [sp, #-32]! // Result can't be > 20 digits
mov x2, #0
strb w2, [x1, #-1]! // Write the NUL terminator
mov x2, #10
0: udiv x3, x0, x2 // div-mod loop to generate the digits
msub x0, x3, x2, x0
add w0, w0, #'0'
strb w0, [x1, #-1]!
mov x0, x3
cbnz x3, 0b
ldrb w0, [x1]
cbnz w0, 1f
mov w0, #'0' // Print "0" for 0, not ""
strb w0, [x1, #-1]!
1: mov x0, x1
bl puts
ldr x30, [sp], #32
ret
endfunction
// Print an unsigned decimal number x0 to stdout, followed by a newline
// Clobbers x0-x5,x8
function putdecn
mov x5, x30
bl putdec
mov x0, #'\n'
bl putc
ret x5
endfunction
// Clobbers x0-x3,x8
function puthexb
str x30, [sp, #-0x10]!
mov w3, w0
lsr w0, w0, #4
bl puthexnibble
mov w0, w3
ldr x30, [sp], #0x10
// fall through to puthexnibble
endfunction
// Clobbers x0-x2,x8
function puthexnibble
and w0, w0, #0xf
cmp w0, #10
blo 1f
add w0, w0, #'a' - ('9' + 1)
1: add w0, w0, #'0'
b putc
endfunction
// x0=data in, x1=size in, clobbers x0-x5,x8
function dumphex
str x30, [sp, #-0x10]!
mov x4, x0
mov x5, x1
0: subs x5, x5, #1
b.lo 1f
ldrb w0, [x4], #1
bl puthexb
b 0b
1: ldr x30, [sp], #0x10
ret
endfunction
// Declare some storate space to shadow the SVE register contents:
.pushsection .text
.data
.align 4
zref:
.space MAXVL_B * NZR
pref:
.space MAXVL_B / 8 * NPR
ffrref:
.space MAXVL_B / 8
scratch:
.space MAXVL_B
.popsection
// Trivial memory copy: copy x2 bytes, starting at address x1, to address x0.
// Clobbers x0-x3
function memcpy
cmp x2, #0
b.eq 1f
0: ldrb w3, [x1], #1
strb w3, [x0], #1
subs x2, x2, #1
b.ne 0b
1: ret
endfunction
// Generate a test pattern for storage in SVE registers
// x0: pid (16 bits)
// x1: register number (6 bits)
// x2: generation (4 bits)
// These values are used to constuct a 32-bit pattern that is repeated in the
// scratch buffer as many times as will fit:
// bits 31:28 generation number (increments once per test_loop)
// bits 27:22 32-bit lane index
// bits 21:16 register number
// bits 15: 0 pid
function pattern
orr w1, w0, w1, lsl #16
orr w2, w1, w2, lsl #28
ldr x0, =scratch
mov w1, #MAXVL_B / 4
0: str w2, [x0], #4
add w2, w2, #(1 << 22)
subs w1, w1, #1
bne 0b
ret
endfunction
// Get the address of shadow data for SVE Z-register Z<xn>
.macro _adrz xd, xn, nrtmp
ldr \xd, =zref
rdvl x\nrtmp, #1
madd \xd, x\nrtmp, \xn, \xd
.endm
// Get the address of shadow data for SVE P-register P<xn - NZR>
.macro _adrp xd, xn, nrtmp
ldr \xd, =pref
rdvl x\nrtmp, #1
lsr x\nrtmp, x\nrtmp, #3
sub \xn, \xn, #NZR
madd \xd, x\nrtmp, \xn, \xd
.endm
// Set up test pattern in a SVE Z-register
// x0: pid
// x1: register number
// x2: generation
function setup_zreg
mov x4, x30
mov x6, x1
bl pattern
_adrz x0, x6, 2
mov x5, x0
ldr x1, =scratch
bl memcpy
mov x0, x6
mov x1, x5
bl setz
ret x4
endfunction
// Set up test pattern in a SVE P-register
// x0: pid
// x1: register number
// x2: generation
function setup_preg
mov x4, x30
mov x6, x1
bl pattern
_adrp x0, x6, 2
mov x5, x0
ldr x1, =scratch
bl memcpy
mov x0, x6
mov x1, x5
bl setp
ret x4
endfunction
// Set up test pattern in the FFR
// x0: pid
// x2: generation
// Beware: corrupts P0.
function setup_ffr
mov x4, x30
bl pattern
ldr x0, =ffrref
ldr x1, =scratch
rdvl x2, #1
lsr x2, x2, #3
bl memcpy
mov x0, #0
ldr x1, =ffrref
bl setp
wrffr p0.b
ret x4
endfunction
// Fill x1 bytes starting at x0 with 0xae (for canary purposes)
// Clobbers x1, x2.
function memfill_ae
mov w2, #0xae
b memfill
endfunction
// Fill x1 bytes starting at x0 with 0.
// Clobbers x1, x2.
function memclr
mov w2, #0
endfunction
// fall through to memfill
// Trivial memory fill: fill x1 bytes starting at address x0 with byte w2
// Clobbers x1
function memfill
cmp x1, #0
b.eq 1f
0: strb w2, [x0], #1
subs x1, x1, #1
b.ne 0b
1: ret
endfunction
// Trivial memory compare: compare x2 bytes starting at address x0 with
// bytes starting at address x1.
// Returns only if all bytes match; otherwise, the program is aborted.
// Clobbers x0-x5.
function memcmp
cbz x2, 2f
stp x0, x1, [sp, #-0x20]!
str x2, [sp, #0x10]
mov x5, #0
0: ldrb w3, [x0, x5]
ldrb w4, [x1, x5]
add x5, x5, #1
cmp w3, w4
b.ne 1f
subs x2, x2, #1
b.ne 0b
1: ldr x2, [sp, #0x10]
ldp x0, x1, [sp], #0x20
b.ne barf
2: ret
endfunction
// Verify that a SVE Z-register matches its shadow in memory, else abort
// x0: reg number
// Clobbers x0-x7.
function check_zreg
mov x3, x30
_adrz x5, x0, 6
mov x4, x0
ldr x7, =scratch
mov x0, x7
mov x1, x6
bl memfill_ae
mov x0, x4
mov x1, x7
bl getz
mov x0, x5
mov x1, x7
mov x2, x6
mov x30, x3
b memcmp
endfunction
// Verify that a SVE P-register matches its shadow in memory, else abort
// x0: reg number
// Clobbers x0-x7.
function check_preg
mov x3, x30
_adrp x5, x0, 6
mov x4, x0
ldr x7, =scratch
mov x0, x7
mov x1, x6
bl memfill_ae
mov x0, x4
mov x1, x7
bl getp
mov x0, x5
mov x1, x7
mov x2, x6
mov x30, x3
b memcmp
endfunction
// Verify that the FFR matches its shadow in memory, else abort
// Beware -- corrupts P0.
// Clobbers x0-x5.
function check_ffr
mov x3, x30
ldr x4, =scratch
rdvl x5, #1
lsr x5, x5, #3
mov x0, x4
mov x1, x5
bl memfill_ae
rdffr p0.b
mov x0, #0
mov x1, x4
bl getp
ldr x0, =ffrref
mov x1, x4
mov x2, x5
mov x30, x3
b memcmp
endfunction
// Any SVE register modified here can cause corruption in the main
// thread -- but *only* the registers modified here.
function irritator_handler
// Increment the irritation signal count (x23):
ldr x0, [x2, #ucontext_regs + 8 * 23]
add x0, x0, #1
str x0, [x2, #ucontext_regs + 8 * 23]
// Corrupt some random Z-regs
adr x0, .text + (irritator_handler - .text) / 16 * 16
movi v0.8b, #1
movi v9.16b, #2
movi v31.8b, #3
// And P0
rdffr p0.b
// And FFR
wrffr p15.b
ret
endfunction
function terminate_handler
mov w21, w0
mov x20, x2
puts "Terminated by signal "
mov w0, w21
bl putdec
puts ", no error, iterations="
ldr x0, [x20, #ucontext_regs + 8 * 22]
bl putdec
puts ", signals="
ldr x0, [x20, #ucontext_regs + 8 * 23]
bl putdecn
mov x0, #0
mov x8, #__NR_exit
svc #0
endfunction
// w0: signal number
// x1: sa_action
// w2: sa_flags
// Clobbers x0-x6,x8
function setsignal
str x30, [sp, #-((sa_sz + 15) / 16 * 16 + 16)]!
mov w4, w0
mov x5, x1
mov w6, w2
add x0, sp, #16
mov x1, #sa_sz
bl memclr
mov w0, w4
add x1, sp, #16
str w6, [x1, #sa_flags]
str x5, [x1, #sa_handler]
mov x2, #0
mov x3, #sa_mask_sz
mov x8, #__NR_rt_sigaction
svc #0
cbz w0, 1f
puts "sigaction failure\n"
b .Labort
1: ldr x30, [sp], #((sa_sz + 15) / 16 * 16 + 16)
ret
endfunction
// Main program entry point
.globl _start
function _start
_start:
// Sanity-check and report the vector length
rdvl x19, #8
cmp x19, #128
b.lo 1f
cmp x19, #2048
b.hi 1f
tst x19, #(8 - 1)
b.eq 2f
1: puts "Bad vector length: "
mov x0, x19
bl putdecn
b .Labort
2: puts "Vector length:\t"
mov x0, x19
bl putdec
puts " bits\n"
// Obtain our PID, to ensure test pattern uniqueness between processes
mov x8, #__NR_getpid
svc #0
mov x20, x0
puts "PID:\t"
mov x0, x20
bl putdecn
mov x23, #0 // Irritation signal count
mov w0, #SIGINT
adr x1, terminate_handler
mov w2, #SA_SIGINFO
bl setsignal
mov w0, #SIGTERM
adr x1, terminate_handler
mov w2, #SA_SIGINFO
bl setsignal
mov w0, #SIGUSR1
adr x1, irritator_handler
mov w2, #SA_SIGINFO
orr w2, w2, #SA_NODEFER
bl setsignal
mov x22, #0 // generation number, increments per iteration
.Ltest_loop:
rdvl x0, #8
cmp x0, x19
b.ne vl_barf
mov x21, #0 // Set up Z-regs & shadow with test pattern
0: mov x0, x20
mov x1, x21
and x2, x22, #0xf
bl setup_zreg
add x21, x21, #1
cmp x21, #NZR
b.lo 0b
mov x0, x20 // Set up FFR & shadow with test pattern
mov x1, #NZR + NPR
and x2, x22, #0xf
bl setup_ffr
0: mov x0, x20 // Set up P-regs & shadow with test pattern
mov x1, x21
and x2, x22, #0xf
bl setup_preg
add x21, x21, #1
cmp x21, #NZR + NPR
b.lo 0b
// Can't do this when SVE state is volatile across SVC:
// mov x8, #__NR_sched_yield // Encourage preemption
// svc #0
mov x21, #0
0: mov x0, x21
bl check_zreg
add x21, x21, #1
cmp x21, #NZR
b.lo 0b
0: mov x0, x21
bl check_preg
add x21, x21, #1
cmp x21, #NZR + NPR
b.lo 0b
bl check_ffr
add x22, x22, #1
b .Ltest_loop
.Labort:
mov x0, #0
mov x1, #SIGABRT
mov x8, #__NR_kill
svc #0
endfunction
function barf
// fpsimd.c acitivty log dump hack
// ldr w0, =0xdeadc0de
// mov w8, #__NR_exit
// svc #0
// end hack
mov x10, x0 // expected data
mov x11, x1 // actual data
mov x12, x2 // data size
puts "Mistatch: PID="
mov x0, x20
bl putdec
puts ", iteration="
mov x0, x22
bl putdec
puts ", reg="
mov x0, x21
bl putdecn
puts "\tExpected ["
mov x0, x10
mov x1, x12
bl dumphex
puts "]\n\tGot ["
mov x0, x11
mov x1, x12
bl dumphex
puts "]\n"
mov x8, #__NR_getpid
svc #0
// fpsimd.c acitivty log dump hack
// ldr w0, =0xdeadc0de
// mov w8, #__NR_exit
// svc #0
// ^ end of hack
mov x1, #SIGABRT
mov x8, #__NR_kill
svc #0
// mov x8, #__NR_exit
// mov x1, #1
// svc #0
endfunction
function vl_barf
mov x10, x0
puts "Bad active VL: "
mov x0, x10
bl putdecn
mov x8, #__NR_exit
mov x1, #1
svc #0
endfunction

155
tools/testing/selftests/arm64/fp/vlset.c Normal file
View File

@ -0,0 +1,155 @@
// SPDX-License-Identifier: GPL-2.0-only
/*
* Copyright (C) 2015-2019 ARM Limited.
* Original author: Dave Martin <Dave.Martin@arm.com>
*/
#define _GNU_SOURCE
#include <assert.h>
#include <errno.h>
#include <limits.h>
#include <stddef.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <getopt.h>
#include <unistd.h>
#include <sys/auxv.h>
#include <sys/prctl.h>
#include <asm/hwcap.h>
#include <asm/sigcontext.h>
static int inherit = 0;
static int no_inherit = 0;
static int force = 0;
static unsigned long vl;
static const struct option options[] = {
{ "force", no_argument, NULL, 'f' },
{ "inherit", no_argument, NULL, 'i' },
{ "max", no_argument, NULL, 'M' },
{ "no-inherit", no_argument, &no_inherit, 1 },
{ "help", no_argument, NULL, '?' },
{}
};
static char const *program_name;
static int parse_options(int argc, char **argv)
{
int c;
char *rest;
program_name = strrchr(argv[0], '/');
if (program_name)
++program_name;
else
program_name = argv[0];
while ((c = getopt_long(argc, argv, "Mfhi", options, NULL)) != -1)
switch (c) {
case 'M': vl = SVE_VL_MAX; break;
case 'f': force = 1; break;
case 'i': inherit = 1; break;
case 0: break;
default: goto error;
}
if (inherit && no_inherit)
goto error;
if (!vl) {
/* vector length */
if (optind >= argc)
goto error;
errno = 0;
vl = strtoul(argv[optind], &rest, 0);
if (*rest) {
vl = ULONG_MAX;
errno = EINVAL;
}
if (vl == ULONG_MAX && errno) {
fprintf(stderr, "%s: %s: %s\n",
program_name, argv[optind], strerror(errno));
goto error;
}
++optind;
}
/* command */
if (optind >= argc)
goto error;
return 0;
error:
fprintf(stderr,
"Usage: %s [-f | --force] "
"[-i | --inherit | --no-inherit] "
"{-M | --max | <vector length>} "
"<command> [<arguments> ...]\n",
program_name);
return -1;
}
int main(int argc, char **argv)
{
int ret = 126; /* same as sh(1) command-not-executable error */
long flags;
char *path;
int t, e;
if (parse_options(argc, argv))
return 2; /* same as sh(1) builtin incorrect-usage */
if (vl & ~(vl & PR_SVE_VL_LEN_MASK)) {
fprintf(stderr, "%s: Invalid vector length %lu\n",
program_name, vl);
return 2; /* same as sh(1) builtin incorrect-usage */
}
if (!(getauxval(AT_HWCAP) & HWCAP_SVE)) {
fprintf(stderr, "%s: Scalable Vector Extension not present\n",
program_name);
if (!force)
goto error;
fputs("Going ahead anyway (--force): "
"This is a debug option. Don't rely on it.\n",
stderr);
}
flags = PR_SVE_SET_VL_ONEXEC;
if (inherit)
flags |= PR_SVE_VL_INHERIT;
t = prctl(PR_SVE_SET_VL, vl | flags);
if (t < 0) {
fprintf(stderr, "%s: PR_SVE_SET_VL: %s\n",
program_name, strerror(errno));
goto error;
}
t = prctl(PR_SVE_GET_VL);
if (t == -1) {
fprintf(stderr, "%s: PR_SVE_GET_VL: %s\n",
program_name, strerror(errno));
goto error;
}
flags = PR_SVE_VL_LEN_MASK;
flags = t & ~flags;
assert(optind < argc);
path = argv[optind];
execvp(path, &argv[optind]);
e = errno;
if (errno == ENOENT)
ret = 127; /* same as sh(1) not-found error */
fprintf(stderr, "%s: %s: %s\n", program_name, path, strerror(e));
error:
return ret; /* same as sh(1) not-executable error */
}

2
tools/testing/selftests/arm64/pauth/.gitignore vendored Normal file
View File

@ -0,0 +1,2 @@
exec_target
pac

39
tools/testing/selftests/arm64/pauth/Makefile Normal file
View File

@ -0,0 +1,39 @@
# SPDX-License-Identifier: GPL-2.0
# Copyright (C) 2020 ARM Limited
# preserve CC value from top level Makefile
ifeq ($(CC),cc)
CC := $(CROSS_COMPILE)gcc
endif
CFLAGS += -mbranch-protection=pac-ret
# check if the compiler supports ARMv8.3 and branch protection with PAuth
pauth_cc_support := $(shell if ($(CC) $(CFLAGS) -march=armv8.3-a -E -x c /dev/null -o /dev/null 2>&1) then echo "1"; fi)
ifeq ($(pauth_cc_support),1)
TEST_GEN_PROGS := pac
TEST_GEN_FILES := pac_corruptor.o helper.o
TEST_GEN_PROGS_EXTENDED := exec_target
endif
include ../../lib.mk
ifeq ($(pauth_cc_support),1)
# pac* and aut* instructions are not available on architectures berfore
# ARMv8.3. Therefore target ARMv8.3 wherever they are used directly
$(OUTPUT)/pac_corruptor.o: pac_corruptor.S
$(CC) -c $^ -o $@ $(CFLAGS) -march=armv8.3-a
$(OUTPUT)/helper.o: helper.c
$(CC) -c $^ -o $@ $(CFLAGS) -march=armv8.3-a
# when -mbranch-protection is enabled and the target architecture is ARMv8.3 or
# greater, gcc emits pac* instructions which are not in HINT NOP space,
# preventing the tests from occurring at all. Compile for ARMv8.2 so tests can
# run on earlier targets and print a meaningful error messages
$(OUTPUT)/exec_target: exec_target.c $(OUTPUT)/helper.o
$(CC) $^ -o $@ $(CFLAGS) -march=armv8.2-a
$(OUTPUT)/pac: pac.c $(OUTPUT)/pac_corruptor.o $(OUTPUT)/helper.o
$(CC) $^ -o $@ $(CFLAGS) -march=armv8.2-a
endif

Some files were not shown because too many files have changed in this diff Show More