The Kconfig symbol for R10000 ll/sc errata workaround in the MIPS JIT was
misspelled, causing the workaround to not take effect when enabled.
Fixes: 72570224bb ("mips, bpf: Add JIT workarounds for CPU errata")
Reported-by: Lukas Bulwahn <lukas.bulwahn@gmail.com>
Signed-off-by: Johan Almbladh <johan.almbladh@anyfinetworks.com>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20211130160824.3781635-1-johan.almbladh@anyfinetworks.com
In emit_tail_call() of bpf_jit_comp32.c, "blez t2" (t2 <= 0) is
not consistent with the comment "t2 < 0", update the comment to
keep consistency.
Signed-off-by: Tiezhu Yang <yangtiezhu@loongson.cn>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Acked-by: Johan Almbladh <johan.almbladh@anyfinetworks.com>
Link: https://lore.kernel.org/bpf/1633915150-13220-3-git-send-email-yangtiezhu@loongson.cn
The config options MIPS_CBPF_JIT and MIPS_EBPF_JIT are useless, remove
them in arch/mips/Kconfig, and then modify arch/mips/net/Makefile.
Signed-off-by: Tiezhu Yang <yangtiezhu@loongson.cn>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Acked-by: Johan Almbladh <johan.almbladh@anyfinetworks.com>
Link: https://lore.kernel.org/bpf/1633915150-13220-2-git-send-email-yangtiezhu@loongson.cn
This patch shaves off a few instructions when loading sparse 64-bit
constants to register. The change is covered by additional tests in
lib/test_bpf.c.
Signed-off-by: Johan Almbladh <johan.almbladh@anyfinetworks.com>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Link: https://lore.kernel.org/bpf/20211007142828.634182-1-johan.almbladh@anyfinetworks.com
This patch removes the old 32-bit cBPF and 64-bit eBPF JIT implementations.
They are replaced by a new eBPF implementation that supports both 32-bit
and 64-bit MIPS CPUs.
Signed-off-by: Johan Almbladh <johan.almbladh@anyfinetworks.com>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20211005165408.2305108-8-johan.almbladh@anyfinetworks.com
This patch enables the new eBPF JITs for 32-bit and 64-bit MIPS. It also
disables the old cBPF JIT to so cBPF programs are converted to use the
new JIT.
Workarounds for R4000 CPU errata are not implemented by the JIT, so the
JIT is disabled if any of those workarounds are configured.
Signed-off-by: Johan Almbladh <johan.almbladh@anyfinetworks.com>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20211005165408.2305108-7-johan.almbladh@anyfinetworks.com
This patch adds workarounds for the following CPU errata to the MIPS
eBPF JIT, if enabled in the kernel configuration.
- R10000 ll/sc weak ordering
- Loongson-3 ll/sc weak ordering
- Loongson-2F jump hang
The Loongson-2F nop errata is implemented in uasm, which the JIT uses,
so no additional mitigations are needed for that.
Signed-off-by: Johan Almbladh <johan.almbladh@anyfinetworks.com>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Reviewed-by: Jiaxun Yang <jiaxun.yang@flygoat.com>
Link: https://lore.kernel.org/bpf/20211005165408.2305108-6-johan.almbladh@anyfinetworks.com
This is an implementation on of an eBPF JIT for 64-bit MIPS III-V and
MIPS64r1-r6. It uses the same framework introduced by the 32-bit JIT.
Signed-off-by: Johan Almbladh <johan.almbladh@anyfinetworks.com>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20211005165408.2305108-5-johan.almbladh@anyfinetworks.com
This is an implementation of an eBPF JIT for 32-bit MIPS I-V and MIPS32.
The implementation supports all 32-bit and 64-bit ALU and JMP operations,
including the recently-added atomics. 64-bit div/mod and 64-bit atomics
are implemented using function calls to math64 and atomic64 functions,
respectively. All 32-bit operations are implemented natively by the JIT,
except if the CPU lacks ll/sc instructions.
Register mapping
================
All 64-bit eBPF registers are mapped to native 32-bit MIPS register pairs,
and does not use any stack scratch space for register swapping. This means
that all eBPF register data is kept in CPU registers all the time, and
this simplifies the register management a lot. It also reduces the JIT's
pressure on temporary registers since we do not have to move data around.
Native register pairs are ordered according to CPU endiannes, following
the O32 calling convention for passing 64-bit arguments and return values.
The eBPF return value, arguments and callee-saved registers are mapped to
their native MIPS equivalents.
Since the 32 highest bits in the eBPF FP (frame pointer) register are
always zero, only one general-purpose register is actually needed for the
mapping. The MIPS fp register is used for this purpose. The high bits are
mapped to MIPS register r0. This saves us one CPU register, which is much
needed for temporaries, while still allowing us to treat the R10 (FP)
register just like any other eBPF register in the JIT.
The MIPS gp (global pointer) and at (assembler temporary) registers are
used as internal temporary registers for constant blinding. CPU registers
t6-t9 are used internally by the JIT when constructing more complex 64-bit
operations. This is precisely what is needed - two registers to store an
operand value, and two more as scratch registers when performing the
operation.
The register mapping is shown below.
R0 - $v1, $v0 return value
R1 - $a1, $a0 argument 1, passed in registers
R2 - $a3, $a2 argument 2, passed in registers
R3 - $t1, $t0 argument 3, passed on stack
R4 - $t3, $t2 argument 4, passed on stack
R5 - $t4, $t3 argument 5, passed on stack
R6 - $s1, $s0 callee-saved
R7 - $s3, $s2 callee-saved
R8 - $s5, $s4 callee-saved
R9 - $s7, $s6 callee-saved
FP - $r0, $fp 32-bit frame pointer
AX - $gp, $at constant-blinding
$t6 - $t9 unallocated, JIT temporaries
Jump offsets
============
The JIT tries to map all conditional JMP operations to MIPS conditional
PC-relative branches. The MIPS branch offset field is 18 bits, in bytes,
which is equivalent to the eBPF 16-bit instruction offset. However, since
the JIT may emit more than one CPU instruction per eBPF instruction, the
field width may overflow. If that happens, the JIT converts the long
conditional jump to a short PC-relative branch with the condition
inverted, jumping over a long unconditional absolute jmp (j).
This conversion will change the instruction offset mapping used for jumps,
and may in turn result in more branch offset overflows. The JIT therefore
dry-runs the translation until no more branches are converted and the
offsets do not change anymore. There is an upper bound on this of course,
and if the JIT hits that limit, the last two iterations are run with all
branches being converted.
Tail call count
===============
The current tail call count is stored in the 16-byte area of the caller's
stack frame that is reserved for the callee in the o32 ABI. The value is
initialized in the prologue, and propagated to the tail-callee by skipping
the initialization instructions when emitting the tail call.
Signed-off-by: Johan Almbladh <johan.almbladh@anyfinetworks.com>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Signed-off-by: Andrii Nakryiko <andrii@kernel.org>
Link: https://lore.kernel.org/bpf/20211005165408.2305108-4-johan.almbladh@anyfinetworks.com
In case of JITs, each of the JIT backends compiles the BPF nospec instruction
/either/ to a machine instruction which emits a speculation barrier /or/ to
/no/ machine instruction in case the underlying architecture is not affected
by Speculative Store Bypass or has different mitigations in place already.
This covers both x86 and (implicitly) arm64: In case of x86, we use 'lfence'
instruction for mitigation. In case of arm64, we rely on the firmware mitigation
as controlled via the ssbd kernel parameter. Whenever the mitigation is enabled,
it works for all of the kernel code with no need to provide any additional
instructions here (hence only comment in arm64 JIT). Other archs can follow
as needed. The BPF nospec instruction is specifically targeting Spectre v4
since i) we don't use a serialization barrier for the Spectre v1 case, and
ii) mitigation instructions for v1 and v4 might be different on some archs.
The BPF nospec is required for a future commit, where the BPF verifier does
annotate intermediate BPF programs with speculation barriers.
Co-developed-by: Piotr Krysiuk <piotras@gmail.com>
Co-developed-by: Benedict Schlueter <benedict.schlueter@rub.de>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Signed-off-by: Piotr Krysiuk <piotras@gmail.com>
Signed-off-by: Benedict Schlueter <benedict.schlueter@rub.de>
Acked-by: Alexei Starovoitov <ast@kernel.org>
A subsequent patch will add additional atomic operations. These new
operations will use the same opcode field as the existing XADD, with
the immediate discriminating different operations.
In preparation, rename the instruction mode BPF_ATOMIC and start
calling the zero immediate BPF_ADD.
This is possible (doesn't break existing valid BPF progs) because the
immediate field is currently reserved MBZ and BPF_ADD is zero.
All uses are removed from the tree but the BPF_XADD definition is
kept around to avoid breaking builds for people including kernel
headers.
Signed-off-by: Brendan Jackman <jackmanb@google.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Acked-by: Björn Töpel <bjorn.topel@gmail.com>
Link: https://lore.kernel.org/bpf/20210114181751.768687-5-jackmanb@google.com
The FIELD_SIZEOF() macro was redundant, and is being removed from the
kernel. Since commit c593642c8b ("treewide: Use sizeof_field() macro")
this is one of the last users of the old macro, so replace it.
Signed-off-by: Kees Cook <keescook@chromium.org>
Signed-off-by: Paul Burton <paulburton@kernel.org>
Cc: linux-kernel@vger.kernel.org
Cc: linux-mips@vger.kernel.org
Commit 716850ab10 ("MIPS: eBPF: Initial eBPF support for MIPS32
architecture.") enabled our eBPF JIT for MIPS32 kernels, whereas it has
previously only been availailable for MIPS64. It was my understanding at
the time that the BPF test suite was passing & JITing a comparable
number of tests to our cBPF JIT [1], but it turns out that was not the
case.
The eBPF JIT has a number of problems on MIPS32:
- Most notably various code paths still result in emission of MIPS64
instructions which will cause reserved instruction exceptions & kernel
panics when run on MIPS32 CPUs.
- The eBPF JIT doesn't account for differences between the O32 ABI used
by MIPS32 kernels versus the N64 ABI used by MIPS64 kernels. Notably
arguments beyond the first 4 are passed on the stack in O32, and this
is entirely unhandled when JITing a BPF_CALL instruction. Stack space
must be reserved for arguments even if they all fit in registers, and
the callee is free to assume that stack space has been reserved for
its use - with the eBPF JIT this is not the case, so calling any
function can result in clobbering values on the stack & unpredictable
behaviour. Function arguments in eBPF are always 64-bit values which
is also entirely unhandled - the JIT still uses a single (32-bit)
register per argument. As a result all function arguments are always
passed incorrectly when JITing a BPF_CALL instruction, leading to
kernel crashes or strange behavior.
- The JIT attempts to bail our on use of ALU64 instructions or 64-bit
memory access instructions. The code doing this at the start of
build_one_insn() incorrectly checks whether BPF_OP() equals BPF_DW,
when it should really be checking BPF_SIZE() & only doing so when
BPF_CLASS() is one of BPF_{LD,LDX,ST,STX}. This results in false
positives that cause more bailouts than intended, and that in turns
hides some of the problems described above.
- The kernel's cBPF->eBPF translation makes heavy use of 64-bit eBPF
instructions that the MIPS32 eBPF JIT bails out on, leading to most
cBPF programs not being JITed at all.
Until these problems are resolved, revert the removal of the cBPF JIT
performed by commit 716850ab10 ("MIPS: eBPF: Initial eBPF support for
MIPS32 architecture."). Together with commit f8fffebdea ("MIPS: BPF:
Disable MIPS32 eBPF JIT") this restores MIPS32 BPF JIT behavior back to
the same state it was prior to the introduction of the broken eBPF JIT
support.
[1] https://lore.kernel.org/linux-mips/MWHPR2201MB13583388481F01A422CE7D66D4410@MWHPR2201MB1358.namprd22.prod.outlook.com/
Signed-off-by: Paul Burton <paulburton@kernel.org>
Fixes: 716850ab10 ("MIPS: eBPF: Initial eBPF support for MIPS32 architecture.")
Cc: Daniel Borkmann <daniel@iogearbox.net>
Cc: Hassan Naveed <hnaveed@wavecomp.com>
Cc: Tony Ambardar <itugrok@yahoo.com>
Cc: bpf@vger.kernel.org
Cc: netdev@vger.kernel.org
Cc: linux-mips@vger.kernel.org
Cc: linux-kernel@vger.kernel.org
- Fill the struct cacheinfo shared_cpu_map field with sensible values,
notably avoiding issues with perf which was unhappy in the absence of
these values.
- A boot fix for Loongson 2E & 2F machines which was fallout from some
refactoring performed this cycle.
- A Kconfig dependency fix for the Loongson CPU HWMon driver.
- A couple of VDSO fixes, ensuring gettimeofday() behaves appropriately
for kernel configurations that don't include support for a clocksource
the VDSO can use & fixing the calling convention for the n32 & n64
VDSOs which would previously clobber the $gp/$28 register.
- A build fix for vmlinuz compressed images which were inappropriately
building with -fsanitize-coverage despite not being part of the kernel
proper, then failing to link due to the missing
__sanitizer_cov_trace_pc() function.
- A couple of eBPF JIT fixes, including disabling it for MIPS32 due to a
large number of issues with the code generated there & reflecting ISA
dependencies in Kconfig to enforce that systems which don't support
the JIT must include the interpreter.
-----BEGIN PGP SIGNATURE-----
iIwEABYIADQWIQRgLjeFAZEXQzy86/s+p5+stXUA3QUCXhDvJxYccGF1bGJ1cnRv
bkBrZXJuZWwub3JnAAoJED6nn6y1dQDdULcBAJCyB5cBMSvl14q7jEICUKIZacaW
OGYrhvpr6g9Dtg+fAP0WpYbIdY/JsydXlJC97CTm6saI5Oh2TDe6Cnl+kfloBw==
=dadp
-----END PGP SIGNATURE-----
Merge tag 'mips_fixes_5.5_1' of git://git.kernel.org/pub/scm/linux/kernel/git/mips/linux
Pull MIPS fixes from Paul Burton:
"A collection of MIPS fixes:
- Fill the struct cacheinfo shared_cpu_map field with sensible
values, notably avoiding issues with perf which was unhappy in the
absence of these values.
- A boot fix for Loongson 2E & 2F machines which was fallout from
some refactoring performed this cycle.
- A Kconfig dependency fix for the Loongson CPU HWMon driver.
- A couple of VDSO fixes, ensuring gettimeofday() behaves
appropriately for kernel configurations that don't include support
for a clocksource the VDSO can use & fixing the calling convention
for the n32 & n64 VDSOs which would previously clobber the $gp/$28
register.
- A build fix for vmlinuz compressed images which were
inappropriately building with -fsanitize-coverage despite not being
part of the kernel proper, then failing to link due to the missing
__sanitizer_cov_trace_pc() function.
- A couple of eBPF JIT fixes, including disabling it for MIPS32 due
to a large number of issues with the code generated there &
reflecting ISA dependencies in Kconfig to enforce that systems
which don't support the JIT must include the interpreter"
* tag 'mips_fixes_5.5_1' of git://git.kernel.org/pub/scm/linux/kernel/git/mips/linux:
MIPS: Avoid VDSO ABI breakage due to global register variable
MIPS: BPF: eBPF JIT: check for MIPS ISA compliance in Kconfig
MIPS: BPF: Disable MIPS32 eBPF JIT
MIPS: Prevent link failure with kcov instrumentation
MIPS: Kconfig: Use correct form for 'depends on'
mips: Fix gettimeofday() in the vdso library
MIPS: Fix boot on Fuloong2 systems
mips: cacheinfo: report shared CPU map
It is completely wrong to check for compile-time MIPS ISA revision in
the body of bpf_int_jit_compile() as it may lead to get MIPS JIT fully
omitted by the CC while the rest system will think that the JIT is
actually present and works [1].
We can check if the selected CPU really supports MIPS eBPF JIT at
configure time and avoid such situations when kernel can be built
without both JIT and interpreter, but with CONFIG_BPF_SYSCALL=y.
[1] https://lore.kernel.org/linux-mips/09d713a59665d745e21d021deeaebe0a@dlink.ru/
Fixes: 716850ab10 ("MIPS: eBPF: Initial eBPF support for MIPS32 architecture.")
Cc: <stable@vger.kernel.org> # v5.2+
Signed-off-by: Alexander Lobakin <alobakin@dlink.ru>
Signed-off-by: Paul Burton <paulburton@kernel.org>
Cc: Ralf Baechle <ralf@linux-mips.org>
Cc: James Hogan <jhogan@kernel.org>
Cc: Hassan Naveed <hnaveed@wavecomp.com>
Cc: Alexei Starovoitov <ast@kernel.org>
Cc: Daniel Borkmann <daniel@iogearbox.net>
Cc: Martin KaFai Lau <kafai@fb.com>
Cc: Song Liu <songliubraving@fb.com>
Cc: Yonghong Song <yhs@fb.com>
Cc: Andrii Nakryiko <andriin@fb.com>
Cc: linux-mips@vger.kernel.org
Cc: linux-kernel@vger.kernel.org
Cc: netdev@vger.kernel.org
Cc: bpf@vger.kernel.org
All BPF JIT compilers except RISC-V's and MIPS' enforce a 33-tail calls
limit at runtime. In addition, a test was recently added, in tailcalls2,
to check this limit.
This patch updates the tail call limit in MIPS' JIT compiler to allow
33 tail calls.
Fixes: b6bd53f9c4 ("MIPS: Add missing file for eBPF JIT.")
Reported-by: Mahshid Khezri <khezri.mahshid@gmail.com>
Signed-off-by: Paul Chaignon <paul.chaignon@orange.com>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Acked-by: Martin KaFai Lau <kafai@fb.com>
Link: https://lore.kernel.org/bpf/b8eb2caac1c25453c539248e56ca22f74b5316af.1575916815.git.paul.chaignon@gmail.com
Based on 1 normalized pattern(s):
this program is free software you can redistribute it and or modify
it under the terms of the gnu general public license as published by
the free software foundation version 2 of the license
extracted by the scancode license scanner the SPDX license identifier
GPL-2.0-only
has been chosen to replace the boilerplate/reference in 315 file(s).
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Reviewed-by: Allison Randal <allison@lohutok.net>
Reviewed-by: Armijn Hemel <armijn@tjaldur.nl>
Cc: linux-spdx@vger.kernel.org
Link: https://lkml.kernel.org/r/20190531190115.503150771@linutronix.de
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Add SPDX license identifiers to all Make/Kconfig files which:
- Have no license information of any form
These files fall under the project license, GPL v2 only. The resulting SPDX
license identifier is:
GPL-2.0-only
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
- A set of memblock initialization improvements thanks to Serge Semin,
tidying up after our conversion from bootmem to memblock back in
v4.20.
- Our eBPF JIT the previously supported only MIPS64r2 through MIPS64r5
is improved to also support MIPS64r6. Support for MIPS32 systems is
introduced, with the caveat that it only works for programs that don't
use 64 bit registers or operations - those will bail out & need to be
interpreted.
- Improvements to the allocation & configuration of our exception vector
that should fix issues seen on some platforms using recent versions of
U-Boot.
- Some minor improvements to code generated for jump labels, along with
enabling them by default for generic kernels.
-----BEGIN PGP SIGNATURE-----
iIsEABYIADMWIQRgLjeFAZEXQzy86/s+p5+stXUA3QUCXNNB2RUccGF1bC5idXJ0
b25AbWlwcy5jb20ACgkQPqefrLV1AN1zeAD/U/ScowcQE8ynoY97nA70d3UmbETH
YETUX5WcOfR65O8A/1hvMX8QJ1x87XUlNTkE6Gdh/itAZJpJWiSo3dnd1GoF
=L9IJ
-----END PGP SIGNATURE-----
Merge tag 'mips_5.2' of git://git.kernel.org/pub/scm/linux/kernel/git/mips/linux
Pull MIPS updates from Paul Burton:
- A set of memblock initialization improvements thanks to Serge Semin,
tidying up after our conversion from bootmem to memblock back in
v4.20.
- Our eBPF JIT the previously supported only MIPS64r2 through MIPS64r5
is improved to also support MIPS64r6. Support for MIPS32 systems is
introduced, with the caveat that it only works for programs that
don't use 64 bit registers or operations - those will bail out & need
to be interpreted.
- Improvements to the allocation & configuration of our exception
vector that should fix issues seen on some platforms using recent
versions of U-Boot.
- Some minor improvements to code generated for jump labels, along with
enabling them by default for generic kernels.
* tag 'mips_5.2' of git://git.kernel.org/pub/scm/linux/kernel/git/mips/linux: (27 commits)
mips: Manually call fdt_init_reserved_mem() method
mips: Make sure dt memory regions are valid
mips: Perform early low memory test
mips: Dump memblock regions for debugging
mips: Add reserve-nomap memory type support
mips: Use memblock to reserve the __nosave memory range
mips: Discard post-CMA-init foreach loop
mips: Reserve memory for the kernel image resources
MIPS: Remove duplicate EBase configuration
MIPS: Sync icache for whole exception vector
MIPS: Always allocate exception vector for MIPSr2+
MIPS: Use memblock_phys_alloc() for exception vector
mips: Combine memblock init and memory reservation loops
mips: Discard rudiments from bootmem_init
mips: Make sure kernel .bss exists in boot mem pool
mips: vdso: drop unnecessary cc-ldoption
Revert "MIPS: ralink: fix cpu clock of mt7621 and add dt clk devices"
MIPS: generic: Enable CONFIG_JUMP_LABEL
MIPS: jump_label: Use compact branches for >= r6
MIPS: jump_label: Remove redundant nops
...
Fix sparse warning:
arch/mips/net/ebpf_jit.c:196:5: warning:
symbol 'ebpf_to_mips_reg' was not declared. Should it be static?
Reported-by: Hulk Robot <hulkci@huawei.com>
Signed-off-by: YueHaibing <yuehaibing@huawei.com>
Acked-by: Yonghong Song <yhs@fb.com>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Currently MIPS32 supports a JIT for classic BPF only, not extended BPF.
This patch adds JIT support for extended BPF on MIPS32, so code is
actually JIT'ed instead of being only interpreted. Instructions with
64-bit operands are not supported at this point.
We can delete classic BPF because the kernel will translate classic BPF
programs into extended BPF and JIT them, eliminating the need for
classic BPF.
Signed-off-by: Hassan Naveed <hnaveed@wavecomp.com>
Signed-off-by: Paul Burton <paul.burton@mips.com>
Cc: kafai@fb.com
Cc: songliubraving@fb.com
Cc: yhs@fb.com
Cc: netdev@vger.kernel.org
Cc: bpf@vger.kernel.org
Cc: linux-mips@vger.kernel.org
Cc: Ralf Baechle <ralf@linux-mips.org>
Cc: James Hogan <jhogan@kernel.org>
Cc: Alexei Starovoitov <ast@kernel.org>
Cc: Daniel Borkmann <daniel@iogearbox.net>
Cc: open list:MIPS <linux-mips@linux-mips.org>
Cc: open list <linux-kernel@vger.kernel.org>
Currently eBPF support is available on MIPS64R2 only. Use MIPS64R6
variants of instructions like multiply, divide, movn, movz so eBPF
can run on the newer ISA. Also, we only need to check ISA revision
before JIT'ing code, because we know the CPU is running a 64-bit
kernel because eBPF JIT is only included in kernels with CONFIG_64BIT=y
due to Kconfig dependencies.
Signed-off-by: Hassan Naveed <hnaveed@wavecomp.com>
Signed-off-by: Paul Burton <paul.burton@mips.com>
Cc: kafai@fb.com
Cc: songliubraving@fb.com
Cc: yhs@fb.com
Cc: netdev@vger.kernel.org
Cc: bpf@vger.kernel.org
Cc: linux-mips@vger.kernel.org
Cc: Ralf Baechle <ralf@linux-mips.org>
Cc: James Hogan <jhogan@kernel.org>
Cc: Alexei Starovoitov <ast@kernel.org>
Cc: Daniel Borkmann <daniel@iogearbox.net>
Cc: open list:MIPS <linux-mips@linux-mips.org>
Cc: open list <linux-kernel@vger.kernel.org>
The MIPS eBPF JIT calls flush_icache_range() in order to ensure the
icache observes the code that we just wrote. Unfortunately it gets the
end address calculation wrong due to some bad pointer arithmetic.
The struct jit_ctx target field is of type pointer to u32, and as such
adding one to it will increment the address being pointed to by 4 bytes.
Therefore in order to find the address of the end of the code we simply
need to add the number of 4 byte instructions emitted, but we mistakenly
add the number of instructions multiplied by 4. This results in the call
to flush_icache_range() operating on a memory region 4x larger than
intended, which is always wasteful and can cause crashes if we overrun
into an unmapped page.
Fix this by correcting the pointer arithmetic to remove the bogus
multiplication, and use braces to remove the need for a set of brackets
whilst also making it obvious that the target field is a pointer.
Signed-off-by: Paul Burton <paul.burton@mips.com>
Fixes: b6bd53f9c4 ("MIPS: Add missing file for eBPF JIT.")
Cc: Alexei Starovoitov <ast@kernel.org>
Cc: Daniel Borkmann <daniel@iogearbox.net>
Cc: Martin KaFai Lau <kafai@fb.com>
Cc: Song Liu <songliubraving@fb.com>
Cc: Yonghong Song <yhs@fb.com>
Cc: netdev@vger.kernel.org
Cc: bpf@vger.kernel.org
Cc: linux-mips@vger.kernel.org
Cc: stable@vger.kernel.org # v4.13+
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
REG_32BIT_ZERO_EX and REG_64BIT are always handled in exactly the same
way, and reg_val_propagate_range() never actually sets any register to
type REG_32BIT_ZERO_EX.
Remove the redundant & unused REG_32BIT_ZERO_EX.
Signed-off-by: Paul Burton <paul.burton@mips.com>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
The function prototype used to call JITed eBPF code (ie. the type of the
struct bpf_prog bpf_func field) returns an unsigned int. The MIPS n64
ABI that MIPS64 kernels target defines that 32 bit integers should
always be sign extended when passed in registers as either arguments or
return values.
This means that when returning any value which may not already be sign
extended (ie. of type REG_64BIT or REG_32BIT_ZERO_EX) we need to perform
that sign extension in order to comply with the n64 ABI. Without this we
see strange looking test failures from test_bpf.ko, such as:
test_bpf: #65 ALU64_MOV_X:
dst = 4294967295 jited:1 ret -1 != -1 FAIL (1 times)
Although the return value printed matches the expected value, this is
only because printf is only examining the least significant 32 bits of
the 64 bit register value we returned. The register holding the expected
value is sign extended whilst the v0 register was set to a zero extended
value by our JITed code, so when compared by a conditional branch
instruction the values are not equal.
We already handle this when the return value register is of type
REG_32BIT_ZERO_EX, so simply extend this to also cover REG_64BIT.
Signed-off-by: Paul Burton <paul.burton@mips.com>
Fixes: b6bd53f9c4 ("MIPS: Add missing file for eBPF JIT.")
Cc: stable@vger.kernel.org # v4.13+
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Jitting of BPF_K is supported already, but not BPF_X. This patch complete
the support for the latter on both MIPS and microMIPS.
Cc: Paul Burton <paul.burton@mips.com>
Cc: linux-mips@vger.kernel.org
Acked-by: Paul Burton <paul.burton@mips.com>
Signed-off-by: Jiong Wang <jiong.wang@netronome.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Replace VLAN_TAG_PRESENT with single bit flag and free up
VLAN.CFI overload. Now VLAN.CFI is visible in networking stack
and can be passed around intact.
Signed-off-by: Michał Mirosław <mirq-linux@rere.qmqm.pl>
Signed-off-by: David S. Miller <davem@davemloft.net>
The ool_skb_header_pointer() and size_to_len() is unused same as
tmp_offset, therefore remove all of them.
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Since LD_ABS/LD_IND instructions are now removed from the core and
reimplemented through a combination of inlined BPF instructions and
a slow-path helper, we can get rid of the complexity from mips64 JIT.
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Acked-by: Alexei Starovoitov <ast@kernel.org>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Remove the need to check that __mips_isa_rev is defined by using the
newly added MIPS_ISA_REV.
Signed-off-by: Matt Redfearn <matt.redfearn@mips.com>
Cc: Ralf Baechle <ralf@linux-mips.org>
Cc: Paul Burton <paul.burton@mips.com>
Cc: David Daney <david.daney@cavium.com>
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/18677/
Signed-off-by: James Hogan <jhogan@kernel.org>
The verifier in both cBPF and eBPF reject div/mod by 0 imm,
so this can never load. Remove emitting such test and reject
it from being JITed instead (the latter is actually also not
needed, but given practice in sparc64, ppc64 today, so
doesn't hurt to add it here either).
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Cc: David Daney <david.daney@cavium.com>
Reviewed-by: David Daney <david.daney@cavium.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Since we've changed div/mod exception handling for src_reg in
eBPF verifier itself, remove the leftovers from mips64 JIT.
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Cc: David Daney <david.daney@cavium.com>
Reviewed-by: David Daney <david.daney@cavium.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Having a pure_initcall() callback just to permanently enable BPF
JITs under CONFIG_BPF_JIT_ALWAYS_ON is unnecessary and could leave
a small race window in future where JIT is still disabled on boot.
Since we know about the setting at compilation time anyway, just
initialize it properly there. Also consolidate all the individual
bpf_jit_enable variables into a single one and move them under one
location. Moreover, don't allow for setting unspecified garbage
values on them.
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Acked-by: Alexei Starovoitov <ast@kernel.org>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
global bpf_jit_enable variable is tested multiple times in JITs,
blinding and verifier core. The malicious root can try to toggle
it while loading the programs. This race condition was accounted
for and there should be no issues, but it's safer to avoid
this race condition.
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Acked-by: Daniel Borkmann <daniel@iogearbox.net>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Compiling ebpf_jit.c with gcc 4.9 results in a (likely spurious)
compiler warning, as gcc has detected that the variable "target" may be
used uninitialised. Since -Werror is active, this is treated as an error
and causes a kernel build failure whenever CONFIG_MIPS_EBPF_JIT is
enabled.
arch/mips/net/ebpf_jit.c: In function 'build_one_insn':
arch/mips/net/ebpf_jit.c:1118:80: error: 'target' may be used
uninitialized in this function [-Werror=maybe-uninitialized]
emit_instr(ctx, j, target);
^
cc1: all warnings being treated as errors
Fix this by initialising "target" to 0. If it really is used
uninitialised this would result in a jump to 0 and a detectable run time
failure.
Signed-off-by: Matt Redfearn <matt.redfearn@imgtec.com>
Fixes: b6bd53f9c4 ("MIPS: Add missing file for eBPF JIT.")
Cc: James Hogan <james.hogan@imgtec.com>
Cc: David Daney <david.daney@cavium.com>
Cc: David S. Miller <davem@davemloft.net>
Cc: Colin Ian King <colin.king@canonical.com>
Cc: Daniel Borkmann <daniel@iogearbox.net>
Cc: linux-mips@linux-mips.org
Cc: linux-kernel@vger.kernel.org
Cc: <stable@vger.kernel.org> # v4.13+
Patchwork: https://patchwork.linux-mips.org/patch/17375/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
There is a missing break causing a fall-through and setting
ctx.use_bbit_insns to the wrong value. Fix this by adding the
missing break.
Detected with cppcheck:
"Variable 'ctx.use_bbit_insns' is reassigned a value before the old
one has been used. 'break;' missing?"
Fixes: 8d8d18c328 ("MIPS,bpf: Fix using smp_processor_id() in preemptible splat.")
Signed-off-by: Colin Ian King <colin.king@canonical.com>
Acked-by: David Daney <david.daney@cavium.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
The code looks a little cleaner if we replace BPF_OP(insn->code) with
the local variable bpf_op. Caching the value this way also saves 300
bytes (about 1%) in the code size of the JIT.
Signed-off-by: David Daney <david.daney@cavium.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
If the kernel is configured with preemption enabled we were getting
warning stack traces for use of current_cpu_type().
Fix by moving the test between preempt_disable()/preempt_enable() and
caching the results of the CPU type tests for use during code
generation.
Signed-off-by: David Daney <david.daney@cavium.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
Inexplicably, commit f381bf6d82 ("MIPS: Add support for eBPF JIT.")
lost a file somewhere on its path to Linus' tree. Add back the
missing ebpf_jit.c so that we can build with CONFIG_BPF_JIT selected.
This version of ebpf_jit.c is identical to the original except for two
minor change need to resolve conflicts with changes merged from the
BPF branch:
A) Set prog->jited_len = image_size;
B) Use BPF_TAIL_CALL instead of BPF_CALL | BPF_X
Fixes: f381bf6d82 ("MIPS: Add support for eBPF JIT.")
Signed-off-by: David Daney <david.daney@cavium.com>
Acked-by: Daniel Borkmann <daniel@iogearbox.net>
Signed-off-by: David S. Miller <davem@davemloft.net>
Since the eBPF machine has 64-bit registers, we only support this in
64-bit kernels. As of the writing of this commit log test-bpf is showing:
test_bpf: Summary: 316 PASSED, 0 FAILED, [308/308 JIT'ed]
All current test cases are successfully compiled.
Many examples in samples/bpf are usable, specifically tracex5 which
uses tail calls works.
Signed-off-by: David Daney <david.daney@cavium.com>
Cc: Alexei Starovoitov <ast@kernel.org>
Cc: Daniel Borkmann <daniel@iogearbox.net>
Cc: Markos Chandras <markos.chandras@imgtec.com>
Cc: Matt Redfearn <matt.redfearn@imgtec.com>
Cc: netdev@vger.kernel.org
Cc: linux-kernel@vger.kernel.org
Cc: linux-mips@linux-mips.org
Patchwork: https://patchwork.linux-mips.org/patch/16369/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
o Socket data is unsigned, so use unsigned accessors instructions.
o Fix path result pointer generation arithmetic.
o Fix half-word byte swapping code for unsigned semantics.
Signed-off-by: David Daney <david.daney@cavium.com>
Cc: James Hogan <james.hogan@imgtec.com>
Cc: Alexei Starovoitov <ast@kernel.org>
Cc: Steven J. Hill <steven.hill@cavium.com>
Cc: linux-mips@linux-mips.org
Cc: netdev@vger.kernel.org
Cc: linux-kernel@vger.kernel.org
Patchwork: https://patchwork.linux-mips.org/patch/15747/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
If bpf_needs_clear_a() returns true, only actually clear it if it is
ever used. If it is not used, we don't save and restore it, so the
clearing has the nasty side effect of clobbering caller state.
Also, don't emit stack pointer adjustment instructions if the
adjustment amount is zero.
Signed-off-by: David Daney <david.daney@cavium.com>
Cc: James Hogan <james.hogan@imgtec.com>
Cc: Alexei Starovoitov <ast@kernel.org>
Cc: Steven J. Hill <steven.hill@cavium.com>
Cc: linux-mips@linux-mips.org
Cc: netdev@vger.kernel.org
Cc: linux-kernel@vger.kernel.org
Patchwork: https://patchwork.linux-mips.org/patch/15745/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
The SKB vlan_tci and queue_mapping fields are unsigned, don't sign
extend these in the BPF JIT. In the vlan_tci case, the value gets
masked so the change is not needed for correctness, but do it anyway
for agreement with the types defined in struct sk_buff.
Signed-off-by: David Daney <david.daney@cavium.com>
Cc: James Hogan <james.hogan@imgtec.com>
Cc: Alexei Starovoitov <ast@kernel.org>
Cc: Steven J. Hill <steven.hill@cavium.com>
Cc: linux-mips@linux-mips.org
Cc: netdev@vger.kernel.org
Cc: linux-kernel@vger.kernel.org
Patchwork: https://patchwork.linux-mips.org/patch/15746/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
This let's us pass some additional "modprobe test-bpf" tests with JIT
enabled.
Reuse the code for SKF_AD_IFINDEX, but substitute the offset and size
of the "type" field.
Signed-off-by: David Daney <david.daney@cavium.com>
Cc: James Hogan <james.hogan@imgtec.com>
Cc: Alexei Starovoitov <ast@kernel.org>
Cc: Steven J. Hill <steven.hill@cavium.com>
Cc: linux-mips@linux-mips.org
Cc: netdev@vger.kernel.org
Cc: linux-kernel@vger.kernel.org
Patchwork: https://patchwork.linux-mips.org/patch/15744/
Signed-off-by: Ralf Baechle <ralf@linux-mips.org>
Kernel source files need not include <linux/kconfig.h> explicitly
because the top Makefile forces to include it with:
-include $(srctree)/include/linux/kconfig.h
This commit removes explicit includes except the following:
* arch/s390/include/asm/facilities_src.h
* tools/testing/radix-tree/linux/kernel.h
These two are used for host programs.
Link: http://lkml.kernel.org/r/1473656164-11929-1-git-send-email-yamada.masahiro@socionext.com
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
