Minki/linux
1
0
Fork 1
mirror of https://github.com/torvalds/linux.git synced 2024-09-20 06:53:04 +00:00
Code Issues Packages Projects Releases Wiki Activity

bpf-next-for-netdev

Browse Source 
-----BEGIN PGP SIGNATURE-----
 
 iHUEABYIAB0WIQTFp0I1jqZrAX+hPRXbK58LschIgwUCZoxN0AAKCRDbK58LschI
 g0c5AQDa3ZV9gfbN42y1zSDoM1uOgO60fb+ydxyOYh8l3+OiQQD/fLfpTY3gBFSY
 9yi/pZhw/QdNzQskHNIBrHFGtJbMxgs=
 =p1Zz
 -----END PGP SIGNATURE-----

Merge tag 'for-netdev' of https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next

Daniel Borkmann says:

====================
pull-request: bpf-next 2024-07-08

The following pull-request contains BPF updates for your *net-next* tree.

We've added 102 non-merge commits during the last 28 day(s) which contain
a total of 127 files changed, 4606 insertions(+), 980 deletions(-).

The main changes are:

1) Support resilient split BTF which cuts down on duplication and makes BTF
   as compact as possible wrt BTF from modules, from Alan Maguire & Eduard Zingerman.

2) Add support for dumping kfunc prototypes from BTF which enables both detecting
   as well as dumping compilable prototypes for kfuncs, from Daniel Xu.

3) Batch of s390x BPF JIT improvements to add support for BPF arena and to implement
   support for BPF exceptions, from Ilya Leoshkevich.

4) Batch of riscv64 BPF JIT improvements in particular to add 12-argument support
   for BPF trampolines and to utilize bpf_prog_pack for the latter, from Pu Lehui.

5) Extend BPF test infrastructure to add a CHECKSUM_COMPLETE validation option
   for skbs and add coverage along with it, from Vadim Fedorenko.

6) Inline bpf_get_current_task/_btf() helpers in the arm64 BPF JIT which gives
   a small 1% performance improvement in micro-benchmarks, from Puranjay Mohan.

7) Extend the BPF verifier to track the delta between linked registers in order
   to better deal with recent LLVM code optimizations, from Alexei Starovoitov.

8) Fix bpf_wq_set_callback_impl() kfunc signature where the third argument should
   have been a pointer to the map value, from Benjamin Tissoires.

9) Extend BPF selftests to add regular expression support for test output matching
   and adjust some of the selftest when compiled under gcc, from Cupertino Miranda.

10) Simplify task_file_seq_get_next() and remove an unnecessary loop which always
    iterates exactly once anyway, from Dan Carpenter.

11) Add the capability to offload the netfilter flowtable in XDP layer through
    kfuncs, from Florian Westphal & Lorenzo Bianconi.

12) Various cleanups in networking helpers in BPF selftests to shave off a few
    lines of open-coded functions on client/server handling, from Geliang Tang.

13) Properly propagate prog->aux->tail_call_reachable out of BPF verifier, so
    that x86 JIT does not need to implement detection, from Leon Hwang.

14) Fix BPF verifier to add a missing check_func_arg_reg_off() to prevent an
    out-of-bounds memory access for dynpointers, from Matt Bobrowski.

15) Fix bpf_session_cookie() kfunc to return __u64 instead of long pointer as
    it might lead to problems on 32-bit archs, from Jiri Olsa.

16) Enhance traffic validation and dynamic batch size support in xsk selftests,
    from Tushar Vyavahare.

bpf-next-for-netdev

* tag 'for-netdev' of https://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next: (102 commits)
  selftests/bpf: DENYLIST.aarch64: Remove fexit_sleep
  selftests/bpf: amend for wrong bpf_wq_set_callback_impl signature
  bpf: helpers: fix bpf_wq_set_callback_impl signature
  libbpf: Add NULL checks to bpf_object__{prev_map,next_map}
  selftests/bpf: Remove exceptions tests from DENYLIST.s390x
  s390/bpf: Implement exceptions
  s390/bpf: Change seen_reg to a mask
  bpf: Remove unnecessary loop in task_file_seq_get_next()
  riscv, bpf: Optimize stack usage of trampoline
  bpf, devmap: Add .map_alloc_check
  selftests/bpf: Remove arena tests from DENYLIST.s390x
  selftests/bpf: Add UAF tests for arena atomics
  selftests/bpf: Introduce __arena_global
  s390/bpf: Support arena atomics
  s390/bpf: Enable arena
  s390/bpf: Support address space cast instruction
  s390/bpf: Support BPF_PROBE_MEM32
  s390/bpf: Land on the next JITed instruction after exception
  s390/bpf: Introduce pre- and post- probe functions
  s390/bpf: Get rid of get_probe_mem_regno()
  ...
====================

Link: https://patch.msgid.link/20240708221438.10974-1-daniel@iogearbox.net
Signed-off-by: Paolo Abeni <pabeni@redhat.com>
This commit is contained in:
Paolo Abeni 2024-07-09 17:01:46 +02:00
commit 7b769adc26
127 changed files with 4623 additions and 997 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

80
Documentation/bpf/standardization/instruction-set.rst
View File

@ -5,12 +5,19 @@
BPF Instruction Set Architecture (ISA)
======================================
eBPF (which is no longer an acronym for anything), also commonly
eBPF, also commonly
referred to as BPF, is a technology with origins in the Linux kernel
that can run untrusted programs in a privileged context such as an
operating system kernel. This document specifies the BPF instruction
set architecture (ISA).
As a historical note, BPF originally stood for Berkeley Packet Filter,
but now that it can do so much more than packet filtering, the acronym
no longer makes sense. BPF is now considered a standalone term that
does not stand for anything. The original BPF is sometimes referred to
as cBPF (classic BPF) to distinguish it from the now widely deployed
eBPF (extended BPF).
Documentation conventions
=========================
@ -18,7 +25,7 @@ The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
"OPTIONAL" in this document are to be interpreted as described in
BCP 14 `<https://www.rfc-editor.org/info/rfc2119>`_
`RFC8174 <https://www.rfc-editor.org/info/rfc8174>`_
`<https://www.rfc-editor.org/info/rfc8174>`_
when, and only when, they appear in all capitals, as shown here.
For brevity and consistency, this document refers to families
@ -59,24 +66,18 @@ numbers.
Functions
---------
* htobe16: Takes an unsigned 16-bit number in host-endian format and
returns the equivalent number as an unsigned 16-bit number in big-endian
format.
* htobe32: Takes an unsigned 32-bit number in host-endian format and
returns the equivalent number as an unsigned 32-bit number in big-endian
format.
* htobe64: Takes an unsigned 64-bit number in host-endian format and
returns the equivalent number as an unsigned 64-bit number in big-endian
format.
* htole16: Takes an unsigned 16-bit number in host-endian format and
returns the equivalent number as an unsigned 16-bit number in little-endian
format.
* htole32: Takes an unsigned 32-bit number in host-endian format and
returns the equivalent number as an unsigned 32-bit number in little-endian
format.
* htole64: Takes an unsigned 64-bit number in host-endian format and
returns the equivalent number as an unsigned 64-bit number in little-endian
format.
The following byteswap functions are direction-agnostic. That is,
the same function is used for conversion in either direction discussed
below.
* be16: Takes an unsigned 16-bit number and converts it between
host byte order and big-endian
(`IEN137 <https://www.rfc-editor.org/ien/ien137.txt>`_) byte order.
* be32: Takes an unsigned 32-bit number and converts it between
host byte order and big-endian byte order.
* be64: Takes an unsigned 64-bit number and converts it between
host byte order and big-endian byte order.
* bswap16: Takes an unsigned 16-bit number in either big- or little-endian
format and returns the equivalent number with the same bit width but
opposite endianness.
@ -86,7 +87,12 @@ Functions
* bswap64: Takes an unsigned 64-bit number in either big- or little-endian
format and returns the equivalent number with the same bit width but
opposite endianness.
* le16: Takes an unsigned 16-bit number and converts it between
host byte order and little-endian byte order.
* le32: Takes an unsigned 32-bit number and converts it between
host byte order and little-endian byte order.
* le64: Takes an unsigned 64-bit number and converts it between
host byte order and little-endian byte order.
Definitions
-----------
@ -437,8 +443,8 @@ and MUST be set to 0.
===== ======== ===== =================================================
class source value description
===== ======== ===== =================================================
ALU TO_LE 0 convert between host byte order and little endian
ALU TO_BE 1 convert between host byte order and big endian
ALU LE 0 convert between host byte order and little endian
ALU BE 1 convert between host byte order and big endian
ALU64 Reserved 0 do byte swap unconditionally
===== ======== ===== =================================================
@ -449,19 +455,19 @@ conformance group.
Examples:
``{END, TO_LE, ALU}`` with 'imm' = 16/32/64 means::
``{END, LE, ALU}`` with 'imm' = 16/32/64 means::
dst = htole16(dst)
dst = htole32(dst)
dst = htole64(dst)
dst = le16(dst)
dst = le32(dst)
dst = le64(dst)
``{END, TO_BE, ALU}`` with 'imm' = 16/32/64 means::
``{END, BE, ALU}`` with 'imm' = 16/32/64 means::
dst = htobe16(dst)
dst = htobe32(dst)
dst = htobe64(dst)
dst = be16(dst)
dst = be32(dst)
dst = be64(dst)
``{END, TO_LE, ALU64}`` with 'imm' = 16/32/64 means::
``{END, TO, ALU64}`` with 'imm' = 16/32/64 means::
dst = bswap16(dst)
dst = bswap32(dst)
@ -541,13 +547,17 @@ Helper functions are a concept whereby BPF programs can call into a
set of function calls exposed by the underlying platform.
Historically, each helper function was identified by a static ID
encoded in the 'imm' field. The available helper functions may differ
for each program type, but static IDs are unique across all program types.
encoded in the 'imm' field. Further documentation of helper functions
is outside the scope of this document and standardization is left for
future work, but use is widely deployed and more information can be
found in platform-specific documentation (e.g., Linux kernel documentation).
Platforms that support the BPF Type Format (BTF) support identifying
a helper function by a BTF ID encoded in the 'imm' field, where the BTF ID
identifies the helper name and type. Further documentation of BTF
is outside the scope of this document and is left for future work.
is outside the scope of this document and standardization is left for
future work, but use is widely deployed and more information can be
found in platform-specific documentation (e.g., Linux kernel documentation).
Program-local functions
~~~~~~~~~~~~~~~~~~~~~~~

12
arch/arm64/net/bpf_jit_comp.c
View File

@ -1244,6 +1244,13 @@ emit_cond_jmp:
break;
}
/* Implement helper call to bpf_get_current_task/_btf() inline */
if (insn->src_reg == 0 && (insn->imm == BPF_FUNC_get_current_task ||
insn->imm == BPF_FUNC_get_current_task_btf)) {
emit(A64_MRS_SP_EL0(r0), ctx);
break;
}
ret = bpf_jit_get_func_addr(ctx->prog, insn, extra_pass,
&func_addr, &func_addr_fixed);
if (ret < 0)
@ -1829,8 +1836,7 @@ skip_init_ctx:
prog->jited_len = 0;
goto out_free_hdr;
}
if (WARN_ON(bpf_jit_binary_pack_finalize(prog, ro_header,
header))) {
if (WARN_ON(bpf_jit_binary_pack_finalize(ro_header, header))) {
/* ro_header has been freed */
ro_header = NULL;
prog = orig_prog;
@ -2581,6 +2587,8 @@ bool bpf_jit_inlines_helper_call(s32 imm)
{
switch (imm) {
case BPF_FUNC_get_smp_processor_id:
case BPF_FUNC_get_current_task:
case BPF_FUNC_get_current_task_btf:
return true;
default:
return false;

4
arch/powerpc/net/bpf_jit_comp.c
View File

@ -225,7 +225,7 @@ skip_init_ctx:
fp->jited_len = proglen + FUNCTION_DESCR_SIZE;
if (!fp->is_func || extra_pass) {
if (bpf_jit_binary_pack_finalize(fp, fhdr, hdr)) {
if (bpf_jit_binary_pack_finalize(fhdr, hdr)) {
fp = org_fp;
goto out_addrs;
}
@ -348,7 +348,7 @@ void bpf_jit_free(struct bpf_prog *fp)
* before freeing it.
*/
if (jit_data) {
bpf_jit_binary_pack_finalize(fp, jit_data->fhdr, jit_data->hdr);
bpf_jit_binary_pack_finalize(jit_data->fhdr, jit_data->hdr);
kvfree(jit_data->addrs);
kfree(jit_data);
}

123
arch/riscv/net/bpf_jit_comp64.c
View File

@ -15,7 +15,10 @@
#include <asm/percpu.h>
#include "bpf_jit.h"
#define RV_MAX_REG_ARGS 8
#define RV_FENTRY_NINSNS 2
/* imm that allows emit_imm to emit max count insns */
#define RV_MAX_COUNT_IMM 0x7FFF7FF7FF7FF7FF
#define RV_REG_TCC RV_REG_A6
#define RV_REG_TCC_SAVED RV_REG_S6 /* Store A6 in S6 if program do calls */
@ -690,26 +693,45 @@ int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type poke_type,
return ret;
}
static void store_args(int nregs, int args_off, struct rv_jit_context *ctx)
static void store_args(int nr_arg_slots, int args_off, struct rv_jit_context *ctx)
{
int i;
for (i = 0; i < nregs; i++) {
emit_sd(RV_REG_FP, -args_off, RV_REG_A0 + i, ctx);
for (i = 0; i < nr_arg_slots; i++) {
if (i < RV_MAX_REG_ARGS) {
emit_sd(RV_REG_FP, -args_off, RV_REG_A0 + i, ctx);
} else {
/* skip slots for T0 and FP of traced function */
emit_ld(RV_REG_T1, 16 + (i - RV_MAX_REG_ARGS) * 8, RV_REG_FP, ctx);
emit_sd(RV_REG_FP, -args_off, RV_REG_T1, ctx);
}
args_off -= 8;
}
}
static void restore_args(int nregs, int args_off, struct rv_jit_context *ctx)
static void restore_args(int nr_reg_args, int args_off, struct rv_jit_context *ctx)
{
int i;
for (i = 0; i < nregs; i++) {
for (i = 0; i < nr_reg_args; i++) {
emit_ld(RV_REG_A0 + i, -args_off, RV_REG_FP, ctx);
args_off -= 8;
}
}
static void restore_stack_args(int nr_stack_args, int args_off, int stk_arg_off,
struct rv_jit_context *ctx)
{
int i;
for (i = 0; i < nr_stack_args; i++) {
emit_ld(RV_REG_T1, -(args_off - RV_MAX_REG_ARGS * 8), RV_REG_FP, ctx);
emit_sd(RV_REG_FP, -stk_arg_off, RV_REG_T1, ctx);
args_off -= 8;
stk_arg_off -= 8;
}
}
static int invoke_bpf_prog(struct bpf_tramp_link *l, int args_off, int retval_off,
int run_ctx_off, bool save_ret, struct rv_jit_context *ctx)
{
@ -782,8 +804,8 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im,
{
int i, ret, offset;
int *branches_off = NULL;
int stack_size = 0, nregs = m->nr_args;
int retval_off, args_off, nregs_off, ip_off, run_ctx_off, sreg_off;
int stack_size = 0, nr_arg_slots = 0;
int retval_off, args_off, nregs_off, ip_off, run_ctx_off, sreg_off, stk_arg_off;
struct bpf_tramp_links *fentry = &tlinks[BPF_TRAMP_FENTRY];
struct bpf_tramp_links *fexit = &tlinks[BPF_TRAMP_FEXIT];
struct bpf_tramp_links *fmod_ret = &tlinks[BPF_TRAMP_MODIFY_RETURN];
@ -829,20 +851,21 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im,
* FP - sreg_off [ callee saved reg ]
*
* [ pads ] pads for 16 bytes alignment
*
* [ stack_argN ]
* [ ... ]
* FP - stk_arg_off [ stack_arg1 ] BPF_TRAMP_F_CALL_ORIG
*/
if (flags & (BPF_TRAMP_F_ORIG_STACK | BPF_TRAMP_F_SHARE_IPMODIFY))
return -ENOTSUPP;
/* extra regiters for struct arguments */
for (i = 0; i < m->nr_args; i++)
if (m->arg_flags[i] & BTF_FMODEL_STRUCT_ARG)
nregs += round_up(m->arg_size[i], 8) / 8 - 1;
/* 8 arguments passed by registers */
if (nregs > 8)
if (m->nr_args > MAX_BPF_FUNC_ARGS)
return -ENOTSUPP;
for (i = 0; i < m->nr_args; i++)
nr_arg_slots += round_up(m->arg_size[i], 8) / 8;
/* room of trampoline frame to store return address and frame pointer */
stack_size += 16;
@ -852,7 +875,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im,
retval_off = stack_size;
}
stack_size += nregs * 8;
stack_size += nr_arg_slots * 8;
args_off = stack_size;
stack_size += 8;
@ -869,8 +892,14 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im,
stack_size += 8;
sreg_off = stack_size;
if ((flags & BPF_TRAMP_F_CALL_ORIG) && (nr_arg_slots - RV_MAX_REG_ARGS > 0))
stack_size += (nr_arg_slots - RV_MAX_REG_ARGS) * 8;
stack_size = round_up(stack_size, STACK_ALIGN);
/* room for args on stack must be at the top of stack */
stk_arg_off = stack_size;
if (!is_struct_ops) {
/* For the trampoline called from function entry,
* the frame of traced function and the frame of
@ -906,17 +935,17 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im,
emit_sd(RV_REG_FP, -ip_off, RV_REG_T1, ctx);
}
emit_li(RV_REG_T1, nregs, ctx);
emit_li(RV_REG_T1, nr_arg_slots, ctx);
emit_sd(RV_REG_FP, -nregs_off, RV_REG_T1, ctx);
store_args(nregs, args_off, ctx);
store_args(nr_arg_slots, args_off, ctx);
/* skip to actual body of traced function */
if (flags & BPF_TRAMP_F_SKIP_FRAME)
orig_call += RV_FENTRY_NINSNS * 4;
if (flags & BPF_TRAMP_F_CALL_ORIG) {
emit_imm(RV_REG_A0, (const s64)im, ctx);
emit_imm(RV_REG_A0, ctx->insns ? (const s64)im : RV_MAX_COUNT_IMM, ctx);
ret = emit_call((const u64)__bpf_tramp_enter, true, ctx);
if (ret)
return ret;
@ -949,13 +978,14 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im,
}
if (flags & BPF_TRAMP_F_CALL_ORIG) {
restore_args(nregs, args_off, ctx);
restore_args(min_t(int, nr_arg_slots, RV_MAX_REG_ARGS), args_off, ctx);
restore_stack_args(nr_arg_slots - RV_MAX_REG_ARGS, args_off, stk_arg_off, ctx);
ret = emit_call((const u64)orig_call, true, ctx);
if (ret)
goto out;
emit_sd(RV_REG_FP, -retval_off, RV_REG_A0, ctx);
emit_sd(RV_REG_FP, -(retval_off - 8), regmap[BPF_REG_0], ctx);
im->ip_after_call = ctx->insns + ctx->ninsns;
im->ip_after_call = ctx->ro_insns + ctx->ninsns;
/* 2 nops reserved for auipc+jalr pair */
emit(rv_nop(), ctx);
emit(rv_nop(), ctx);
@ -976,15 +1006,15 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im,
}
if (flags & BPF_TRAMP_F_CALL_ORIG) {
im->ip_epilogue = ctx->insns + ctx->ninsns;
emit_imm(RV_REG_A0, (const s64)im, ctx);
im->ip_epilogue = ctx->ro_insns + ctx->ninsns;
emit_imm(RV_REG_A0, ctx->insns ? (const s64)im : RV_MAX_COUNT_IMM, ctx);
ret = emit_call((const u64)__bpf_tramp_exit, true, ctx);
if (ret)
goto out;
}
if (flags & BPF_TRAMP_F_RESTORE_REGS)
restore_args(nregs, args_off, ctx);
restore_args(min_t(int, nr_arg_slots, RV_MAX_REG_ARGS), args_off, ctx);
if (save_ret) {
emit_ld(RV_REG_A0, -retval_off, RV_REG_FP, ctx);
@ -1039,31 +1069,52 @@ int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags,
return ret < 0 ? ret : ninsns_rvoff(ctx.ninsns);
}
int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image,
void *image_end, const struct btf_func_model *m,
void *arch_alloc_bpf_trampoline(unsigned int size)
{
return bpf_prog_pack_alloc(size, bpf_fill_ill_insns);
}
void arch_free_bpf_trampoline(void *image, unsigned int size)
{
bpf_prog_pack_free(image, size);
}
int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *ro_image,
void *ro_image_end, const struct btf_func_model *m,
u32 flags, struct bpf_tramp_links *tlinks,
void *func_addr)
{
int ret;
void *image, *res;
struct rv_jit_context ctx;
u32 size = ro_image_end - ro_image;
image = kvmalloc(size, GFP_KERNEL);
if (!image)
return -ENOMEM;
ctx.ninsns = 0;
/*
* The bpf_int_jit_compile() uses a RW buffer (ctx.insns) to write the
* JITed instructions and later copies it to a RX region (ctx.ro_insns).
* It also uses ctx.ro_insns to calculate offsets for jumps etc. As the
* trampoline image uses the same memory area for writing and execution,
* both ctx.insns and ctx.ro_insns can be set to image.
*/
ctx.insns = image;
ctx.ro_insns = image;
ctx.ro_insns = ro_image;
ret = __arch_prepare_bpf_trampoline(im, m, tlinks, func_addr, flags, &ctx);
if (ret < 0)
return ret;
goto out;
bpf_flush_icache(ctx.insns, ctx.insns + ctx.ninsns);
if (WARN_ON(size < ninsns_rvoff(ctx.ninsns))) {
ret = -E2BIG;
goto out;
}
return ninsns_rvoff(ret);
res = bpf_arch_text_copy(ro_image, image, size);
if (IS_ERR(res)) {
ret = PTR_ERR(res);
goto out;
}
bpf_flush_icache(ro_image, ro_image_end);
out:
kvfree(image);
return ret < 0 ? ret : size;
}
int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx,

5
arch/riscv/net/bpf_jit_core.c
View File

@ -178,8 +178,7 @@ skip_init_ctx:
prog->jited_len = prog_size - cfi_get_offset();
if (!prog->is_func || extra_pass) {
if (WARN_ON(bpf_jit_binary_pack_finalize(prog, jit_data->ro_header,
jit_data->header))) {
if (WARN_ON(bpf_jit_binary_pack_finalize(jit_data->ro_header, jit_data->header))) {
/* ro_header has been freed */
jit_data->ro_header = NULL;
prog = orig_prog;
@ -258,7 +257,7 @@ void bpf_jit_free(struct bpf_prog *prog)
* before freeing it.
*/
if (jit_data) {
bpf_jit_binary_pack_finalize(prog, jit_data->ro_header, jit_data->header);
bpf_jit_binary_pack_finalize(jit_data->ro_header, jit_data->header);
kfree(jit_data);
}
hdr = bpf_jit_binary_pack_hdr(prog);

489
arch/s390/net/bpf_jit_comp.c
View File

@ -31,11 +31,12 @@
#include <asm/nospec-branch.h>
#include <asm/set_memory.h>
#include <asm/text-patching.h>
#include <asm/unwind.h>
#include "bpf_jit.h"
struct bpf_jit {
u32 seen; /* Flags to remember seen eBPF instructions */
u32 seen_reg[16]; /* Array to remember which registers are used */
u16 seen_regs; /* Mask to remember which registers are used */
u32 *addrs; /* Array with relative instruction addresses */
u8 *prg_buf; /* Start of program */
int size; /* Size of program and literal pool */
@ -53,6 +54,8 @@ struct bpf_jit {
int excnt; /* Number of exception table entries */
int prologue_plt_ret; /* Return address for prologue hotpatch PLT */
int prologue_plt; /* Start of prologue hotpatch PLT */
int kern_arena; /* Pool offset of kernel arena address */
u64 user_arena; /* User arena address */
};
#define SEEN_MEM BIT(0) /* use mem[] for temporary storage */
@ -60,6 +63,8 @@ struct bpf_jit {
#define SEEN_FUNC BIT(2) /* calls C functions */
#define SEEN_STACK (SEEN_FUNC | SEEN_MEM)
#define NVREGS 0xffc0 /* %r6-%r15 */
/*
* s390 registers
*/
@ -118,8 +123,8 @@ static inline void reg_set_seen(struct bpf_jit *jit, u32 b1)
{
u32 r1 = reg2hex[b1];
if (r1 >= 6 && r1 <= 15 && !jit->seen_reg[r1])
jit->seen_reg[r1] = 1;
if (r1 >= 6 && r1 <= 15)
jit->seen_regs |= (1 << r1);
}
#define REG_SET_SEEN(b1) \
@ -127,8 +132,6 @@ static inline void reg_set_seen(struct bpf_jit *jit, u32 b1)
reg_set_seen(jit, b1); \
})
#define REG_SEEN(b1) jit->seen_reg[reg2hex[(b1)]]
/*
* EMIT macros for code generation
*/
@ -436,12 +439,12 @@ static void restore_regs(struct bpf_jit *jit, u32 rs, u32 re, u32 stack_depth)
/*
* Return first seen register (from start)
*/
static int get_start(struct bpf_jit *jit, int start)
static int get_start(u16 seen_regs, int start)
{
int i;
for (i = start; i <= 15; i++) {
if (jit->seen_reg[i])
if (seen_regs & (1 << i))
return i;
}
return 0;
@ -450,15 +453,15 @@ static int get_start(struct bpf_jit *jit, int start)
/*
* Return last seen register (from start) (gap >= 2)
*/
static int get_end(struct bpf_jit *jit, int start)
static int get_end(u16 seen_regs, int start)
{
int i;
for (i = start; i < 15; i++) {
if (!jit->seen_reg[i] && !jit->seen_reg[i + 1])
if (!(seen_regs & (3 << i)))
return i - 1;
}
return jit->seen_reg[15] ? 15 : 14;
return (seen_regs & (1 << 15)) ? 15 : 14;
}
#define REGS_SAVE 1
@ -467,8 +470,10 @@ static int get_end(struct bpf_jit *jit, int start)
* Save and restore clobbered registers (6-15) on stack.
* We save/restore registers in chunks with gap >= 2 registers.
*/
static void save_restore_regs(struct bpf_jit *jit, int op, u32 stack_depth)
static void save_restore_regs(struct bpf_jit *jit, int op, u32 stack_depth,
u16 extra_regs)
{
u16 seen_regs = jit->seen_regs | extra_regs;
const int last = 15, save_restore_size = 6;
int re = 6, rs;
@ -482,10 +487,10 @@ static void save_restore_regs(struct bpf_jit *jit, int op, u32 stack_depth)
}
do {
rs = get_start(jit, re);
rs = get_start(seen_regs, re);
if (!rs)
break;
re = get_end(jit, rs + 1);
re = get_end(seen_regs, rs + 1);
if (op == REGS_SAVE)
save_regs(jit, rs, re);
else
@ -570,8 +575,21 @@ static void bpf_jit_prologue(struct bpf_jit *jit, struct bpf_prog *fp,
}
/* Tail calls have to skip above initialization */
jit->tail_call_start = jit->prg;
/* Save registers */
save_restore_regs(jit, REGS_SAVE, stack_depth);
if (fp->aux->exception_cb) {
/*
* Switch stack, the new address is in the 2nd parameter.
*
* Arrange the restoration of %r6-%r15 in the epilogue.
* Do not restore them now, the prog does not need them.
*/
/* lgr %r15,%r3 */
EMIT4(0xb9040000, REG_15, REG_3);
jit->seen_regs |= NVREGS;
} else {
/* Save registers */
save_restore_regs(jit, REGS_SAVE, stack_depth,
fp->aux->exception_boundary ? NVREGS : 0);
}
/* Setup literal pool */
if (is_first_pass(jit) || (jit->seen & SEEN_LITERAL)) {
if (!is_first_pass(jit) &&
@ -647,7 +665,7 @@ static void bpf_jit_epilogue(struct bpf_jit *jit, u32 stack_depth)
/* Load exit code: lgr %r2,%b0 */
EMIT4(0xb9040000, REG_2, BPF_REG_0);
/* Restore registers */
save_restore_regs(jit, REGS_RESTORE, stack_depth);
save_restore_regs(jit, REGS_RESTORE, stack_depth, 0);
if (nospec_uses_trampoline()) {
jit->r14_thunk_ip = jit->prg;
/* Generate __s390_indirect_jump_r14 thunk */
@ -667,50 +685,111 @@ static void bpf_jit_epilogue(struct bpf_jit *jit, u32 stack_depth)
jit->prg += sizeof(struct bpf_plt);
}
static int get_probe_mem_regno(const u8 *insn)
{
/*
* insn must point to llgc, llgh, llgf, lg, lgb, lgh or lgf, which have
* destination register at the same position.
*/
if (insn[0] != 0xe3) /* common prefix */
return -1;
if (insn[5] != 0x90 && /* llgc */
insn[5] != 0x91 && /* llgh */
insn[5] != 0x16 && /* llgf */
insn[5] != 0x04 && /* lg */
insn[5] != 0x77 && /* lgb */
insn[5] != 0x15 && /* lgh */
insn[5] != 0x14) /* lgf */
return -1;
return insn[1] >> 4;
}
bool ex_handler_bpf(const struct exception_table_entry *x, struct pt_regs *regs)
{
regs->psw.addr = extable_fixup(x);
regs->gprs[x->data] = 0;
if (x->data != -1)
regs->gprs[x->data] = 0;
return true;
}
static int bpf_jit_probe_mem(struct bpf_jit *jit, struct bpf_prog *fp,
int probe_prg, int nop_prg)
/*
* A single BPF probe instruction
*/
struct bpf_jit_probe {
int prg; /* JITed instruction offset */
int nop_prg; /* JITed nop offset */
int reg; /* Register to clear on exception */
int arena_reg; /* Register to use for arena addressing */
};
static void bpf_jit_probe_init(struct bpf_jit_probe *probe)
{
probe->prg = -1;
probe->nop_prg = -1;
probe->reg = -1;
probe->arena_reg = REG_0;
}
/*
* Handlers of certain exceptions leave psw.addr pointing to the instruction
* directly after the failing one. Therefore, create two exception table
* entries and also add a nop in case two probing instructions come directly
* after each other.
*/
static void bpf_jit_probe_emit_nop(struct bpf_jit *jit,
struct bpf_jit_probe *probe)
{
if (probe->prg == -1 || probe->nop_prg != -1)
/* The probe is not armed or nop is already emitted. */
return;
probe->nop_prg = jit->prg;
/* bcr 0,%0 */
_EMIT2(0x0700);
}
static void bpf_jit_probe_load_pre(struct bpf_jit *jit, struct bpf_insn *insn,
struct bpf_jit_probe *probe)
{
if (BPF_MODE(insn->code) != BPF_PROBE_MEM &&
BPF_MODE(insn->code) != BPF_PROBE_MEMSX &&
BPF_MODE(insn->code) != BPF_PROBE_MEM32)
return;
if (BPF_MODE(insn->code) == BPF_PROBE_MEM32) {
/* lgrl %r1,kern_arena */
EMIT6_PCREL_RILB(0xc4080000, REG_W1, jit->kern_arena);
probe->arena_reg = REG_W1;
}
probe->prg = jit->prg;
probe->reg = reg2hex[insn->dst_reg];
}
static void bpf_jit_probe_store_pre(struct bpf_jit *jit, struct bpf_insn *insn,
struct bpf_jit_probe *probe)
{
if (BPF_MODE(insn->code) != BPF_PROBE_MEM32)
return;
/* lgrl %r1,kern_arena */
EMIT6_PCREL_RILB(0xc4080000, REG_W1, jit->kern_arena);
probe->arena_reg = REG_W1;
probe->prg = jit->prg;
}
static void bpf_jit_probe_atomic_pre(struct bpf_jit *jit,
struct bpf_insn *insn,
struct bpf_jit_probe *probe)
{
if (BPF_MODE(insn->code) != BPF_PROBE_ATOMIC)
return;
/* lgrl %r1,kern_arena */
EMIT6_PCREL_RILB(0xc4080000, REG_W1, jit->kern_arena);
/* agr %r1,%dst */
EMIT4(0xb9080000, REG_W1, insn->dst_reg);
probe->arena_reg = REG_W1;
probe->prg = jit->prg;
}
static int bpf_jit_probe_post(struct bpf_jit *jit, struct bpf_prog *fp,
struct bpf_jit_probe *probe)
{
struct exception_table_entry *ex;
int reg, prg;
int i, prg;
s64 delta;
u8 *insn;
int i;
if (probe->prg == -1)
/* The probe is not armed. */
return 0;
bpf_jit_probe_emit_nop(jit, probe);
if (!fp->aux->extable)
/* Do nothing during early JIT passes. */
return 0;
insn = jit->prg_buf + probe_prg;
reg = get_probe_mem_regno(insn);
if (WARN_ON_ONCE(reg < 0))
/* JIT bug - unexpected probe instruction. */
return -1;
if (WARN_ON_ONCE(probe_prg + insn_length(*insn) != nop_prg))
insn = jit->prg_buf + probe->prg;
if (WARN_ON_ONCE(probe->prg + insn_length(*insn) != probe->nop_prg))
/* JIT bug - gap between probe and nop instructions. */
return -1;
for (i = 0; i < 2; i++) {
@ -719,23 +798,24 @@ static int bpf_jit_probe_mem(struct bpf_jit *jit, struct bpf_prog *fp,
return -1;
ex = &fp->aux->extable[jit->excnt];
/* Add extable entries for probe and nop instructions. */
prg = i == 0 ? probe_prg : nop_prg;
prg = i == 0 ? probe->prg : probe->nop_prg;
delta = jit->prg_buf + prg - (u8 *)&ex->insn;
if (WARN_ON_ONCE(delta < INT_MIN || delta > INT_MAX))
/* JIT bug - code and extable must be close. */
return -1;
ex->insn = delta;
/*
* Always land on the nop. Note that extable infrastructure
* ignores fixup field, it is handled by ex_handler_bpf().
* Land on the current instruction. Note that the extable
* infrastructure ignores the fixup field; it is handled by
* ex_handler_bpf().
*/
delta = jit->prg_buf + nop_prg - (u8 *)&ex->fixup;
delta = jit->prg_buf + jit->prg - (u8 *)&ex->fixup;
if (WARN_ON_ONCE(delta < INT_MIN || delta > INT_MAX))
/* JIT bug - landing pad and extable must be close. */
return -1;
ex->fixup = delta;
ex->type = EX_TYPE_BPF;
ex->data = reg;
ex->data = probe->reg;
jit->excnt++;
}
return 0;
@ -782,19 +862,15 @@ static noinline int bpf_jit_insn(struct bpf_jit *jit, struct bpf_prog *fp,
s32 branch_oc_off = insn->off;
u32 dst_reg = insn->dst_reg;
u32 src_reg = insn->src_reg;
struct bpf_jit_probe probe;
int last, insn_count = 1;
u32 *addrs = jit->addrs;
s32 imm = insn->imm;
s16 off = insn->off;
int probe_prg = -1;
unsigned int mask;
int nop_prg;
int err;
if (BPF_CLASS(insn->code) == BPF_LDX &&
(BPF_MODE(insn->code) == BPF_PROBE_MEM ||
BPF_MODE(insn->code) == BPF_PROBE_MEMSX))
probe_prg = jit->prg;
bpf_jit_probe_init(&probe);
switch (insn->code) {
/*
@ -823,6 +899,22 @@ static noinline int bpf_jit_insn(struct bpf_jit *jit, struct bpf_prog *fp,
}
break;
case BPF_ALU64 | BPF_MOV | BPF_X:
if (insn_is_cast_user(insn)) {
int patch_brc;
/* ltgr %dst,%src */
EMIT4(0xb9020000, dst_reg, src_reg);
/* brc 8,0f */
patch_brc = jit->prg;
EMIT4_PCREL_RIC(0xa7040000, 8, 0);
/* iihf %dst,user_arena>>32 */
EMIT6_IMM(0xc0080000, dst_reg, jit->user_arena >> 32);
/* 0: */
if (jit->prg_buf)
*(u16 *)(jit->prg_buf + patch_brc + 2) =
(jit->prg - patch_brc) >> 1;
break;
}
switch (insn->off) {
case 0: /* DST = SRC */
/* lgr %dst,%src */
@ -1366,51 +1458,99 @@ static noinline int bpf_jit_insn(struct bpf_jit *jit, struct bpf_prog *fp,
* BPF_ST(X)
*/
case BPF_STX | BPF_MEM | BPF_B: /* *(u8 *)(dst + off) = src_reg */
/* stcy %src,off(%dst) */
EMIT6_DISP_LH(0xe3000000, 0x0072, src_reg, dst_reg, REG_0, off);
case BPF_STX | BPF_PROBE_MEM32 | BPF_B:
bpf_jit_probe_store_pre(jit, insn, &probe);
/* stcy %src,off(%dst,%arena) */
EMIT6_DISP_LH(0xe3000000, 0x0072, src_reg, dst_reg,
probe.arena_reg, off);
err = bpf_jit_probe_post(jit, fp, &probe);
if (err < 0)
return err;
jit->seen |= SEEN_MEM;
break;
case BPF_STX | BPF_MEM | BPF_H: /* (u16 *)(dst + off) = src */
/* sthy %src,off(%dst) */
EMIT6_DISP_LH(0xe3000000, 0x0070, src_reg, dst_reg, REG_0, off);
case BPF_STX | BPF_PROBE_MEM32 | BPF_H:
bpf_jit_probe_store_pre(jit, insn, &probe);
/* sthy %src,off(%dst,%arena) */
EMIT6_DISP_LH(0xe3000000, 0x0070, src_reg, dst_reg,
probe.arena_reg, off);
err = bpf_jit_probe_post(jit, fp, &probe);
if (err < 0)
return err;
jit->seen |= SEEN_MEM;
break;
case BPF_STX | BPF_MEM | BPF_W: /* *(u32 *)(dst + off) = src */
/* sty %src,off(%dst) */
EMIT6_DISP_LH(0xe3000000, 0x0050, src_reg, dst_reg, REG_0, off);
case BPF_STX | BPF_PROBE_MEM32 | BPF_W:
bpf_jit_probe_store_pre(jit, insn, &probe);
/* sty %src,off(%dst,%arena) */
EMIT6_DISP_LH(0xe3000000, 0x0050, src_reg, dst_reg,
probe.arena_reg, off);
err = bpf_jit_probe_post(jit, fp, &probe);
if (err < 0)
return err;
jit->seen |= SEEN_MEM;
break;
case BPF_STX | BPF_MEM | BPF_DW: /* (u64 *)(dst + off) = src */
/* stg %src,off(%dst) */
EMIT6_DISP_LH(0xe3000000, 0x0024, src_reg, dst_reg, REG_0, off);
case BPF_STX | BPF_PROBE_MEM32 | BPF_DW:
bpf_jit_probe_store_pre(jit, insn, &probe);
/* stg %src,off(%dst,%arena) */
EMIT6_DISP_LH(0xe3000000, 0x0024, src_reg, dst_reg,
probe.arena_reg, off);
err = bpf_jit_probe_post(jit, fp, &probe);
if (err < 0)
return err;
jit->seen |= SEEN_MEM;
break;
case BPF_ST | BPF_MEM | BPF_B: /* *(u8 *)(dst + off) = imm */
case BPF_ST | BPF_PROBE_MEM32 | BPF_B:
/* lhi %w0,imm */
EMIT4_IMM(0xa7080000, REG_W0, (u8) imm);
/* stcy %w0,off(dst) */
EMIT6_DISP_LH(0xe3000000, 0x0072, REG_W0, dst_reg, REG_0, off);
bpf_jit_probe_store_pre(jit, insn, &probe);
/* stcy %w0,off(%dst,%arena) */
EMIT6_DISP_LH(0xe3000000, 0x0072, REG_W0, dst_reg,
probe.arena_reg, off);
err = bpf_jit_probe_post(jit, fp, &probe);
if (err < 0)
return err;
jit->seen |= SEEN_MEM;
break;
case BPF_ST | BPF_MEM | BPF_H: /* (u16 *)(dst + off) = imm */
case BPF_ST | BPF_PROBE_MEM32 | BPF_H:
/* lhi %w0,imm */
EMIT4_IMM(0xa7080000, REG_W0, (u16) imm);
/* sthy %w0,off(dst) */
EMIT6_DISP_LH(0xe3000000, 0x0070, REG_W0, dst_reg, REG_0, off);
bpf_jit_probe_store_pre(jit, insn, &probe);
/* sthy %w0,off(%dst,%arena) */
EMIT6_DISP_LH(0xe3000000, 0x0070, REG_W0, dst_reg,
probe.arena_reg, off);
err = bpf_jit_probe_post(jit, fp, &probe);
if (err < 0)
return err;
jit->seen |= SEEN_MEM;
break;
case BPF_ST | BPF_MEM | BPF_W: /* *(u32 *)(dst + off) = imm */
case BPF_ST | BPF_PROBE_MEM32 | BPF_W:
/* llilf %w0,imm */
EMIT6_IMM(0xc00f0000, REG_W0, (u32) imm);
/* sty %w0,off(%dst) */
EMIT6_DISP_LH(0xe3000000, 0x0050, REG_W0, dst_reg, REG_0, off);
bpf_jit_probe_store_pre(jit, insn, &probe);
/* sty %w0,off(%dst,%arena) */
EMIT6_DISP_LH(0xe3000000, 0x0050, REG_W0, dst_reg,
probe.arena_reg, off);
err = bpf_jit_probe_post(jit, fp, &probe);
if (err < 0)
return err;
jit->seen |= SEEN_MEM;
break;
case BPF_ST | BPF_MEM | BPF_DW: /* *(u64 *)(dst + off) = imm */
case BPF_ST | BPF_PROBE_MEM32 | BPF_DW:
/* lgfi %w0,imm */
EMIT6_IMM(0xc0010000, REG_W0, imm);
/* stg %w0,off(%dst) */
EMIT6_DISP_LH(0xe3000000, 0x0024, REG_W0, dst_reg, REG_0, off);
bpf_jit_probe_store_pre(jit, insn, &probe);
/* stg %w0,off(%dst,%arena) */
EMIT6_DISP_LH(0xe3000000, 0x0024, REG_W0, dst_reg,
probe.arena_reg, off);
err = bpf_jit_probe_post(jit, fp, &probe);
if (err < 0)
return err;
jit->seen |= SEEN_MEM;
break;
/*
@ -1418,15 +1558,30 @@ static noinline int bpf_jit_insn(struct bpf_jit *jit, struct bpf_prog *fp,
*/
case BPF_STX | BPF_ATOMIC | BPF_DW:
case BPF_STX | BPF_ATOMIC | BPF_W:
case BPF_STX | BPF_PROBE_ATOMIC | BPF_DW:
case BPF_STX | BPF_PROBE_ATOMIC | BPF_W:
{
bool is32 = BPF_SIZE(insn->code) == BPF_W;
/*
* Unlike loads and stores, atomics have only a base register,
* but no index register. For the non-arena case, simply use
* %dst as a base. For the arena case, use the work register
* %r1: first, load the arena base into it, and then add %dst
* to it.
*/
probe.arena_reg = dst_reg;
switch (insn->imm) {
/* {op32|op64} {%w0|%src},%src,off(%dst) */
#define EMIT_ATOMIC(op32, op64) do { \
bpf_jit_probe_atomic_pre(jit, insn, &probe); \
/* {op32|op64} {%w0|%src},%src,off(%arena) */ \
EMIT6_DISP_LH(0xeb000000, is32 ? (op32) : (op64), \
(insn->imm & BPF_FETCH) ? src_reg : REG_W0, \
src_reg, dst_reg, off); \
src_reg, probe.arena_reg, off); \
err = bpf_jit_probe_post(jit, fp, &probe); \
if (err < 0) \
return err; \
if (insn->imm & BPF_FETCH) { \
/* bcr 14,0 - see atomic_fetch_{add,and,or,xor}() */ \
_EMIT2(0x07e0); \
@ -1455,25 +1610,50 @@ static noinline int bpf_jit_insn(struct bpf_jit *jit, struct bpf_prog *fp,
EMIT_ATOMIC(0x00f7, 0x00e7);
break;
#undef EMIT_ATOMIC
case BPF_XCHG:
/* {ly|lg} %w0,off(%dst) */
case BPF_XCHG: {
struct bpf_jit_probe load_probe = probe;
int loop_start;
bpf_jit_probe_atomic_pre(jit, insn, &load_probe);
/* {ly|lg} %w0,off(%arena) */
EMIT6_DISP_LH(0xe3000000,
is32 ? 0x0058 : 0x0004, REG_W0, REG_0,
dst_reg, off);
/* 0: {csy|csg} %w0,%src,off(%dst) */
load_probe.arena_reg, off);
bpf_jit_probe_emit_nop(jit, &load_probe);
/* Reuse {ly|lg}'s arena_reg for {csy|csg}. */
if (load_probe.prg != -1) {
probe.prg = jit->prg;
probe.arena_reg = load_probe.arena_reg;
}
loop_start = jit->prg;
/* 0: {csy|csg} %w0,%src,off(%arena) */
EMIT6_DISP_LH(0xeb000000, is32 ? 0x0014 : 0x0030,
REG_W0, src_reg, dst_reg, off);
REG_W0, src_reg, probe.arena_reg, off);
bpf_jit_probe_emit_nop(jit, &probe);
/* brc 4,0b */
EMIT4_PCREL_RIC(0xa7040000, 4, jit->prg - 6);
EMIT4_PCREL_RIC(0xa7040000, 4, loop_start);
/* {llgfr|lgr} %src,%w0 */
EMIT4(is32 ? 0xb9160000 : 0xb9040000, src_reg, REG_W0);
/* Both probes should land here on exception. */
err = bpf_jit_probe_post(jit, fp, &load_probe);
if (err < 0)
return err;
err = bpf_jit_probe_post(jit, fp, &probe);
if (err < 0)
return err;
if (is32 && insn_is_zext(&insn[1]))
insn_count = 2;
break;
}
case BPF_CMPXCHG:
/* 0: {csy|csg} %b0,%src,off(%dst) */
bpf_jit_probe_atomic_pre(jit, insn, &probe);
/* 0: {csy|csg} %b0,%src,off(%arena) */
EMIT6_DISP_LH(0xeb000000, is32 ? 0x0014 : 0x0030,
BPF_REG_0, src_reg, dst_reg, off);
BPF_REG_0, src_reg,
probe.arena_reg, off);
err = bpf_jit_probe_post(jit, fp, &probe);
if (err < 0)
return err;
break;
default:
pr_err("Unknown atomic operation %02x\n", insn->imm);
@ -1488,51 +1668,87 @@ static noinline int bpf_jit_insn(struct bpf_jit *jit, struct bpf_prog *fp,
*/
case BPF_LDX | BPF_MEM | BPF_B: /* dst = *(u8 *)(ul) (src + off) */
case BPF_LDX | BPF_PROBE_MEM | BPF_B:
/* llgc %dst,0(off,%src) */
EMIT6_DISP_LH(0xe3000000, 0x0090, dst_reg, src_reg, REG_0, off);
case BPF_LDX | BPF_PROBE_MEM32 | BPF_B:
bpf_jit_probe_load_pre(jit, insn, &probe);
/* llgc %dst,off(%src,%arena) */
EMIT6_DISP_LH(0xe3000000, 0x0090, dst_reg, src_reg,
probe.arena_reg, off);
err = bpf_jit_probe_post(jit, fp, &probe);
if (err < 0)
return err;
jit->seen |= SEEN_MEM;
if (insn_is_zext(&insn[1]))
insn_count = 2;
break;
case BPF_LDX | BPF_MEMSX | BPF_B: /* dst = *(s8 *)(ul) (src + off) */
case BPF_LDX | BPF_PROBE_MEMSX | BPF_B:
/* lgb %dst,0(off,%src) */
bpf_jit_probe_load_pre(jit, insn, &probe);
/* lgb %dst,off(%src) */
EMIT6_DISP_LH(0xe3000000, 0x0077, dst_reg, src_reg, REG_0, off);
err = bpf_jit_probe_post(jit, fp, &probe);
if (err < 0)
return err;
jit->seen |= SEEN_MEM;
break;
case BPF_LDX | BPF_MEM | BPF_H: /* dst = *(u16 *)(ul) (src + off) */
case BPF_LDX | BPF_PROBE_MEM | BPF_H:
/* llgh %dst,0(off,%src) */
EMIT6_DISP_LH(0xe3000000, 0x0091, dst_reg, src_reg, REG_0, off);
case BPF_LDX | BPF_PROBE_MEM32 | BPF_H:
bpf_jit_probe_load_pre(jit, insn, &probe);
/* llgh %dst,off(%src,%arena) */
EMIT6_DISP_LH(0xe3000000, 0x0091, dst_reg, src_reg,
probe.arena_reg, off);
err = bpf_jit_probe_post(jit, fp, &probe);
if (err < 0)
return err;
jit->seen |= SEEN_MEM;
if (insn_is_zext(&insn[1]))
insn_count = 2;
break;
case BPF_LDX | BPF_MEMSX | BPF_H: /* dst = *(s16 *)(ul) (src + off) */
case BPF_LDX | BPF_PROBE_MEMSX | BPF_H:
/* lgh %dst,0(off,%src) */
bpf_jit_probe_load_pre(jit, insn, &probe);
/* lgh %dst,off(%src) */
EMIT6_DISP_LH(0xe3000000, 0x0015, dst_reg, src_reg, REG_0, off);
err = bpf_jit_probe_post(jit, fp, &probe);
if (err < 0)
return err;
jit->seen |= SEEN_MEM;
break;
case BPF_LDX | BPF_MEM | BPF_W: /* dst = *(u32 *)(ul) (src + off) */
case BPF_LDX | BPF_PROBE_MEM | BPF_W:
case BPF_LDX | BPF_PROBE_MEM32 | BPF_W:
bpf_jit_probe_load_pre(jit, insn, &probe);
/* llgf %dst,off(%src) */
jit->seen |= SEEN_MEM;
EMIT6_DISP_LH(0xe3000000, 0x0016, dst_reg, src_reg, REG_0, off);
EMIT6_DISP_LH(0xe3000000, 0x0016, dst_reg, src_reg,
probe.arena_reg, off);
err = bpf_jit_probe_post(jit, fp, &probe);
if (err < 0)
return err;
if (insn_is_zext(&insn[1]))
insn_count = 2;
break;
case BPF_LDX | BPF_MEMSX | BPF_W: /* dst = *(s32 *)(ul) (src + off) */
case BPF_LDX | BPF_PROBE_MEMSX | BPF_W:
bpf_jit_probe_load_pre(jit, insn, &probe);
/* lgf %dst,off(%src) */
jit->seen |= SEEN_MEM;
EMIT6_DISP_LH(0xe3000000, 0x0014, dst_reg, src_reg, REG_0, off);
err = bpf_jit_probe_post(jit, fp, &probe);
if (err < 0)
return err;
break;
case BPF_LDX | BPF_MEM | BPF_DW: /* dst = *(u64 *)(ul) (src + off) */
case BPF_LDX | BPF_PROBE_MEM | BPF_DW:
/* lg %dst,0(off,%src) */
case BPF_LDX | BPF_PROBE_MEM32 | BPF_DW:
bpf_jit_probe_load_pre(jit, insn, &probe);
/* lg %dst,off(%src,%arena) */
jit->seen |= SEEN_MEM;
EMIT6_DISP_LH(0xe3000000, 0x0004, dst_reg, src_reg, REG_0, off);
EMIT6_DISP_LH(0xe3000000, 0x0004, dst_reg, src_reg,
probe.arena_reg, off);
err = bpf_jit_probe_post(jit, fp, &probe);
if (err < 0)
return err;
break;
/*
* BPF_JMP / CALL
@ -1647,7 +1863,7 @@ static noinline int bpf_jit_insn(struct bpf_jit *jit, struct bpf_prog *fp,
/*
* Restore registers before calling function
*/
save_restore_regs(jit, REGS_RESTORE, stack_depth);
save_restore_regs(jit, REGS_RESTORE, stack_depth, 0);
/*
* goto *(prog->bpf_func + tail_call_start);
@ -1897,22 +2113,6 @@ branch_oc:
return -1;
}
if (probe_prg != -1) {
/*
* Handlers of certain exceptions leave psw.addr pointing to
* the instruction directly after the failing one. Therefore,
* create two exception table entries and also add a nop in
* case two probing instructions come directly after each
* other.
*/
nop_prg = jit->prg;
/* bcr 0,%0 */
_EMIT2(0x0700);
err = bpf_jit_probe_mem(jit, fp, probe_prg, nop_prg);
if (err < 0)
return err;
}
return insn_count;
}
@ -1958,12 +2158,18 @@ static int bpf_jit_prog(struct bpf_jit *jit, struct bpf_prog *fp,
bool extra_pass, u32 stack_depth)
{
int i, insn_count, lit32_size, lit64_size;
u64 kern_arena;
jit->lit32 = jit->lit32_start;
jit->lit64 = jit->lit64_start;
jit->prg = 0;
jit->excnt = 0;
kern_arena = bpf_arena_get_kern_vm_start(fp->aux->arena);
if (kern_arena)
jit->kern_arena = _EMIT_CONST_U64(kern_arena);
jit->user_arena = bpf_arena_get_user_vm_start(fp->aux->arena);
bpf_jit_prologue(jit, fp, stack_depth);
if (bpf_set_addr(jit, 0) < 0)
return -1;
@ -2011,9 +2217,25 @@ static struct bpf_binary_header *bpf_jit_alloc(struct bpf_jit *jit,
struct bpf_prog *fp)
{
struct bpf_binary_header *header;
struct bpf_insn *insn;
u32 extable_size;
u32 code_size;
int i;
for (i = 0; i < fp->len; i++) {
insn = &fp->insnsi[i];
if (BPF_CLASS(insn->code) == BPF_STX &&
BPF_MODE(insn->code) == BPF_PROBE_ATOMIC &&
(BPF_SIZE(insn->code) == BPF_DW ||
BPF_SIZE(insn->code) == BPF_W) &&
insn->imm == BPF_XCHG)
/*
* bpf_jit_insn() emits a load and a compare-and-swap,
* both of which need to be probed.
*/
fp->aux->num_exentries += 1;
}
/* We need two entries per insn. */
fp->aux->num_exentries *= 2;
@ -2689,3 +2911,52 @@ bool bpf_jit_supports_subprog_tailcalls(void)
{
return true;
}
bool bpf_jit_supports_arena(void)
{
return true;
}
bool bpf_jit_supports_insn(struct bpf_insn *insn, bool in_arena)
{
/*
* Currently the verifier uses this function only to check which
* atomic stores to arena are supported, and they all are.
*/
return true;
}
bool bpf_jit_supports_exceptions(void)
{
/*
* Exceptions require unwinding support, which is always available,
* because the kernel is always built with backchain.
*/
return true;
}
void arch_bpf_stack_walk(bool (*consume_fn)(void *, u64, u64, u64),
void *cookie)
{
unsigned long addr, prev_addr = 0;
struct unwind_state state;
unwind_for_each_frame(&state, NULL, NULL, 0) {
addr = unwind_get_return_address(&state);
if (!addr)
break;
/*
* addr is a return address and state.sp is the value of %r15
* at this address. exception_cb needs %r15 at entry to the
* function containing addr, so take the next state.sp.
*
* There is no bp, and the exception_cb prog does not need one
* to perform a quasi-longjmp. The common code requires a
* non-zero bp, so pass sp there as well.
*/
if (prev_addr && !consume_fn(cookie, prev_addr, state.sp,
state.sp))
break;
prev_addr = addr;
}
}

15
arch/x86/net/bpf_jit_comp.c
View File

@ -1234,13 +1234,11 @@ bool ex_handler_bpf(const struct exception_table_entry *x, struct pt_regs *regs)
}
static void detect_reg_usage(struct bpf_insn *insn, int insn_cnt,
bool *regs_used, bool *tail_call_seen)
bool *regs_used)
{
int i;
for (i = 1; i <= insn_cnt; i++, insn++) {
if (insn->code == (BPF_JMP | BPF_TAIL_CALL))
*tail_call_seen = true;
if (insn->dst_reg == BPF_REG_6 || insn->src_reg == BPF_REG_6)
regs_used[0] = true;
if (insn->dst_reg == BPF_REG_7 || insn->src_reg == BPF_REG_7)
@ -1324,7 +1322,6 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image
struct bpf_insn *insn = bpf_prog->insnsi;
bool callee_regs_used[4] = {};
int insn_cnt = bpf_prog->len;
bool tail_call_seen = false;
bool seen_exit = false;
u8 temp[BPF_MAX_INSN_SIZE + BPF_INSN_SAFETY];
u64 arena_vm_start, user_vm_start;
@ -1336,11 +1333,7 @@ static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image
arena_vm_start = bpf_arena_get_kern_vm_start(bpf_prog->aux->arena);
user_vm_start = bpf_arena_get_user_vm_start(bpf_prog->aux->arena);
detect_reg_usage(insn, insn_cnt, callee_regs_used,
&tail_call_seen);
/* tail call's presence in current prog implies it is reachable */
tail_call_reachable |= tail_call_seen;
detect_reg_usage(insn, insn_cnt, callee_regs_used);
emit_prologue(&prog, bpf_prog->aux->stack_depth,
bpf_prog_was_classic(bpf_prog), tail_call_reachable,
@ -3363,7 +3356,7 @@ out_image:
*
* Both cases are serious bugs and justify WARN_ON.
*/
if (WARN_ON(bpf_jit_binary_pack_finalize(prog, header, rw_header))) {
if (WARN_ON(bpf_jit_binary_pack_finalize(header, rw_header))) {
/* header has been freed */
header = NULL;
goto out_image;
@ -3442,7 +3435,7 @@ void bpf_jit_free(struct bpf_prog *prog)
* before freeing it.
*/
if (jit_data) {
bpf_jit_binary_pack_finalize(prog, jit_data->header,
bpf_jit_binary_pack_finalize(jit_data->header,
jit_data->rw_header);
kvfree(jit_data->addrs);
kfree(jit_data);

5
fs/verity/measure.c
View File

@ -111,14 +111,15 @@ __bpf_kfunc_start_defs();
/**
* bpf_get_fsverity_digest: read fsverity digest of file
* @file: file to get digest from
* @digest_ptr: (out) dynptr for struct fsverity_digest
* @digest_p: (out) dynptr for struct fsverity_digest
*
* Read fsverity_digest of *file* into *digest_ptr*.
*
* Return: 0 on success, a negative value on error.
*/
__bpf_kfunc int bpf_get_fsverity_digest(struct file *file, struct bpf_dynptr_kern *digest_ptr)
__bpf_kfunc int bpf_get_fsverity_digest(struct file *file, struct bpf_dynptr *digest_p)
{
struct bpf_dynptr_kern *digest_ptr = (struct bpf_dynptr_kern *)digest_p;
const struct inode *inode = file_inode(file);
u32 dynptr_sz = __bpf_dynptr_size(digest_ptr);
struct fsverity_digest *arg;

11
include/linux/bpf.h
View File

@ -2931,8 +2931,7 @@ bpf_probe_read_kernel_common(void *dst, u32 size, const void *unsafe_ptr)
return ret;
}
void __bpf_free_used_btfs(struct bpf_prog_aux *aux,
struct btf_mod_pair *used_btfs, u32 len);
void __bpf_free_used_btfs(struct btf_mod_pair *used_btfs, u32 len);
static inline struct bpf_prog *bpf_prog_get_type(u32 ufd,
enum bpf_prog_type type)
@ -3263,8 +3262,8 @@ u32 bpf_sock_convert_ctx_access(enum bpf_access_type type,
struct bpf_insn *insn_buf,
struct bpf_prog *prog,
u32 *target_size);
int bpf_dynptr_from_skb_rdonly(struct sk_buff *skb, u64 flags,
struct bpf_dynptr_kern *ptr);
int bpf_dynptr_from_skb_rdonly(struct __sk_buff *skb, u64 flags,
struct bpf_dynptr *ptr);
#else
static inline bool bpf_sock_common_is_valid_access(int off, int size,
enum bpf_access_type type,
@ -3286,8 +3285,8 @@ static inline u32 bpf_sock_convert_ctx_access(enum bpf_access_type type,
{
return 0;
}
static inline int bpf_dynptr_from_skb_rdonly(struct sk_buff *skb, u64 flags,
struct bpf_dynptr_kern *ptr)
static inline int bpf_dynptr_from_skb_rdonly(struct __sk_buff *skb, u64 flags,
struct bpf_dynptr *ptr)
{
return -EOPNOTSUPP;
}

12
include/linux/bpf_verifier.h
View File

@ -73,7 +73,10 @@ enum bpf_iter_state {
struct bpf_reg_state {
/* Ordering of fields matters. See states_equal() */
enum bpf_reg_type type;
/* Fixed part of pointer offset, pointer types only */
/*
* Fixed part of pointer offset, pointer types only.
* Or constant delta between "linked" scalars with the same ID.
*/
s32 off;
union {
/* valid when type == PTR_TO_PACKET */
@ -167,6 +170,13 @@ struct bpf_reg_state {
* Similarly to dynptrs, we use ID to track "belonging" of a reference
* to a specific instance of bpf_iter.
*/
/*
* Upper bit of ID is used to remember relationship between "linked"
* registers. Example:
* r1 = r2; both will have r1->id == r2->id == N
* r1 += 10; r1->id == N | BPF_ADD_CONST and r1->off == 10
*/
#define BPF_ADD_CONST (1U << 31)
u32 id;
/* PTR_TO_SOCKET and PTR_TO_TCP_SOCK could be a ptr returned
* from a pointer-cast helper, bpf_sk_fullsock() and

65
include/linux/btf.h
View File

@ -140,6 +140,7 @@ extern const struct file_operations btf_fops;
const char *btf_get_name(const struct btf *btf);
void btf_get(struct btf *btf);
void btf_put(struct btf *btf);
const struct btf_header *btf_header(const struct btf *btf);
int btf_new_fd(const union bpf_attr *attr, bpfptr_t uattr, u32 uattr_sz);
struct btf *btf_get_by_fd(int fd);
int btf_get_info_by_fd(const struct btf *btf,
@ -212,8 +213,10 @@ int btf_get_fd_by_id(u32 id);
u32 btf_obj_id(const struct btf *btf);
bool btf_is_kernel(const struct btf *btf);
bool btf_is_module(const struct btf *btf);
bool btf_is_vmlinux(const struct btf *btf);
struct module *btf_try_get_module(const struct btf *btf);
u32 btf_nr_types(const struct btf *btf);
struct btf *btf_base_btf(const struct btf *btf);
bool btf_member_is_reg_int(const struct btf *btf, const struct btf_type *s,
const struct btf_member *m,
u32 expected_offset, u32 expected_size);
@ -339,6 +342,11 @@ static inline u8 btf_int_offset(const struct btf_type *t)
return BTF_INT_OFFSET(*(u32 *)(t + 1));
}
static inline __u8 btf_int_bits(const struct btf_type *t)
{
return BTF_INT_BITS(*(__u32 *)(t + 1));
}
static inline bool btf_type_is_scalar(const struct btf_type *t)
{
return btf_type_is_int(t) || btf_type_is_enum(t);
@ -478,6 +486,11 @@ static inline struct btf_param *btf_params(const struct btf_type *t)
return (struct btf_param *)(t + 1);
}
static inline struct btf_decl_tag *btf_decl_tag(const struct btf_type *t)
{
return (struct btf_decl_tag *)(t + 1);
}
static inline int btf_id_cmp_func(const void *a, const void *b)
{
const int *pa = a, *pb = b;
@ -515,9 +528,38 @@ static inline const struct bpf_struct_ops_desc *bpf_struct_ops_find(struct btf *
}
#endif
enum btf_field_iter_kind {
BTF_FIELD_ITER_IDS,
BTF_FIELD_ITER_STRS,
};
struct btf_field_desc {
/* once-per-type offsets */
int t_off_cnt, t_offs[2];
/* member struct size, or zero, if no members */
int m_sz;
/* repeated per-member offsets */
int m_off_cnt, m_offs[1];
};
struct btf_field_iter {
struct btf_field_desc desc;
void *p;
int m_idx;
int off_idx;
int vlen;
};
#ifdef CONFIG_BPF_SYSCALL
const struct btf_type *btf_type_by_id(const struct btf *btf, u32 type_id);
void btf_set_base_btf(struct btf *btf, const struct btf *base_btf);
int btf_relocate(struct btf *btf, const struct btf *base_btf, __u32 **map_ids);
int btf_field_iter_init(struct btf_field_iter *it, struct btf_type *t,
enum btf_field_iter_kind iter_kind);
__u32 *btf_field_iter_next(struct btf_field_iter *it);
const char *btf_name_by_offset(const struct btf *btf, u32 offset);
const char *btf_str_by_offset(const struct btf *btf, u32 offset);
struct btf *btf_parse_vmlinux(void);
struct btf *bpf_prog_get_target_btf(const struct bpf_prog *prog);
u32 *btf_kfunc_id_set_contains(const struct btf *btf, u32 kfunc_btf_id,
@ -531,6 +573,7 @@ s32 btf_find_dtor_kfunc(struct btf *btf, u32 btf_id);
int register_btf_id_dtor_kfuncs(const struct btf_id_dtor_kfunc *dtors, u32 add_cnt,
struct module *owner);
struct btf_struct_meta *btf_find_struct_meta(const struct btf *btf, u32 btf_id);
bool btf_is_projection_of(const char *pname, const char *tname);
bool btf_is_prog_ctx_type(struct bpf_verifier_log *log, const struct btf *btf,
const struct btf_type *t, enum bpf_prog_type prog_type,
int arg);
@ -543,6 +586,28 @@ static inline const struct btf_type *btf_type_by_id(const struct btf *btf,
{
return NULL;
}
static inline void btf_set_base_btf(struct btf *btf, const struct btf *base_btf)
{
}
static inline int btf_relocate(void *log, struct btf *btf, const struct btf *base_btf,
__u32 **map_ids)
{
return -EOPNOTSUPP;
}
static inline int btf_field_iter_init(struct btf_field_iter *it, struct btf_type *t,
enum btf_field_iter_kind iter_kind)
{
return -EOPNOTSUPP;
}
static inline __u32 *btf_field_iter_next(struct btf_field_iter *it)
{
return NULL;
}
static inline const char *btf_name_by_offset(const struct btf *btf,
u32 offset)
{

3
include/linux/filter.h
View File

@ -1234,8 +1234,7 @@ bpf_jit_binary_pack_alloc(unsigned int proglen, u8 **ro_image,
struct bpf_binary_header **rw_hdr,
u8 **rw_image,
bpf_jit_fill_hole_t bpf_fill_ill_insns);
int bpf_jit_binary_pack_finalize(struct bpf_prog *prog,
struct bpf_binary_header *ro_header,
int bpf_jit_binary_pack_finalize(struct bpf_binary_header *ro_header,
struct bpf_binary_header *rw_header);
void bpf_jit_binary_pack_free(struct bpf_binary_header *ro_header,
struct bpf_binary_header *rw_header);

2
include/linux/module.h
View File

@ -509,7 +509,9 @@ struct module {
#endif
#ifdef CONFIG_DEBUG_INFO_BTF_MODULES
unsigned int btf_data_size;
unsigned int btf_base_data_size;
void *btf_data;
void *btf_base_data;
#endif
#ifdef CONFIG_JUMP_LABEL
struct jump_entry *jump_entries;

15
include/net/netfilter/nf_flow_table.h
View File

@ -305,11 +305,26 @@ struct flow_ports {
__be16 source, dest;
};
struct nf_flowtable *nf_flowtable_by_dev(const struct net_device *dev);
int nf_flow_offload_xdp_setup(struct nf_flowtable *flowtable,
struct net_device *dev,
enum flow_block_command cmd);
unsigned int nf_flow_offload_ip_hook(void *priv, struct sk_buff *skb,
const struct nf_hook_state *state);
unsigned int nf_flow_offload_ipv6_hook(void *priv, struct sk_buff *skb,
const struct nf_hook_state *state);
#if (IS_BUILTIN(CONFIG_NF_FLOW_TABLE) && IS_ENABLED(CONFIG_DEBUG_INFO_BTF)) || \
(IS_MODULE(CONFIG_NF_FLOW_TABLE) && IS_ENABLED(CONFIG_DEBUG_INFO_BTF_MODULES))
extern int nf_flow_register_bpf(void);
#else
static inline int nf_flow_register_bpf(void)
{
return 0;
}
#endif
#define MODULE_ALIAS_NF_FLOWTABLE(family) \
MODULE_ALIAS("nf-flowtable-" __stringify(family))

2
include/uapi/linux/bpf.h
View File

@ -1425,6 +1425,8 @@ enum {
#define BPF_F_TEST_RUN_ON_CPU (1U << 0)
/* If set, XDP frames will be transmitted after processing */
#define BPF_F_TEST_XDP_LIVE_FRAMES (1U << 1)
/* If set, apply CHECKSUM_COMPLETE to skb and validate the checksum */
#define BPF_F_TEST_SKB_CHECKSUM_COMPLETE (1U << 2)
/* type for BPF_ENABLE_STATS */
enum bpf_stats_type {

8
kernel/bpf/Makefile
View File

@ -50,5 +50,11 @@ endif
obj-$(CONFIG_BPF_PRELOAD) += preload/
obj-$(CONFIG_BPF_SYSCALL) += relo_core.o
$(obj)/relo_core.o: $(srctree)/tools/lib/bpf/relo_core.c FORCE
obj-$(CONFIG_BPF_SYSCALL) += btf_iter.o
obj-$(CONFIG_BPF_SYSCALL) += btf_relocate.o
# Some source files are common to libbpf.
vpath %.c $(srctree)/kernel/bpf:$(srctree)/tools/lib/bpf
$(obj)/%.o: %.c FORCE
$(call if_changed_rule,cc_o_c)

1
kernel/bpf/bpf_lsm.c
View File

@ -280,6 +280,7 @@ BTF_ID(func, bpf_lsm_cred_prepare)
BTF_ID(func, bpf_lsm_file_ioctl)
BTF_ID(func, bpf_lsm_file_lock)
BTF_ID(func, bpf_lsm_file_open)
BTF_ID(func, bpf_lsm_file_post_open)
BTF_ID(func, bpf_lsm_file_receive)
BTF_ID(func, bpf_lsm_inode_create)

2
kernel/bpf/bpf_struct_ops.c
View File

@ -573,7 +573,7 @@ int bpf_struct_ops_prepare_trampoline(struct bpf_tramp_links *tlinks,
}
size = arch_prepare_bpf_trampoline(NULL, image + image_off,
image + PAGE_SIZE,
image + image_off + size,
model, flags, tlinks, stub_func);
if (size <= 0) {
if (image != *_image)

191
kernel/bpf/btf.c
View File

@ -274,6 +274,7 @@ struct btf {
u32 start_str_off; /* first string offset (0 for base BTF) */
char name[MODULE_NAME_LEN];
bool kernel_btf;
__u32 *base_id_map; /* map from distilled base BTF -> vmlinux BTF ids */
};
enum verifier_phase {
@ -530,6 +531,11 @@ static bool btf_type_is_decl_tag_target(const struct btf_type *t)
btf_type_is_var(t) || btf_type_is_typedef(t);
}
bool btf_is_vmlinux(const struct btf *btf)
{
return btf->kernel_btf && !btf->base_btf;
}
u32 btf_nr_types(const struct btf *btf)
{
u32 total = 0;
@ -772,7 +778,7 @@ static bool __btf_name_char_ok(char c, bool first)
return true;
}
static const char *btf_str_by_offset(const struct btf *btf, u32 offset)
const char *btf_str_by_offset(const struct btf *btf, u32 offset)
{
while (offset < btf->start_str_off)
btf = btf->base_btf;
@ -1670,14 +1676,8 @@ static void btf_free_kfunc_set_tab(struct btf *btf)
if (!tab)
return;
/* For module BTF, we directly assign the sets being registered, so
* there is nothing to free except kfunc_set_tab.
*/
if (btf_is_module(btf))
goto free_tab;
for (hook = 0; hook < ARRAY_SIZE(tab->sets); hook++)
kfree(tab->sets[hook]);
free_tab:
kfree(tab);
btf->kfunc_set_tab = NULL;
}
@ -1735,7 +1735,12 @@ static void btf_free(struct btf *btf)
kvfree(btf->types);
kvfree(btf->resolved_sizes);
kvfree(btf->resolved_ids);
kvfree(btf->data);
/* vmlinux does not allocate btf->data, it simply points it at
* __start_BTF.
*/
if (!btf_is_vmlinux(btf))
kvfree(btf->data);
kvfree(btf->base_id_map);
kfree(btf);
}
@ -1764,6 +1769,23 @@ void btf_put(struct btf *btf)
}
}
struct btf *btf_base_btf(const struct btf *btf)
{
return btf->base_btf;
}
const struct btf_header *btf_header(const struct btf *btf)
{
return &btf->hdr;
}
void btf_set_base_btf(struct btf *btf, const struct btf *base_btf)
{
btf->base_btf = (struct btf *)base_btf;
btf->start_id = btf_nr_types(base_btf);
btf->start_str_off = base_btf->hdr.str_len;
}
static int env_resolve_init(struct btf_verifier_env *env)
{
struct btf *btf = env->btf;
@ -5820,6 +5842,15 @@ static int find_kern_ctx_type_id(enum bpf_prog_type prog_type)
return ctx_type->type;
}
bool btf_is_projection_of(const char *pname, const char *tname)
{
if (strcmp(pname, "__sk_buff") == 0 && strcmp(tname, "sk_buff") == 0)
return true;
if (strcmp(pname, "xdp_md") == 0 && strcmp(tname, "xdp_buff") == 0)
return true;
return false;
}
bool btf_is_prog_ctx_type(struct bpf_verifier_log *log, const struct btf *btf,
const struct btf_type *t, enum bpf_prog_type prog_type,
int arg)
@ -5882,9 +5913,7 @@ again:
* int socket_filter_bpf_prog(struct __sk_buff *skb)
* { // no fields of skb are ever used }
*/
if (strcmp(ctx_tname, "__sk_buff") == 0 && strcmp(tname, "sk_buff") == 0)
return true;
if (strcmp(ctx_tname, "xdp_md") == 0 && strcmp(tname, "xdp_buff") == 0)
if (btf_is_projection_of(ctx_tname, tname))
return true;
if (strcmp(ctx_tname, tname)) {
/* bpf_user_pt_regs_t is a typedef, so resolve it to
@ -6076,23 +6105,15 @@ int get_kern_ctx_btf_id(struct bpf_verifier_log *log, enum bpf_prog_type prog_ty
BTF_ID_LIST(bpf_ctx_convert_btf_id)
BTF_ID(struct, bpf_ctx_convert)
struct btf *btf_parse_vmlinux(void)
static struct btf *btf_parse_base(struct btf_verifier_env *env, const char *name,
void *data, unsigned int data_size)
{
struct btf_verifier_env *env = NULL;
struct bpf_verifier_log *log;
struct btf *btf = NULL;
int err;
if (!IS_ENABLED(CONFIG_DEBUG_INFO_BTF))
return ERR_PTR(-ENOENT);
env = kzalloc(sizeof(*env), GFP_KERNEL | __GFP_NOWARN);
if (!env)
return ERR_PTR(-ENOMEM);
log = &env->log;
log->level = BPF_LOG_KERNEL;
btf = kzalloc(sizeof(*btf), GFP_KERNEL | __GFP_NOWARN);
if (!btf) {
err = -ENOMEM;
@ -6100,10 +6121,10 @@ struct btf *btf_parse_vmlinux(void)
}
env->btf = btf;
btf->data = __start_BTF;
btf->data_size = __stop_BTF - __start_BTF;
btf->data = data;
btf->data_size = data_size;
btf->kernel_btf = true;
snprintf(btf->name, sizeof(btf->name), "vmlinux");
snprintf(btf->name, sizeof(btf->name), "%s", name);
err = btf_parse_hdr(env);
if (err)
@ -6123,20 +6144,11 @@ struct btf *btf_parse_vmlinux(void)
if (err)
goto errout;
/* btf_parse_vmlinux() runs under bpf_verifier_lock */
bpf_ctx_convert.t = btf_type_by_id(btf, bpf_ctx_convert_btf_id[0]);
refcount_set(&btf->refcnt, 1);
err = btf_alloc_id(btf);
if (err)
goto errout;
btf_verifier_env_free(env);
return btf;
errout:
btf_verifier_env_free(env);
if (btf) {
kvfree(btf->types);
kfree(btf);
@ -6144,19 +6156,61 @@ errout:
return ERR_PTR(err);
}
#ifdef CONFIG_DEBUG_INFO_BTF_MODULES
static struct btf *btf_parse_module(const char *module_name, const void *data, unsigned int data_size)
struct btf *btf_parse_vmlinux(void)
{
struct btf_verifier_env *env = NULL;
struct bpf_verifier_log *log;
struct btf *btf = NULL, *base_btf;
struct btf *btf;
int err;
base_btf = bpf_get_btf_vmlinux();
if (IS_ERR(base_btf))
return base_btf;
if (!base_btf)
env = kzalloc(sizeof(*env), GFP_KERNEL | __GFP_NOWARN);
if (!env)
return ERR_PTR(-ENOMEM);
log = &env->log;
log->level = BPF_LOG_KERNEL;
btf = btf_parse_base(env, "vmlinux", __start_BTF, __stop_BTF - __start_BTF);
if (IS_ERR(btf))
goto err_out;
/* btf_parse_vmlinux() runs under bpf_verifier_lock */
bpf_ctx_convert.t = btf_type_by_id(btf, bpf_ctx_convert_btf_id[0]);
err = btf_alloc_id(btf);
if (err) {
btf_free(btf);
btf = ERR_PTR(err);
}
err_out:
btf_verifier_env_free(env);
return btf;
}
/* If .BTF_ids section was created with distilled base BTF, both base and
* split BTF ids will need to be mapped to actual base/split ids for
* BTF now that it has been relocated.
*/
static __u32 btf_relocate_id(const struct btf *btf, __u32 id)
{
if (!btf->base_btf || !btf->base_id_map)
return id;
return btf->base_id_map[id];
}
#ifdef CONFIG_DEBUG_INFO_BTF_MODULES
static struct btf *btf_parse_module(const char *module_name, const void *data,
unsigned int data_size, void *base_data,
unsigned int base_data_size)
{
struct btf *btf = NULL, *vmlinux_btf, *base_btf = NULL;
struct btf_verifier_env *env = NULL;
struct bpf_verifier_log *log;
int err = 0;
vmlinux_btf = bpf_get_btf_vmlinux();
if (IS_ERR(vmlinux_btf))
return vmlinux_btf;
if (!vmlinux_btf)
return ERR_PTR(-EINVAL);
env = kzalloc(sizeof(*env), GFP_KERNEL | __GFP_NOWARN);
@ -6166,6 +6220,16 @@ static struct btf *btf_parse_module(const char *module_name, const void *data, u
log = &env->log;
log->level = BPF_LOG_KERNEL;
if (base_data) {
base_btf = btf_parse_base(env, ".BTF.base", base_data, base_data_size);
if (IS_ERR(base_btf)) {
err = PTR_ERR(base_btf);
goto errout;
}
} else {
base_btf = vmlinux_btf;
}
btf = kzalloc(sizeof(*btf), GFP_KERNEL | __GFP_NOWARN);
if (!btf) {
err = -ENOMEM;
@ -6205,12 +6269,22 @@ static struct btf *btf_parse_module(const char *module_name, const void *data, u
if (err)
goto errout;
if (base_btf != vmlinux_btf) {
err = btf_relocate(btf, vmlinux_btf, &btf->base_id_map);
if (err)
goto errout;
btf_free(base_btf);
base_btf = vmlinux_btf;
}
btf_verifier_env_free(env);
refcount_set(&btf->refcnt, 1);
return btf;
errout:
btf_verifier_env_free(env);
if (base_btf != vmlinux_btf)
btf_free(base_btf);
if (btf) {
kvfree(btf->data);
kvfree(btf->types);
@ -7763,7 +7837,8 @@ static int btf_module_notify(struct notifier_block *nb, unsigned long op,
err = -ENOMEM;
goto out;
}
btf = btf_parse_module(mod->name, mod->btf_data, mod->btf_data_size);
btf = btf_parse_module(mod->name, mod->btf_data, mod->btf_data_size,
mod->btf_base_data, mod->btf_base_data_size);
if (IS_ERR(btf)) {
kfree(btf_mod);
if (!IS_ENABLED(CONFIG_MODULE_ALLOW_BTF_MISMATCH)) {
@ -8087,7 +8162,7 @@ static int btf_populate_kfunc_set(struct btf *btf, enum btf_kfunc_hook hook,
bool add_filter = !!kset->filter;
struct btf_kfunc_set_tab *tab;
struct btf_id_set8 *set;
u32 set_cnt;
u32 set_cnt, i;
int ret;
if (hook >= BTF_KFUNC_HOOK_MAX) {
@ -8133,21 +8208,15 @@ static int btf_populate_kfunc_set(struct btf *btf, enum btf_kfunc_hook hook,
goto end;
}
/* We don't need to allocate, concatenate, and sort module sets, because
* only one is allowed per hook. Hence, we can directly assign the
* pointer and return.
*/
if (!vmlinux_set) {
tab->sets[hook] = add_set;
goto do_add_filter;
}
/* In case of vmlinux sets, there may be more than one set being
* registered per hook. To create a unified set, we allocate a new set
* and concatenate all individual sets being registered. While each set
* is individually sorted, they may become unsorted when concatenated,
* hence re-sorting the final set again is required to make binary
* searching the set using btf_id_set8_contains function work.
*
* For module sets, we need to allocate as we may need to relocate
* BTF ids.
*/
set_cnt = set ? set->cnt : 0;
@ -8177,11 +8246,14 @@ static int btf_populate_kfunc_set(struct btf *btf, enum btf_kfunc_hook hook,
/* Concatenate the two sets */
memcpy(set->pairs + set->cnt, add_set->pairs, add_set->cnt * sizeof(set->pairs[0]));
/* Now that the set is copied, update with relocated BTF ids */
for (i = set->cnt; i < set->cnt + add_set->cnt; i++)
set->pairs[i].id = btf_relocate_id(btf, set->pairs[i].id);
set->cnt += add_set->cnt;
sort(set->pairs, set->cnt, sizeof(set->pairs[0]), btf_id_cmp_func, NULL);
do_add_filter:
if (add_filter) {
hook_filter = &tab->hook_filters[hook];
hook_filter->filters[hook_filter->nr_filters++] = kset->filter;
@ -8301,7 +8373,7 @@ static int __register_btf_kfunc_id_set(enum btf_kfunc_hook hook,
return PTR_ERR(btf);
for (i = 0; i < kset->set->cnt; i++) {
ret = btf_check_kfunc_protos(btf, kset->set->pairs[i].id,
ret = btf_check_kfunc_protos(btf, btf_relocate_id(btf, kset->set->pairs[i].id),
kset->set->pairs[i].flags);
if (ret)
goto err_out;
@ -8365,7 +8437,7 @@ static int btf_check_dtor_kfuncs(struct btf *btf, const struct btf_id_dtor_kfunc
u32 nr_args, i;
for (i = 0; i < cnt; i++) {
dtor_btf_id = dtors[i].kfunc_btf_id;
dtor_btf_id = btf_relocate_id(btf, dtors[i].kfunc_btf_id);
dtor_func = btf_type_by_id(btf, dtor_btf_id);
if (!dtor_func || !btf_type_is_func(dtor_func))
@ -8400,7 +8472,7 @@ int register_btf_id_dtor_kfuncs(const struct btf_id_dtor_kfunc *dtors, u32 add_c
{
struct btf_id_dtor_kfunc_tab *tab;
struct btf *btf;
u32 tab_cnt;
u32 tab_cnt, i;
int ret;
btf = btf_get_module_btf(owner);
@ -8451,6 +8523,13 @@ int register_btf_id_dtor_kfuncs(const struct btf_id_dtor_kfunc *dtors, u32 add_c
btf->dtor_kfunc_tab = tab;
memcpy(tab->dtors + tab->cnt, dtors, add_cnt * sizeof(tab->dtors[0]));
/* remap BTF ids based on BTF relocation (if any) */
for (i = tab_cnt; i < tab_cnt + add_cnt; i++) {
tab->dtors[i].btf_id = btf_relocate_id(btf, tab->dtors[i].btf_id);
tab->dtors[i].kfunc_btf_id = btf_relocate_id(btf, tab->dtors[i].kfunc_btf_id);
}
tab->cnt += add_cnt;
sort(tab->dtors, tab->cnt, sizeof(tab->dtors[0]), btf_id_cmp_func, NULL);

8
kernel/bpf/core.c
View File

@ -1173,8 +1173,7 @@ bpf_jit_binary_pack_alloc(unsigned int proglen, u8 **image_ptr,
}
/* Copy JITed text from rw_header to its final location, the ro_header. */
int bpf_jit_binary_pack_finalize(struct bpf_prog *prog,
struct bpf_binary_header *ro_header,
int bpf_jit_binary_pack_finalize(struct bpf_binary_header *ro_header,
struct bpf_binary_header *rw_header)
{
void *ptr;
@ -2742,8 +2741,7 @@ static void bpf_free_used_maps(struct bpf_prog_aux *aux)
kfree(aux->used_maps);
}
void __bpf_free_used_btfs(struct bpf_prog_aux *aux,
struct btf_mod_pair *used_btfs, u32 len)
void __bpf_free_used_btfs(struct btf_mod_pair *used_btfs, u32 len)
{
#ifdef CONFIG_BPF_SYSCALL
struct btf_mod_pair *btf_mod;
@ -2760,7 +2758,7 @@ void __bpf_free_used_btfs(struct bpf_prog_aux *aux,
static void bpf_free_used_btfs(struct bpf_prog_aux *aux)
{
__bpf_free_used_btfs(aux, aux->used_btfs, aux->used_btf_cnt);
__bpf_free_used_btfs(aux->used_btfs, aux->used_btf_cnt);
kfree(aux->used_btfs);
}

42
kernel/bpf/crypto.c
View File

@ -275,7 +275,7 @@ static int bpf_crypto_crypt(const struct bpf_crypto_ctx *ctx,
if (__bpf_dynptr_is_rdonly(dst))
return -EINVAL;
siv_len = __bpf_dynptr_size(siv);
siv_len = siv ? __bpf_dynptr_size(siv) : 0;
src_len = __bpf_dynptr_size(src);
dst_len = __bpf_dynptr_size(dst);
if (!src_len || !dst_len)
@ -303,36 +303,44 @@ static int bpf_crypto_crypt(const struct bpf_crypto_ctx *ctx,
/**
* bpf_crypto_decrypt() - Decrypt buffer using configured context and IV provided.
* @ctx: The crypto context being used. The ctx must be a trusted pointer.
* @src: bpf_dynptr to the encrypted data. Must be a trusted pointer.
* @dst: bpf_dynptr to the buffer where to store the result. Must be a trusted pointer.
* @siv: bpf_dynptr to IV data and state data to be used by decryptor.
* @ctx: The crypto context being used. The ctx must be a trusted pointer.
* @src: bpf_dynptr to the encrypted data. Must be a trusted pointer.
* @dst: bpf_dynptr to the buffer where to store the result. Must be a trusted pointer.
* @siv__nullable: bpf_dynptr to IV data and state data to be used by decryptor. May be NULL.
*
* Decrypts provided buffer using IV data and the crypto context. Crypto context must be configured.
*/
__bpf_kfunc int bpf_crypto_decrypt(struct bpf_crypto_ctx *ctx,
const struct bpf_dynptr_kern *src,
const struct bpf_dynptr_kern *dst,
const struct bpf_dynptr_kern *siv)
const struct bpf_dynptr *src,
const struct bpf_dynptr *dst,
const struct bpf_dynptr *siv__nullable)
{
return bpf_crypto_crypt(ctx, src, dst, siv, true);
const struct bpf_dynptr_kern *src_kern = (struct bpf_dynptr_kern *)src;
const struct bpf_dynptr_kern *dst_kern = (struct bpf_dynptr_kern *)dst;
const struct bpf_dynptr_kern *siv_kern = (struct bpf_dynptr_kern *)siv__nullable;
return bpf_crypto_crypt(ctx, src_kern, dst_kern, siv_kern, true);
}
/**
* bpf_crypto_encrypt() - Encrypt buffer using configured context and IV provided.
* @ctx: The crypto context being used. The ctx must be a trusted pointer.
* @src: bpf_dynptr to the plain data. Must be a trusted pointer.
* @dst: bpf_dynptr to buffer where to store the result. Must be a trusted pointer.
* @siv: bpf_dynptr to IV data and state data to be used by decryptor.
* @ctx: The crypto context being used. The ctx must be a trusted pointer.
* @src: bpf_dynptr to the plain data. Must be a trusted pointer.
* @dst: bpf_dynptr to the buffer where to store the result. Must be a trusted pointer.
* @siv__nullable: bpf_dynptr to IV data and state data to be used by decryptor. May be NULL.
*
* Encrypts provided buffer using IV data and the crypto context. Crypto context must be configured.
*/
__bpf_kfunc int bpf_crypto_encrypt(struct bpf_crypto_ctx *ctx,
const struct bpf_dynptr_kern *src,
const struct bpf_dynptr_kern *dst,
const struct bpf_dynptr_kern *siv)
const struct bpf_dynptr *src,
const struct bpf_dynptr *dst,
const struct bpf_dynptr *siv__nullable)
{
return bpf_crypto_crypt(ctx, src, dst, siv, false);
const struct bpf_dynptr_kern *src_kern = (struct bpf_dynptr_kern *)src;
const struct bpf_dynptr_kern *dst_kern = (struct bpf_dynptr_kern *)dst;
const struct bpf_dynptr_kern *siv_kern = (struct bpf_dynptr_kern *)siv__nullable;
return bpf_crypto_crypt(ctx, src_kern, dst_kern, siv_kern, false);
}
__bpf_kfunc_end_defs();

27
kernel/bpf/devmap.c
View File

@ -106,7 +106,7 @@ static inline struct hlist_head *dev_map_index_hash(struct bpf_dtab *dtab,
return &dtab->dev_index_head[idx & (dtab->n_buckets - 1)];
}
static int dev_map_init_map(struct bpf_dtab *dtab, union bpf_attr *attr)
static int dev_map_alloc_check(union bpf_attr *attr)
{
u32 valsize = attr->value_size;
@ -120,23 +120,28 @@ static int dev_map_init_map(struct bpf_dtab *dtab, union bpf_attr *attr)
attr->map_flags & ~DEV_CREATE_FLAG_MASK)
return -EINVAL;
if (attr->map_type == BPF_MAP_TYPE_DEVMAP_HASH) {
/* Hash table size must be power of 2; roundup_pow_of_two()
* can overflow into UB on 32-bit arches
*/
if (attr->max_entries > 1UL << 31)
return -EINVAL;
}
return 0;
}
static int dev_map_init_map(struct bpf_dtab *dtab, union bpf_attr *attr)
{
/* Lookup returns a pointer straight to dev->ifindex, so make sure the
* verifier prevents writes from the BPF side
*/
attr->map_flags |= BPF_F_RDONLY_PROG;
bpf_map_init_from_attr(&dtab->map, attr);
if (attr->map_type == BPF_MAP_TYPE_DEVMAP_HASH) {
/* hash table size must be power of 2; roundup_pow_of_two() can
* overflow into UB on 32-bit arches, so check that first
*/
if (dtab->map.max_entries > 1UL << 31)
return -EINVAL;
/* Hash table size must be power of 2 */
dtab->n_buckets = roundup_pow_of_two(dtab->map.max_entries);
dtab->dev_index_head = dev_map_create_hash(dtab->n_buckets,
dtab->map.numa_node);
if (!dtab->dev_index_head)
@ -1036,6 +1041,7 @@ static u64 dev_map_mem_usage(const struct bpf_map *map)
BTF_ID_LIST_SINGLE(dev_map_btf_ids, struct, bpf_dtab)
const struct bpf_map_ops dev_map_ops = {
.map_meta_equal = bpf_map_meta_equal,
.map_alloc_check = dev_map_alloc_check,
.map_alloc = dev_map_alloc,
.map_free = dev_map_free,
.map_get_next_key = dev_map_get_next_key,
@ -1050,6 +1056,7 @@ const struct bpf_map_ops dev_map_ops = {
const struct bpf_map_ops dev_map_hash_ops = {
.map_meta_equal = bpf_map_meta_equal,
.map_alloc_check = dev_map_alloc_check,
.map_alloc = dev_map_alloc,
.map_free = dev_map_free,
.map_get_next_key = dev_map_hash_get_next_key,

45
kernel/bpf/helpers.c
View File

@ -2433,7 +2433,7 @@ __bpf_kfunc struct task_struct *bpf_task_from_pid(s32 pid)
/**
* bpf_dynptr_slice() - Obtain a read-only pointer to the dynptr data.
* @ptr: The dynptr whose data slice to retrieve
* @p: The dynptr whose data slice to retrieve
* @offset: Offset into the dynptr
* @buffer__opt: User-provided buffer to copy contents into. May be NULL
* @buffer__szk: Size (in bytes) of the buffer if present. This is the
@ -2459,9 +2459,10 @@ __bpf_kfunc struct task_struct *bpf_task_from_pid(s32 pid)
* provided buffer, with its contents containing the data, if unable to obtain
* direct pointer)
*/
__bpf_kfunc void *bpf_dynptr_slice(const struct bpf_dynptr_kern *ptr, u32 offset,
__bpf_kfunc void *bpf_dynptr_slice(const struct bpf_dynptr *p, u32 offset,
void *buffer__opt, u32 buffer__szk)
{
const struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
enum bpf_dynptr_type type;
u32 len = buffer__szk;
int err;
@ -2503,7 +2504,7 @@ __bpf_kfunc void *bpf_dynptr_slice(const struct bpf_dynptr_kern *ptr, u32 offset
/**
* bpf_dynptr_slice_rdwr() - Obtain a writable pointer to the dynptr data.
* @ptr: The dynptr whose data slice to retrieve
* @p: The dynptr whose data slice to retrieve
* @offset: Offset into the dynptr
* @buffer__opt: User-provided buffer to copy contents into. May be NULL
* @buffer__szk: Size (in bytes) of the buffer if present. This is the
@ -2543,9 +2544,11 @@ __bpf_kfunc void *bpf_dynptr_slice(const struct bpf_dynptr_kern *ptr, u32 offset
* provided buffer, with its contents containing the data, if unable to obtain
* direct pointer)
*/
__bpf_kfunc void *bpf_dynptr_slice_rdwr(const struct bpf_dynptr_kern *ptr, u32 offset,
__bpf_kfunc void *bpf_dynptr_slice_rdwr(const struct bpf_dynptr *p, u32 offset,
void *buffer__opt, u32 buffer__szk)
{
const struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
if (!ptr->data || __bpf_dynptr_is_rdonly(ptr))
return NULL;
@ -2571,11 +2574,12 @@ __bpf_kfunc void *bpf_dynptr_slice_rdwr(const struct bpf_dynptr_kern *ptr, u32 o
* will be copied out into the buffer and the user will need to call
* bpf_dynptr_write() to commit changes.
*/
return bpf_dynptr_slice(ptr, offset, buffer__opt, buffer__szk);
return bpf_dynptr_slice(p, offset, buffer__opt, buffer__szk);
}
__bpf_kfunc int bpf_dynptr_adjust(struct bpf_dynptr_kern *ptr, u32 start, u32 end)
__bpf_kfunc int bpf_dynptr_adjust(const struct bpf_dynptr *p, u32 start, u32 end)
{
struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
u32 size;
if (!ptr->data || start > end)
@ -2592,36 +2596,45 @@ __bpf_kfunc int bpf_dynptr_adjust(struct bpf_dynptr_kern *ptr, u32 start, u32 en
return 0;
}
__bpf_kfunc bool bpf_dynptr_is_null(struct bpf_dynptr_kern *ptr)
__bpf_kfunc bool bpf_dynptr_is_null(const struct bpf_dynptr *p)
{
struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
return !ptr->data;
}
__bpf_kfunc bool bpf_dynptr_is_rdonly(struct bpf_dynptr_kern *ptr)
__bpf_kfunc bool bpf_dynptr_is_rdonly(const struct bpf_dynptr *p)
{
struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
if (!ptr->data)
return false;
return __bpf_dynptr_is_rdonly(ptr);
}
__bpf_kfunc __u32 bpf_dynptr_size(const struct bpf_dynptr_kern *ptr)
__bpf_kfunc __u32 bpf_dynptr_size(const struct bpf_dynptr *p)
{
struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
if (!ptr->data)
return -EINVAL;
return __bpf_dynptr_size(ptr);
}
__bpf_kfunc int bpf_dynptr_clone(struct bpf_dynptr_kern *ptr,
struct bpf_dynptr_kern *clone__uninit)
__bpf_kfunc int bpf_dynptr_clone(const struct bpf_dynptr *p,
struct bpf_dynptr *clone__uninit)
{
struct bpf_dynptr_kern *clone = (struct bpf_dynptr_kern *)clone__uninit;
struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p;
if (!ptr->data) {
bpf_dynptr_set_null(clone__uninit);
bpf_dynptr_set_null(clone);
return -EINVAL;
}
*clone__uninit = *ptr;
*clone = *ptr;
return 0;
}
@ -2721,7 +2734,7 @@ __bpf_kfunc int bpf_wq_start(struct bpf_wq *wq, unsigned int flags)
}
__bpf_kfunc int bpf_wq_set_callback_impl(struct bpf_wq *wq,
int (callback_fn)(void *map, int *key, struct bpf_wq *wq),
int (callback_fn)(void *map, int *key, void *value),
unsigned int flags,
void *aux__ign)
{
@ -2986,7 +2999,9 @@ late_initcall(kfunc_init);
*/
const void *__bpf_dynptr_data(const struct bpf_dynptr_kern *ptr, u32 len)
{
return bpf_dynptr_slice(ptr, 0, NULL, len);
const struct bpf_dynptr *p = (struct bpf_dynptr *)ptr;
return bpf_dynptr_slice(p, 0, NULL, len);
}
/* Get a pointer to dynptr data up to len bytes for read write access. If

6
kernel/bpf/log.c
View File

@ -91,7 +91,7 @@ void bpf_verifier_vlog(struct bpf_verifier_log *log, const char *fmt,
goto fail;
} else {
u64 new_end, new_start;
u32 buf_start, buf_end, new_n;
u32 buf_start, buf_end;
new_end = log->end_pos + n;
if (new_end - log->start_pos >= log->len_total)
@ -708,7 +708,9 @@ static void print_reg_state(struct bpf_verifier_env *env,
verbose(env, "%s", btf_type_name(reg->btf, reg->btf_id));
verbose(env, "(");
if (reg->id)
verbose_a("id=%d", reg->id);
verbose_a("id=%d", reg->id & ~BPF_ADD_CONST);
if (reg->id & BPF_ADD_CONST)
verbose(env, "%+d", reg->off);
if (reg->ref_obj_id)
verbose_a("ref_obj_id=%d", reg->ref_obj_id);
if (type_is_non_owning_ref(reg->type))

9
kernel/bpf/task_iter.c
View File

@ -261,6 +261,7 @@ task_file_seq_get_next(struct bpf_iter_seq_task_file_info *info)
u32 saved_tid = info->tid;
struct task_struct *curr_task;
unsigned int curr_fd = info->fd;
struct file *f;
/* If this function returns a non-NULL file object,
* it held a reference to the task/file.
@ -286,12 +287,8 @@ again:
}
rcu_read_lock();
for (;; curr_fd++) {
struct file *f;
f = task_lookup_next_fdget_rcu(curr_task, &curr_fd);
if (!f)
break;
f = task_lookup_next_fdget_rcu(curr_task, &curr_fd);
if (f) {
/* set info->fd */
info->fd = curr_fd;
rcu_read_unlock();

140
kernel/bpf/verifier.c
View File

@ -2982,8 +2982,10 @@ static int check_subprogs(struct bpf_verifier_env *env)
if (code == (BPF_JMP | BPF_CALL) &&
insn[i].src_reg == 0 &&
insn[i].imm == BPF_FUNC_tail_call)
insn[i].imm == BPF_FUNC_tail_call) {
subprog[cur_subprog].has_tail_call = true;
subprog[cur_subprog].tail_call_reachable = true;
}
if (BPF_CLASS(code) == BPF_LD &&
(BPF_MODE(code) == BPF_ABS || BPF_MODE(code) == BPF_IND))
subprog[cur_subprog].has_ld_abs = true;
@ -3215,7 +3217,8 @@ static int insn_def_regno(const struct bpf_insn *insn)
case BPF_ST:
return -1;
case BPF_STX:
if (BPF_MODE(insn->code) == BPF_ATOMIC &&
if ((BPF_MODE(insn->code) == BPF_ATOMIC ||
BPF_MODE(insn->code) == BPF_PROBE_ATOMIC) &&
(insn->imm & BPF_FETCH)) {
if (insn->imm == BPF_CMPXCHG)
return BPF_REG_0;
@ -3991,7 +3994,7 @@ static bool idset_contains(struct bpf_idset *s, u32 id)
u32 i;
for (i = 0; i < s->count; ++i)
if (s->ids[i] == id)
if (s->ids[i] == (id & ~BPF_ADD_CONST))
return true;
return false;
@ -4001,7 +4004,7 @@ static int idset_push(struct bpf_idset *s, u32 id)
{
if (WARN_ON_ONCE(s->count >= ARRAY_SIZE(s->ids)))
return -EFAULT;
s->ids[s->count++] = id;
s->ids[s->count++] = id & ~BPF_ADD_CONST;
return 0;
}
@ -4438,8 +4441,20 @@ static bool __is_pointer_value(bool allow_ptr_leaks,
static void assign_scalar_id_before_mov(struct bpf_verifier_env *env,
struct bpf_reg_state *src_reg)
{
if (src_reg->type == SCALAR_VALUE && !src_reg->id &&
!tnum_is_const(src_reg->var_off))
if (src_reg->type != SCALAR_VALUE)
return;
if (src_reg->id & BPF_ADD_CONST) {
/*
* The verifier is processing rX = rY insn and
* rY->id has special linked register already.
* Cleared it, since multiple rX += const are not supported.
*/
src_reg->id = 0;
src_reg->off = 0;
}
if (!src_reg->id && !tnum_is_const(src_reg->var_off))
/* Ensure that src_reg has a valid ID that will be copied to
* dst_reg and then will be used by find_equal_scalars() to
* propagate min/max range.
@ -7715,6 +7730,13 @@ static int process_dynptr_func(struct bpf_verifier_env *env, int regno, int insn
struct bpf_reg_state *regs = cur_regs(env), *reg = &regs[regno];
int err;
if (reg->type != PTR_TO_STACK && reg->type != CONST_PTR_TO_DYNPTR) {
verbose(env,
"arg#%d expected pointer to stack or const struct bpf_dynptr\n",
regno);
return -EINVAL;
}
/* MEM_UNINIT and MEM_RDONLY are exclusive, when applied to an
* ARG_PTR_TO_DYNPTR (or ARG_PTR_TO_DYNPTR | DYNPTR_TYPE_*):
*/
@ -9464,6 +9486,10 @@ static int btf_check_func_arg_match(struct bpf_verifier_env *env, int subprog,
return -EINVAL;
}
} else if (arg->arg_type == (ARG_PTR_TO_DYNPTR | MEM_RDONLY)) {
ret = check_func_arg_reg_off(env, reg, regno, ARG_PTR_TO_DYNPTR);
if (ret)
return ret;
ret = process_dynptr_func(env, regno, -1, arg->arg_type, 0);
if (ret)
return ret;
@ -10917,7 +10943,7 @@ enum {
};
BTF_ID_LIST(kf_arg_btf_ids)
BTF_ID(struct, bpf_dynptr_kern)
BTF_ID(struct, bpf_dynptr)
BTF_ID(struct, bpf_list_head)
BTF_ID(struct, bpf_list_node)
BTF_ID(struct, bpf_rb_root)
@ -11190,6 +11216,9 @@ get_kfunc_ptr_arg_type(struct bpf_verifier_env *env,
if (btf_is_prog_ctx_type(&env->log, meta->btf, t, resolve_prog_type(env->prog), argno))
return KF_ARG_PTR_TO_CTX;
if (is_kfunc_arg_nullable(meta->btf, &args[argno]) && register_is_null(reg))
return KF_ARG_PTR_TO_NULL;
if (is_kfunc_arg_alloc_obj(meta->btf, &args[argno]))
return KF_ARG_PTR_TO_ALLOC_BTF_ID;
@ -11235,9 +11264,6 @@ get_kfunc_ptr_arg_type(struct bpf_verifier_env *env,
if (is_kfunc_arg_callback(env, meta->btf, &args[argno]))
return KF_ARG_PTR_TO_CALLBACK;
if (is_kfunc_arg_nullable(meta->btf, &args[argno]) && register_is_null(reg))
return KF_ARG_PTR_TO_NULL;
if (argno + 1 < nargs &&
(is_kfunc_arg_mem_size(meta->btf, &args[argno + 1], &regs[regno + 1]) ||
is_kfunc_arg_const_mem_size(meta->btf, &args[argno + 1], &regs[regno + 1])))
@ -11268,6 +11294,8 @@ static int process_kf_arg_ptr_to_btf_id(struct bpf_verifier_env *env,
bool strict_type_match = false;
const struct btf *reg_btf;
const char *reg_ref_tname;
bool taking_projection;
bool struct_same;
u32 reg_ref_id;
if (base_type(reg->type) == PTR_TO_BTF_ID) {
@ -11311,7 +11339,13 @@ static int process_kf_arg_ptr_to_btf_id(struct bpf_verifier_env *env,
reg_ref_t = btf_type_skip_modifiers(reg_btf, reg_ref_id, &reg_ref_id);
reg_ref_tname = btf_name_by_offset(reg_btf, reg_ref_t->name_off);
if (!btf_struct_ids_match(&env->log, reg_btf, reg_ref_id, reg->off, meta->btf, ref_id, strict_type_match)) {
struct_same = btf_struct_ids_match(&env->log, reg_btf, reg_ref_id, reg->off, meta->btf, ref_id, strict_type_match);
/* If kfunc is accepting a projection type (ie. __sk_buff), it cannot
* actually use it -- it must cast to the underlying type. So we allow
* caller to pass in the underlying type.
*/
taking_projection = btf_is_projection_of(ref_tname, reg_ref_tname);
if (!taking_projection && !struct_same) {
verbose(env, "kernel function %s args#%d expected pointer to %s %s but R%d has a pointer to %s %s\n",
meta->func_name, argno, btf_type_str(ref_t), ref_tname, argno + 1,
btf_type_str(reg_ref_t), reg_ref_tname);
@ -11957,12 +11991,6 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_
enum bpf_arg_type dynptr_arg_type = ARG_PTR_TO_DYNPTR;
int clone_ref_obj_id = 0;
if (reg->type != PTR_TO_STACK &&
reg->type != CONST_PTR_TO_DYNPTR) {
verbose(env, "arg#%d expected pointer to stack or dynptr_ptr\n", i);
return -EINVAL;
}
if (reg->type == CONST_PTR_TO_DYNPTR)
dynptr_arg_type |= MEM_RDONLY;
@ -14047,6 +14075,7 @@ static int adjust_reg_min_max_vals(struct bpf_verifier_env *env,
struct bpf_func_state *state = vstate->frame[vstate->curframe];
struct bpf_reg_state *regs = state->regs, *dst_reg, *src_reg;
struct bpf_reg_state *ptr_reg = NULL, off_reg = {0};
bool alu32 = (BPF_CLASS(insn->code) != BPF_ALU64);
u8 opcode = BPF_OP(insn->code);
int err;
@ -14069,11 +14098,7 @@ static int adjust_reg_min_max_vals(struct bpf_verifier_env *env,
if (dst_reg->type != SCALAR_VALUE)
ptr_reg = dst_reg;
else
/* Make sure ID is cleared otherwise dst_reg min/max could be
* incorrectly propagated into other registers by find_equal_scalars()
*/
dst_reg->id = 0;
if (BPF_SRC(insn->code) == BPF_X) {
src_reg = &regs[insn->src_reg];
if (src_reg->type != SCALAR_VALUE) {
@ -14137,7 +14162,43 @@ static int adjust_reg_min_max_vals(struct bpf_verifier_env *env,
verbose(env, "verifier internal error: no src_reg\n");
return -EINVAL;
}
return adjust_scalar_min_max_vals(env, insn, dst_reg, *src_reg);
err = adjust_scalar_min_max_vals(env, insn, dst_reg, *src_reg);
if (err)
return err;
/*
* Compilers can generate the code
* r1 = r2
* r1 += 0x1
* if r2 < 1000 goto ...
* use r1 in memory access
* So remember constant delta between r2 and r1 and update r1 after
* 'if' condition.
*/
if (env->bpf_capable && BPF_OP(insn->code) == BPF_ADD &&
dst_reg->id && is_reg_const(src_reg, alu32)) {
u64 val = reg_const_value(src_reg, alu32);
if ((dst_reg->id & BPF_ADD_CONST) ||
/* prevent overflow in find_equal_scalars() later */
val > (u32)S32_MAX) {
/*
* If the register already went through rX += val
* we cannot accumulate another val into rx->off.
*/
dst_reg->off = 0;
dst_reg->id = 0;
} else {
dst_reg->id |= BPF_ADD_CONST;
dst_reg->off = val;
}
} else {
/*
* Make sure ID is cleared otherwise dst_reg min/max could be
* incorrectly propagated into other registers by find_equal_scalars()
*/
dst_reg->id = 0;
}
return 0;
}
/* check validity of 32-bit and 64-bit arithmetic operations */
@ -15109,12 +15170,36 @@ static bool try_match_pkt_pointers(const struct bpf_insn *insn,
static void find_equal_scalars(struct bpf_verifier_state *vstate,
struct bpf_reg_state *known_reg)
{
struct bpf_reg_state fake_reg;
struct bpf_func_state *state;
struct bpf_reg_state *reg;
bpf_for_each_reg_in_vstate(vstate, state, reg, ({
if (reg->type == SCALAR_VALUE && reg->id == known_reg->id)
if (reg->type != SCALAR_VALUE || reg == known_reg)
continue;
if ((reg->id & ~BPF_ADD_CONST) != (known_reg->id & ~BPF_ADD_CONST))
continue;
if ((!(reg->id & BPF_ADD_CONST) && !(known_reg->id & BPF_ADD_CONST)) ||
reg->off == known_reg->off) {
copy_register_state(reg, known_reg);
} else {
s32 saved_off = reg->off;
fake_reg.type = SCALAR_VALUE;
__mark_reg_known(&fake_reg, (s32)reg->off - (s32)known_reg->off);
/* reg = known_reg; reg += delta */
copy_register_state(reg, known_reg);
/*
* Must preserve off, id and add_const flag,
* otherwise another find_equal_scalars() will be incorrect.
*/
reg->off = saved_off;
scalar32_min_max_add(reg, &fake_reg);
scalar_min_max_add(reg, &fake_reg);
reg->var_off = tnum_add(reg->var_off, fake_reg.var_off);
}
}));
}
@ -16749,6 +16834,10 @@ static bool regsafe(struct bpf_verifier_env *env, struct bpf_reg_state *rold,
}
if (!rold->precise && exact == NOT_EXACT)
return true;
if ((rold->id & BPF_ADD_CONST) != (rcur->id & BPF_ADD_CONST))
return false;
if ((rold->id & BPF_ADD_CONST) && (rold->off != rcur->off))
return false;
/* Why check_ids() for scalar registers?
*
* Consider the following BPF code:
@ -18630,8 +18719,7 @@ static void release_maps(struct bpf_verifier_env *env)
/* drop refcnt of maps used by the rejected program */
static void release_btfs(struct bpf_verifier_env *env)
{
__bpf_free_used_btfs(env->prog->aux, env->used_btfs,
env->used_btf_cnt);
__bpf_free_used_btfs(env->used_btfs, env->used_btf_cnt);
}
/* convert pseudo BPF_LD_IMM64 into generic BPF_LD_IMM64 */

5
kernel/module/main.c
View File

@ -2166,6 +2166,8 @@ static int find_module_sections(struct module *mod, struct load_info *info)
#endif
#ifdef CONFIG_DEBUG_INFO_BTF_MODULES
mod->btf_data = any_section_objs(info, ".BTF", 1, &mod->btf_data_size);
mod->btf_base_data = any_section_objs(info, ".BTF.base", 1,
&mod->btf_base_data_size);
#endif
#ifdef CONFIG_JUMP_LABEL
mod->jump_entries = section_objs(info, "__jump_table",
@ -2590,8 +2592,9 @@ static noinline int do_init_module(struct module *mod)
}
#ifdef CONFIG_DEBUG_INFO_BTF_MODULES
/* .BTF is not SHF_ALLOC and will get removed, so sanitize pointer */
/* .BTF is not SHF_ALLOC and will get removed, so sanitize pointers */
mod->btf_data = NULL;
mod->btf_base_data = NULL;
#endif
/*
* We want to free module_init, but be aware that kallsyms may be

15
kernel/trace/bpf_trace.c
View File

@ -1369,8 +1369,8 @@ __bpf_kfunc void bpf_key_put(struct bpf_key *bkey)
#ifdef CONFIG_SYSTEM_DATA_VERIFICATION
/**
* bpf_verify_pkcs7_signature - verify a PKCS#7 signature
* @data_ptr: data to verify
* @sig_ptr: signature of the data
* @data_p: data to verify
* @sig_p: signature of the data
* @trusted_keyring: keyring with keys trusted for signature verification
*
* Verify the PKCS#7 signature *sig_ptr* against the supplied *data_ptr*
@ -1378,10 +1378,12 @@ __bpf_kfunc void bpf_key_put(struct bpf_key *bkey)
*
* Return: 0 on success, a negative value on error.
*/
__bpf_kfunc int bpf_verify_pkcs7_signature(struct bpf_dynptr_kern *data_ptr,
struct bpf_dynptr_kern *sig_ptr,
__bpf_kfunc int bpf_verify_pkcs7_signature(struct bpf_dynptr *data_p,
struct bpf_dynptr *sig_p,
struct bpf_key *trusted_keyring)
{
struct bpf_dynptr_kern *data_ptr = (struct bpf_dynptr_kern *)data_p;
struct bpf_dynptr_kern *sig_ptr = (struct bpf_dynptr_kern *)sig_p;
const void *data, *sig;
u32 data_len, sig_len;
int ret;
@ -1444,7 +1446,7 @@ __bpf_kfunc_start_defs();
* bpf_get_file_xattr - get xattr of a file
* @file: file to get xattr from
* @name__str: name of the xattr
* @value_ptr: output buffer of the xattr value
* @value_p: output buffer of the xattr value
*
* Get xattr *name__str* of *file* and store the output in *value_ptr*.
*
@ -1453,8 +1455,9 @@ __bpf_kfunc_start_defs();
* Return: 0 on success, a negative value on error.
*/
__bpf_kfunc int bpf_get_file_xattr(struct file *file, const char *name__str,
struct bpf_dynptr_kern *value_ptr)
struct bpf_dynptr *value_p)
{
struct bpf_dynptr_kern *value_ptr = (struct bpf_dynptr_kern *)value_p;
struct dentry *dentry;
u32 value_len;
void *value;

10
lib/test_bpf.c
View File

@ -15198,6 +15198,7 @@ struct tail_call_test {
int flags;
int result;
int stack_depth;
bool has_tail_call;
};
/* Flags that can be passed to tail call test cases */
@ -15273,6 +15274,7 @@ static struct tail_call_test tail_call_tests[] = {
BPF_EXIT_INSN(),
},
.result = 3,
.has_tail_call = true,
},
{
"Tail call 3",
@ -15283,6 +15285,7 @@ static struct tail_call_test tail_call_tests[] = {
BPF_EXIT_INSN(),
},
.result = 6,
.has_tail_call = true,
},
{
"Tail call 4",
@ -15293,6 +15296,7 @@ static struct tail_call_test tail_call_tests[] = {
BPF_EXIT_INSN(),
},
.result = 10,
.has_tail_call = true,
},
{
"Tail call load/store leaf",
@ -15323,6 +15327,7 @@ static struct tail_call_test tail_call_tests[] = {
},
.result = 0,
.stack_depth = 16,
.has_tail_call = true,
},
{
"Tail call error path, max count reached",
@ -15335,6 +15340,7 @@ static struct tail_call_test tail_call_tests[] = {
},
.flags = FLAG_NEED_STATE | FLAG_RESULT_IN_STATE,
.result = (MAX_TAIL_CALL_CNT + 1) * MAX_TESTRUNS,
.has_tail_call = true,
},
{
"Tail call count preserved across function calls",
@ -15357,6 +15363,7 @@ static struct tail_call_test tail_call_tests[] = {
.stack_depth = 8,
.flags = FLAG_NEED_STATE | FLAG_RESULT_IN_STATE,
.result = (MAX_TAIL_CALL_CNT + 1) * MAX_TESTRUNS,
.has_tail_call = true,
},
{
"Tail call error path, NULL target",
@ -15369,6 +15376,7 @@ static struct tail_call_test tail_call_tests[] = {
},
.flags = FLAG_NEED_STATE | FLAG_RESULT_IN_STATE,
.result = MAX_TESTRUNS,
.has_tail_call = true,
},
{
"Tail call error path, index out of range",
@ -15381,6 +15389,7 @@ static struct tail_call_test tail_call_tests[] = {
},
.flags = FLAG_NEED_STATE | FLAG_RESULT_IN_STATE,
.result = MAX_TESTRUNS,
.has_tail_call = true,
},
};
@ -15430,6 +15439,7 @@ static __init int prepare_tail_call_tests(struct bpf_array **pprogs)
fp->len = len;
fp->type = BPF_PROG_TYPE_SOCKET_FILTER;
fp->aux->stack_depth = test->stack_depth;
fp->aux->tail_call_reachable = test->has_tail_call;
memcpy(fp->insnsi, test->insns, len * sizeof(struct bpf_insn));
/* Relocate runtime tail call offsets and addresses */

28
net/bpf/test_run.c
View File

@ -993,7 +993,8 @@ int bpf_prog_test_run_skb(struct bpf_prog *prog, const union bpf_attr *kattr,
void *data;
int ret;
if (kattr->test.flags || kattr->test.cpu || kattr->test.batch_size)
if ((kattr->test.flags & ~BPF_F_TEST_SKB_CHECKSUM_COMPLETE) ||
kattr->test.cpu || kattr->test.batch_size)
return -EINVAL;
data = bpf_test_init(kattr, kattr->test.data_size_in,
@ -1041,6 +1042,7 @@ int bpf_prog_test_run_skb(struct bpf_prog *prog, const union bpf_attr *kattr,
skb_reserve(skb, NET_SKB_PAD + NET_IP_ALIGN);
__skb_put(skb, size);
if (ctx && ctx->ifindex > 1) {
dev = dev_get_by_index(net, ctx->ifindex);
if (!dev) {
@ -1076,9 +1078,19 @@ int bpf_prog_test_run_skb(struct bpf_prog *prog, const union bpf_attr *kattr,
__skb_push(skb, hh_len);
if (is_direct_pkt_access)
bpf_compute_data_pointers(skb);
ret = convert___skb_to_skb(skb, ctx);
if (ret)
goto out;
if (kattr->test.flags & BPF_F_TEST_SKB_CHECKSUM_COMPLETE) {
const int off = skb_network_offset(skb);
int len = skb->len - off;
skb->csum = skb_checksum(skb, off, len, 0);
skb->ip_summed = CHECKSUM_COMPLETE;
}
ret = bpf_test_run(prog, skb, repeat, &retval, &duration, false);
if (ret)
goto out;
@ -1093,6 +1105,20 @@ int bpf_prog_test_run_skb(struct bpf_prog *prog, const union bpf_attr *kattr,
}
memset(__skb_push(skb, hh_len), 0, hh_len);
}
if (kattr->test.flags & BPF_F_TEST_SKB_CHECKSUM_COMPLETE) {
const int off = skb_network_offset(skb);
int len = skb->len - off;
__wsum csum;
csum = skb_checksum(skb, off, len, 0);
if (csum_fold(skb->csum) != csum_fold(csum)) {
ret = -EBADMSG;
goto out;
}
}
convert_skb_to___skb(skb, ctx);
size = skb->len;

56
net/core/filter.c
View File

@ -6827,7 +6827,7 @@ static const struct bpf_func_proto bpf_skc_lookup_tcp_proto = {
.ret_type = RET_PTR_TO_SOCK_COMMON_OR_NULL,
.arg1_type = ARG_PTR_TO_CTX,
.arg2_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg3_type = ARG_CONST_SIZE,
.arg3_type = ARG_CONST_SIZE_OR_ZERO,
.arg4_type = ARG_ANYTHING,
.arg5_type = ARG_ANYTHING,
};
@ -6846,7 +6846,7 @@ static const struct bpf_func_proto bpf_sk_lookup_tcp_proto = {
.ret_type = RET_PTR_TO_SOCKET_OR_NULL,
.arg1_type = ARG_PTR_TO_CTX,
.arg2_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg3_type = ARG_CONST_SIZE,
.arg3_type = ARG_CONST_SIZE_OR_ZERO,
.arg4_type = ARG_ANYTHING,
.arg5_type = ARG_ANYTHING,
};
@ -6865,7 +6865,7 @@ static const struct bpf_func_proto bpf_sk_lookup_udp_proto = {
.ret_type = RET_PTR_TO_SOCKET_OR_NULL,
.arg1_type = ARG_PTR_TO_CTX,
.arg2_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg3_type = ARG_CONST_SIZE,
.arg3_type = ARG_CONST_SIZE_OR_ZERO,
.arg4_type = ARG_ANYTHING,
.arg5_type = ARG_ANYTHING,
};
@ -6889,7 +6889,7 @@ static const struct bpf_func_proto bpf_tc_skc_lookup_tcp_proto = {
.ret_type = RET_PTR_TO_SOCK_COMMON_OR_NULL,
.arg1_type = ARG_PTR_TO_CTX,
.arg2_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg3_type = ARG_CONST_SIZE,
.arg3_type = ARG_CONST_SIZE_OR_ZERO,
.arg4_type = ARG_ANYTHING,
.arg5_type = ARG_ANYTHING,
};
@ -6913,7 +6913,7 @@ static const struct bpf_func_proto bpf_tc_sk_lookup_tcp_proto = {
.ret_type = RET_PTR_TO_SOCKET_OR_NULL,
.arg1_type = ARG_PTR_TO_CTX,
.arg2_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg3_type = ARG_CONST_SIZE,
.arg3_type = ARG_CONST_SIZE_OR_ZERO,
.arg4_type = ARG_ANYTHING,
.arg5_type = ARG_ANYTHING,
};
@ -6937,7 +6937,7 @@ static const struct bpf_func_proto bpf_tc_sk_lookup_udp_proto = {
.ret_type = RET_PTR_TO_SOCKET_OR_NULL,
.arg1_type = ARG_PTR_TO_CTX,
.arg2_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg3_type = ARG_CONST_SIZE,
.arg3_type = ARG_CONST_SIZE_OR_ZERO,
.arg4_type = ARG_ANYTHING,
.arg5_type = ARG_ANYTHING,
};
@ -6975,7 +6975,7 @@ static const struct bpf_func_proto bpf_xdp_sk_lookup_udp_proto = {
.ret_type = RET_PTR_TO_SOCKET_OR_NULL,
.arg1_type = ARG_PTR_TO_CTX,
.arg2_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg3_type = ARG_CONST_SIZE,
.arg3_type = ARG_CONST_SIZE_OR_ZERO,
.arg4_type = ARG_ANYTHING,
.arg5_type = ARG_ANYTHING,
};
@ -6999,7 +6999,7 @@ static const struct bpf_func_proto bpf_xdp_skc_lookup_tcp_proto = {
.ret_type = RET_PTR_TO_SOCK_COMMON_OR_NULL,
.arg1_type = ARG_PTR_TO_CTX,
.arg2_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg3_type = ARG_CONST_SIZE,
.arg3_type = ARG_CONST_SIZE_OR_ZERO,
.arg4_type = ARG_ANYTHING,
.arg5_type = ARG_ANYTHING,
};
@ -7023,7 +7023,7 @@ static const struct bpf_func_proto bpf_xdp_sk_lookup_tcp_proto = {
.ret_type = RET_PTR_TO_SOCKET_OR_NULL,
.arg1_type = ARG_PTR_TO_CTX,
.arg2_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg3_type = ARG_CONST_SIZE,
.arg3_type = ARG_CONST_SIZE_OR_ZERO,
.arg4_type = ARG_ANYTHING,
.arg5_type = ARG_ANYTHING,
};
@ -7043,7 +7043,7 @@ static const struct bpf_func_proto bpf_sock_addr_skc_lookup_tcp_proto = {
.ret_type = RET_PTR_TO_SOCK_COMMON_OR_NULL,
.arg1_type = ARG_PTR_TO_CTX,
.arg2_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg3_type = ARG_CONST_SIZE,
.arg3_type = ARG_CONST_SIZE_OR_ZERO,
.arg4_type = ARG_ANYTHING,
.arg5_type = ARG_ANYTHING,
};
@ -7062,7 +7062,7 @@ static const struct bpf_func_proto bpf_sock_addr_sk_lookup_tcp_proto = {
.ret_type = RET_PTR_TO_SOCKET_OR_NULL,
.arg1_type = ARG_PTR_TO_CTX,
.arg2_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg3_type = ARG_CONST_SIZE,
.arg3_type = ARG_CONST_SIZE_OR_ZERO,
.arg4_type = ARG_ANYTHING,
.arg5_type = ARG_ANYTHING,
};
@ -7081,7 +7081,7 @@ static const struct bpf_func_proto bpf_sock_addr_sk_lookup_udp_proto = {
.ret_type = RET_PTR_TO_SOCKET_OR_NULL,
.arg1_type = ARG_PTR_TO_CTX,
.arg2_type = ARG_PTR_TO_MEM | MEM_RDONLY,
.arg3_type = ARG_CONST_SIZE,
.arg3_type = ARG_CONST_SIZE_OR_ZERO,
.arg4_type = ARG_ANYTHING,
.arg5_type = ARG_ANYTHING,
};
@ -11871,28 +11871,34 @@ bpf_sk_base_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
}
__bpf_kfunc_start_defs();
__bpf_kfunc int bpf_dynptr_from_skb(struct sk_buff *skb, u64 flags,
struct bpf_dynptr_kern *ptr__uninit)
__bpf_kfunc int bpf_dynptr_from_skb(struct __sk_buff *s, u64 flags,
struct bpf_dynptr *ptr__uninit)
{
struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)ptr__uninit;
struct sk_buff *skb = (struct sk_buff *)s;
if (flags) {
bpf_dynptr_set_null(ptr__uninit);
bpf_dynptr_set_null(ptr);
return -EINVAL;
}
bpf_dynptr_init(ptr__uninit, skb, BPF_DYNPTR_TYPE_SKB, 0, skb->len);
bpf_dynptr_init(ptr, skb, BPF_DYNPTR_TYPE_SKB, 0, skb->len);
return 0;
}
__bpf_kfunc int bpf_dynptr_from_xdp(struct xdp_buff *xdp, u64 flags,
struct bpf_dynptr_kern *ptr__uninit)
__bpf_kfunc int bpf_dynptr_from_xdp(struct xdp_md *x, u64 flags,
struct bpf_dynptr *ptr__uninit)
{
struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)ptr__uninit;
struct xdp_buff *xdp = (struct xdp_buff *)x;
if (flags) {
bpf_dynptr_set_null(ptr__uninit);
bpf_dynptr_set_null(ptr);
return -EINVAL;
}
bpf_dynptr_init(ptr__uninit, xdp, BPF_DYNPTR_TYPE_XDP, 0, xdp_get_buff_len(xdp));
bpf_dynptr_init(ptr, xdp, BPF_DYNPTR_TYPE_XDP, 0, xdp_get_buff_len(xdp));
return 0;
}
@ -11918,10 +11924,11 @@ __bpf_kfunc int bpf_sock_addr_set_sun_path(struct bpf_sock_addr_kern *sa_kern,
return 0;
}
__bpf_kfunc int bpf_sk_assign_tcp_reqsk(struct sk_buff *skb, struct sock *sk,
__bpf_kfunc int bpf_sk_assign_tcp_reqsk(struct __sk_buff *s, struct sock *sk,
struct bpf_tcp_req_attrs *attrs, int attrs__sz)
{
#if IS_ENABLED(CONFIG_SYN_COOKIES)
struct sk_buff *skb = (struct sk_buff *)s;
const struct request_sock_ops *ops;
struct inet_request_sock *ireq;
struct tcp_request_sock *treq;
@ -12016,16 +12023,17 @@ __bpf_kfunc int bpf_sk_assign_tcp_reqsk(struct sk_buff *skb, struct sock *sk,
__bpf_kfunc_end_defs();
int bpf_dynptr_from_skb_rdonly(struct sk_buff *skb, u64 flags,
struct bpf_dynptr_kern *ptr__uninit)
int bpf_dynptr_from_skb_rdonly(struct __sk_buff *skb, u64 flags,
struct bpf_dynptr *ptr__uninit)
{
struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)ptr__uninit;
int err;
err = bpf_dynptr_from_skb(skb, flags, ptr__uninit);
if (err)
return err;
bpf_dynptr_set_rdonly(ptr__uninit);
bpf_dynptr_set_rdonly(ptr);
return 0;
}

7
net/netfilter/Makefile
View File

@ -142,8 +142,13 @@ obj-$(CONFIG_NFT_FWD_NETDEV) += nft_fwd_netdev.o
# flow table infrastructure
obj-$(CONFIG_NF_FLOW_TABLE) += nf_flow_table.o
nf_flow_table-objs := nf_flow_table_core.o nf_flow_table_ip.o \
nf_flow_table_offload.o
nf_flow_table_offload.o nf_flow_table_xdp.o
nf_flow_table-$(CONFIG_NF_FLOW_TABLE_PROCFS) += nf_flow_table_procfs.o
ifeq ($(CONFIG_NF_FLOW_TABLE),m)
nf_flow_table-$(CONFIG_DEBUG_INFO_BTF_MODULES) += nf_flow_table_bpf.o
else ifeq ($(CONFIG_NF_FLOW_TABLE),y)
nf_flow_table-$(CONFIG_DEBUG_INFO_BTF) += nf_flow_table_bpf.o
endif
obj-$(CONFIG_NF_FLOW_TABLE_INET) += nf_flow_table_inet.o

121
net/netfilter/nf_flow_table_bpf.c Normal file
View File

@ -0,0 +1,121 @@
// SPDX-License-Identifier: GPL-2.0-only
/* Unstable Flow Table Helpers for XDP hook
*
* These are called from the XDP programs.
* Note that it is allowed to break compatibility for these functions since
* the interface they are exposed through to BPF programs is explicitly
* unstable.
*/
#include <linux/kernel.h>
#include <linux/init.h>
#include <linux/module.h>
#include <net/netfilter/nf_flow_table.h>
#include <linux/bpf.h>
#include <linux/btf.h>
#include <net/xdp.h>
/* bpf_flowtable_opts - options for bpf flowtable helpers
* @error: out parameter, set for any encountered error
*/
struct bpf_flowtable_opts {
s32 error;
};
enum {
NF_BPF_FLOWTABLE_OPTS_SZ = 4,
};
__diag_push();
__diag_ignore_all("-Wmissing-prototypes",
"Global functions as their definitions will be in nf_flow_table BTF");
__bpf_kfunc_start_defs();
static struct flow_offload_tuple_rhash *
bpf_xdp_flow_tuple_lookup(struct net_device *dev,
struct flow_offload_tuple *tuple, __be16 proto)
{
struct flow_offload_tuple_rhash *tuplehash;
struct nf_flowtable *nf_flow_table;
struct flow_offload *nf_flow;
nf_flow_table = nf_flowtable_by_dev(dev);
if (!nf_flow_table)
return ERR_PTR(-ENOENT);
tuplehash = flow_offload_lookup(nf_flow_table, tuple);
if (!tuplehash)
return ERR_PTR(-ENOENT);
nf_flow = container_of(tuplehash, struct flow_offload,
tuplehash[tuplehash->tuple.dir]);
flow_offload_refresh(nf_flow_table, nf_flow, false);
return tuplehash;
}
__bpf_kfunc struct flow_offload_tuple_rhash *
bpf_xdp_flow_lookup(struct xdp_md *ctx, struct bpf_fib_lookup *fib_tuple,
struct bpf_flowtable_opts *opts, u32 opts_len)
{
struct xdp_buff *xdp = (struct xdp_buff *)ctx;
struct flow_offload_tuple tuple = {
.iifidx = fib_tuple->ifindex,
.l3proto = fib_tuple->family,
.l4proto = fib_tuple->l4_protocol,
.src_port = fib_tuple->sport,
.dst_port = fib_tuple->dport,
};
struct flow_offload_tuple_rhash *tuplehash;
__be16 proto;
if (opts_len != NF_BPF_FLOWTABLE_OPTS_SZ) {
opts->error = -EINVAL;
return NULL;
}
switch (fib_tuple->family) {
case AF_INET:
tuple.src_v4.s_addr = fib_tuple->ipv4_src;
tuple.dst_v4.s_addr = fib_tuple->ipv4_dst;
proto = htons(ETH_P_IP);
break;
case AF_INET6:
tuple.src_v6 = *(struct in6_addr *)&fib_tuple->ipv6_src;
tuple.dst_v6 = *(struct in6_addr *)&fib_tuple->ipv6_dst;
proto = htons(ETH_P_IPV6);
break;
default:
opts->error = -EAFNOSUPPORT;
return NULL;
}
tuplehash = bpf_xdp_flow_tuple_lookup(xdp->rxq->dev, &tuple, proto);
if (IS_ERR(tuplehash)) {
opts->error = PTR_ERR(tuplehash);
return NULL;
}
return tuplehash;
}
__diag_pop()
__bpf_kfunc_end_defs();
BTF_KFUNCS_START(nf_ft_kfunc_set)
BTF_ID_FLAGS(func, bpf_xdp_flow_lookup, KF_TRUSTED_ARGS | KF_RET_NULL)
BTF_KFUNCS_END(nf_ft_kfunc_set)
static const struct btf_kfunc_id_set nf_flow_kfunc_set = {
.owner = THIS_MODULE,
.set = &nf_ft_kfunc_set,
};
int nf_flow_register_bpf(void)
{
return register_btf_kfunc_id_set(BPF_PROG_TYPE_XDP,
&nf_flow_kfunc_set);
}
EXPORT_SYMBOL_GPL(nf_flow_register_bpf);

2
net/netfilter/nf_flow_table_inet.c
View File

@ -98,7 +98,7 @@ static int __init nf_flow_inet_module_init(void)
nft_register_flowtable_type(&flowtable_ipv6);
nft_register_flowtable_type(&flowtable_inet);
return 0;
return nf_flow_register_bpf();
}
static void __exit nf_flow_inet_module_exit(void)

2
net/netfilter/nf_flow_table_offload.c
View File

@ -1192,7 +1192,7 @@ int nf_flow_table_offload_setup(struct nf_flowtable *flowtable,
int err;
if (!nf_flowtable_hw_offload(flowtable))
return 0;
return nf_flow_offload_xdp_setup(flowtable, dev, cmd);
if (dev->netdev_ops->ndo_setup_tc)
err = nf_flow_table_offload_cmd(&bo, flowtable, dev, cmd,

147
net/netfilter/nf_flow_table_xdp.c Normal file
View File

@ -0,0 +1,147 @@
// SPDX-License-Identifier: GPL-2.0-only
#include <linux/kernel.h>
#include <linux/module.h>
#include <linux/netfilter.h>
#include <linux/rhashtable.h>
#include <linux/netdevice.h>
#include <net/flow_offload.h>
#include <net/netfilter/nf_flow_table.h>
struct flow_offload_xdp_ft {
struct list_head head;
struct nf_flowtable *ft;
struct rcu_head rcuhead;
};
struct flow_offload_xdp {
struct hlist_node hnode;
unsigned long net_device_addr;
struct list_head head;
};
#define NF_XDP_HT_BITS 4
static DEFINE_HASHTABLE(nf_xdp_hashtable, NF_XDP_HT_BITS);
static DEFINE_MUTEX(nf_xdp_hashtable_lock);
/* caller must hold rcu read lock */
struct nf_flowtable *nf_flowtable_by_dev(const struct net_device *dev)
{
unsigned long key = (unsigned long)dev;
struct flow_offload_xdp *iter;
hash_for_each_possible_rcu(nf_xdp_hashtable, iter, hnode, key) {
if (key == iter->net_device_addr) {
struct flow_offload_xdp_ft *ft_elem;
/* The user is supposed to insert a given net_device
* just into a single nf_flowtable so we always return
* the first element here.
*/
ft_elem = list_first_or_null_rcu(&iter->head,
struct flow_offload_xdp_ft,
head);
return ft_elem ? ft_elem->ft : NULL;
}
}
return NULL;
}
static int nf_flowtable_by_dev_insert(struct nf_flowtable *ft,
const struct net_device *dev)
{
struct flow_offload_xdp *iter, *elem = NULL;
unsigned long key = (unsigned long)dev;
struct flow_offload_xdp_ft *ft_elem;
ft_elem = kzalloc(sizeof(*ft_elem), GFP_KERNEL_ACCOUNT);
if (!ft_elem)
return -ENOMEM;
ft_elem->ft = ft;
mutex_lock(&nf_xdp_hashtable_lock);
hash_for_each_possible(nf_xdp_hashtable, iter, hnode, key) {
if (key == iter->net_device_addr) {
elem = iter;
break;
}
}
if (!elem) {
elem = kzalloc(sizeof(*elem), GFP_KERNEL_ACCOUNT);
if (!elem)
goto err_unlock;
elem->net_device_addr = key;
INIT_LIST_HEAD(&elem->head);
hash_add_rcu(nf_xdp_hashtable, &elem->hnode, key);
}
list_add_tail_rcu(&ft_elem->head, &elem->head);
mutex_unlock(&nf_xdp_hashtable_lock);
return 0;
err_unlock:
mutex_unlock(&nf_xdp_hashtable_lock);
kfree(ft_elem);
return -ENOMEM;
}
static void nf_flowtable_by_dev_remove(struct nf_flowtable *ft,
const struct net_device *dev)
{
struct flow_offload_xdp *iter, *elem = NULL;
unsigned long key = (unsigned long)dev;
mutex_lock(&nf_xdp_hashtable_lock);
hash_for_each_possible(nf_xdp_hashtable, iter, hnode, key) {
if (key == iter->net_device_addr) {
elem = iter;
break;
}
}
if (elem) {
struct flow_offload_xdp_ft *ft_elem, *ft_next;
list_for_each_entry_safe(ft_elem, ft_next, &elem->head, head) {
if (ft_elem->ft == ft) {
list_del_rcu(&ft_elem->head);
kfree_rcu(ft_elem, rcuhead);
}
}
if (list_empty(&elem->head))
hash_del_rcu(&elem->hnode);
else
elem = NULL;
}
mutex_unlock(&nf_xdp_hashtable_lock);
if (elem) {
synchronize_rcu();
kfree(elem);
}
}
int nf_flow_offload_xdp_setup(struct nf_flowtable *flowtable,
struct net_device *dev,
enum flow_block_command cmd)
{
switch (cmd) {
case FLOW_BLOCK_BIND:
return nf_flowtable_by_dev_insert(flowtable, dev);
case FLOW_BLOCK_UNBIND:
nf_flowtable_by_dev_remove(flowtable, dev);
return 0;
}
WARN_ON_ONCE(1);
return 0;
}

7
scripts/Makefile.btf
View File

@ -19,10 +19,15 @@ pahole-flags-$(call test-ge, $(pahole-ver), 125) += --skip_encoding_btf_inconsis
else
# Switch to using --btf_features for v1.26 and later.
pahole-flags-$(call test-ge, $(pahole-ver), 126) = -j --btf_features=encode_force,var,float,enum64,decl_tag,type_tag,optimized_func,consistent_func
pahole-flags-$(call test-ge, $(pahole-ver), 126) = -j --btf_features=encode_force,var,float,enum64,decl_tag,type_tag,optimized_func,consistent_func,decl_tag_kfuncs
ifneq ($(KBUILD_EXTMOD),)
module-pahole-flags-$(call test-ge, $(pahole-ver), 126) += --btf_features=distilled_base
endif
endif
pahole-flags-$(CONFIG_PAHOLE_HAS_LANG_EXCLUDE) += --lang_exclude=rust
export PAHOLE_FLAGS := $(pahole-flags-y)
export MODULE_PAHOLE_FLAGS := $(module-pahole-flags-y)

2
scripts/Makefile.modfinal
View File

@ -41,7 +41,7 @@ quiet_cmd_btf_ko = BTF [M] $@
if [ ! -f vmlinux ]; then \
printf "Skipping BTF generation for %s due to unavailability of vmlinux\n" $@ 1>&2; \
else \
LLVM_OBJCOPY="$(OBJCOPY)" $(PAHOLE) -J $(PAHOLE_FLAGS) --btf_base vmlinux $@; \
LLVM_OBJCOPY="$(OBJCOPY)" $(PAHOLE) -J $(PAHOLE_FLAGS) $(MODULE_PAHOLE_FLAGS) --btf_base vmlinux $@; \
$(RESOLVE_BTFIDS) -b vmlinux $@; \
fi;

55
tools/bpf/bpftool/btf.c
View File

@ -20,6 +20,8 @@
#include "json_writer.h"
#include "main.h"
#define KFUNC_DECL_TAG "bpf_kfunc"
static const char * const btf_kind_str[NR_BTF_KINDS] = {
[BTF_KIND_UNKN] = "UNKNOWN",
[BTF_KIND_INT] = "INT",
@ -461,6 +463,49 @@ static int dump_btf_raw(const struct btf *btf,
return 0;
}
static int dump_btf_kfuncs(struct btf_dump *d, const struct btf *btf)
{
LIBBPF_OPTS(btf_dump_emit_type_decl_opts, opts);
int cnt = btf__type_cnt(btf);
int i;
printf("\n/* BPF kfuncs */\n");
printf("#ifndef BPF_NO_KFUNC_PROTOTYPES\n");
for (i = 1; i < cnt; i++) {
const struct btf_type *t = btf__type_by_id(btf, i);
const char *name;
int err;
if (!btf_is_decl_tag(t))
continue;
if (btf_decl_tag(t)->component_idx != -1)
continue;
name = btf__name_by_offset(btf, t->name_off);
if (strncmp(name, KFUNC_DECL_TAG, sizeof(KFUNC_DECL_TAG)))
continue;
t = btf__type_by_id(btf, t->type);
if (!btf_is_func(t))
continue;
printf("extern ");
opts.field_name = btf__name_by_offset(btf, t->name_off);
err = btf_dump__emit_type_decl(d, t->type, &opts);
if (err)
return err;
printf(" __weak __ksym;\n");
}
printf("#endif\n\n");
return 0;
}
static void __printf(2, 0) btf_dump_printf(void *ctx,
const char *fmt, va_list args)
{
@ -596,6 +641,12 @@ static int dump_btf_c(const struct btf *btf,
printf("#ifndef BPF_NO_PRESERVE_ACCESS_INDEX\n");
printf("#pragma clang attribute push (__attribute__((preserve_access_index)), apply_to = record)\n");
printf("#endif\n\n");
printf("#ifndef __ksym\n");
printf("#define __ksym __attribute__((section(\".ksyms\")))\n");
printf("#endif\n\n");
printf("#ifndef __weak\n");
printf("#define __weak __attribute__((weak))\n");
printf("#endif\n\n");
if (root_type_cnt) {
for (i = 0; i < root_type_cnt; i++) {
@ -615,6 +666,10 @@ static int dump_btf_c(const struct btf *btf,
if (err)
goto done;
}
err = dump_btf_kfuncs(d, btf);
if (err)
goto done;
}
printf("#ifndef BPF_NO_PRESERVE_ACCESS_INDEX\n");

40
tools/bpf/bpftool/cgroup.c
View File

@ -19,6 +19,38 @@
#include "main.h"
static const int cgroup_attach_types[] = {
BPF_CGROUP_INET_INGRESS,
BPF_CGROUP_INET_EGRESS,
BPF_CGROUP_INET_SOCK_CREATE,
BPF_CGROUP_INET_SOCK_RELEASE,
BPF_CGROUP_INET4_BIND,
BPF_CGROUP_INET6_BIND,
BPF_CGROUP_INET4_POST_BIND,
BPF_CGROUP_INET6_POST_BIND,
BPF_CGROUP_INET4_CONNECT,
BPF_CGROUP_INET6_CONNECT,
BPF_CGROUP_UNIX_CONNECT,
BPF_CGROUP_INET4_GETPEERNAME,
BPF_CGROUP_INET6_GETPEERNAME,
BPF_CGROUP_UNIX_GETPEERNAME,
BPF_CGROUP_INET4_GETSOCKNAME,
BPF_CGROUP_INET6_GETSOCKNAME,
BPF_CGROUP_UNIX_GETSOCKNAME,
BPF_CGROUP_UDP4_SENDMSG,
BPF_CGROUP_UDP6_SENDMSG,
BPF_CGROUP_UNIX_SENDMSG,
BPF_CGROUP_UDP4_RECVMSG,
BPF_CGROUP_UDP6_RECVMSG,
BPF_CGROUP_UNIX_RECVMSG,
BPF_CGROUP_SOCK_OPS,
BPF_CGROUP_DEVICE,
BPF_CGROUP_SYSCTL,
BPF_CGROUP_GETSOCKOPT,
BPF_CGROUP_SETSOCKOPT,
BPF_LSM_CGROUP
};
#define HELP_SPEC_ATTACH_FLAGS \
"ATTACH_FLAGS := { multi | override }"
@ -183,13 +215,13 @@ static int count_attached_bpf_progs(int cgroup_fd, enum bpf_attach_type type)
static int cgroup_has_attached_progs(int cgroup_fd)
{
enum bpf_attach_type type;
unsigned int i = 0;
bool no_prog = true;
for (type = 0; type < __MAX_BPF_ATTACH_TYPE; type++) {
int count = count_attached_bpf_progs(cgroup_fd, type);
for (i = 0; i < ARRAY_SIZE(cgroup_attach_types); i++) {
int count = count_attached_bpf_progs(cgroup_fd, cgroup_attach_types[i]);
if (count < 0 && errno != EINVAL)
if (count < 0)
return -1;
if (count > 0) {

2
tools/bpf/bpftool/gen.c
View File

@ -1272,6 +1272,8 @@ static int do_skeleton(int argc, char **argv)
#include <stdlib.h> \n\
#include <bpf/libbpf.h> \n\
\n\
#define BPF_SKEL_SUPPORTS_MAP_AUTO_ATTACH 1 \n\
\n\
struct %1$s { \n\
struct bpf_object_skeleton *skeleton; \n\
struct bpf_object *obj; \n\

4
tools/bpf/bpftool/prog.c
View File

@ -1813,6 +1813,10 @@ offload_dev:
}
if (pinmaps) {
err = create_and_mount_bpffs_dir(pinmaps);
if (err)
goto err_unpin;
err = bpf_object__pin_maps(obj, pinmaps);
if (err) {
p_err("failed to pin all maps");

8
tools/bpf/resolve_btfids/main.c
View File

@ -409,6 +409,14 @@ static int elf_collect(struct object *obj)
obj->efile.idlist = data;
obj->efile.idlist_shndx = idx;
obj->efile.idlist_addr = sh.sh_addr;
} else if (!strcmp(name, BTF_BASE_ELF_SEC)) {
/* If a .BTF.base section is found, do not resolve
* BTF ids relative to vmlinux; resolve relative
* to the .BTF.base section instead. btf__parse_split()
* will take care of this once the base BTF it is
* passed is NULL.
*/
obj->base_btf_path = NULL;
}
if (compressed_section_fix(elf, scn, &sh))

2
tools/include/uapi/linux/bpf.h
View File

@ -1425,6 +1425,8 @@ enum {
#define BPF_F_TEST_RUN_ON_CPU (1U << 0)
/* If set, XDP frames will be transmitted after processing */
#define BPF_F_TEST_XDP_LIVE_FRAMES (1U << 1)
/* If set, apply CHECKSUM_COMPLETE to skb and validate the checksum */
#define BPF_F_TEST_SKB_CHECKSUM_COMPLETE (1U << 2)
/* type for BPF_ENABLE_STATS */
enum bpf_stats_type {

2
tools/lib/bpf/Build
View File

@ -1,4 +1,4 @@
libbpf-y := libbpf.o bpf.o nlattr.o btf.o libbpf_errno.o str_error.o \
netlink.o bpf_prog_linfo.o libbpf_probes.o hashmap.o \
btf_dump.o ringbuf.o strset.o linker.o gen_loader.o relo_core.o \
usdt.o zip.o elf.o features.o
usdt.o zip.o elf.o features.o btf_iter.o btf_relocate.o

684
tools/lib/bpf/btf.c
View File

@ -116,6 +116,9 @@ struct btf {
/* whether strings are already deduplicated */
bool strs_deduped;
/* whether base_btf should be freed in btf_free for this instance */
bool owns_base;
/* BTF object FD, if loaded into kernel */
int fd;
@ -598,7 +601,7 @@ static int btf_sanity_check(const struct btf *btf)
__u32 i, n = btf__type_cnt(btf);
int err;
for (i = 1; i < n; i++) {
for (i = btf->start_id; i < n; i++) {
t = btf_type_by_id(btf, i);
err = btf_validate_type(btf, t, i);
if (err)
@ -969,6 +972,8 @@ void btf__free(struct btf *btf)
free(btf->raw_data);
free(btf->raw_data_swapped);
free(btf->type_offs);
if (btf->owns_base)
btf__free(btf->base_btf);
free(btf);
}
@ -1084,16 +1089,86 @@ struct btf *btf__new_split(const void *data, __u32 size, struct btf *base_btf)
return libbpf_ptr(btf_new(data, size, base_btf));
}
struct btf_elf_secs {
Elf_Data *btf_data;
Elf_Data *btf_ext_data;
Elf_Data *btf_base_data;
};
static int btf_find_elf_sections(Elf *elf, const char *path, struct btf_elf_secs *secs)
{
Elf_Scn *scn = NULL;
Elf_Data *data;
GElf_Ehdr ehdr;
size_t shstrndx;
int idx = 0;
if (!gelf_getehdr(elf, &ehdr)) {
pr_warn("failed to get EHDR from %s\n", path);
goto err;
}
if (elf_getshdrstrndx(elf, &shstrndx)) {
pr_warn("failed to get section names section index for %s\n",
path);
goto err;
}
if (!elf_rawdata(elf_getscn(elf, shstrndx), NULL)) {
pr_warn("failed to get e_shstrndx from %s\n", path);
goto err;
}
while ((scn = elf_nextscn(elf, scn)) != NULL) {
Elf_Data **field;
GElf_Shdr sh;
char *name;
idx++;
if (gelf_getshdr(scn, &sh) != &sh) {
pr_warn("failed to get section(%d) header from %s\n",
idx, path);
goto err;
}
name = elf_strptr(elf, shstrndx, sh.sh_name);
if (!name) {
pr_warn("failed to get section(%d) name from %s\n",
idx, path);
goto err;
}
if (strcmp(name, BTF_ELF_SEC) == 0)
field = &secs->btf_data;
else if (strcmp(name, BTF_EXT_ELF_SEC) == 0)
field = &secs->btf_ext_data;
else if (strcmp(name, BTF_BASE_ELF_SEC) == 0)
field = &secs->btf_base_data;
else
continue;
data = elf_getdata(scn, 0);
if (!data) {
pr_warn("failed to get section(%d, %s) data from %s\n",
idx, name, path);
goto err;
}
*field = data;
}
return 0;
err:
return -LIBBPF_ERRNO__FORMAT;
}
static struct btf *btf_parse_elf(const char *path, struct btf *base_btf,
struct btf_ext **btf_ext)
{
Elf_Data *btf_data = NULL, *btf_ext_data = NULL;
int err = 0, fd = -1, idx = 0;
struct btf_elf_secs secs = {};
struct btf *dist_base_btf = NULL;
struct btf *btf = NULL;
Elf_Scn *scn = NULL;
int err = 0, fd = -1;
Elf *elf = NULL;
GElf_Ehdr ehdr;
size_t shstrndx;
if (elf_version(EV_CURRENT) == EV_NONE) {
pr_warn("failed to init libelf for %s\n", path);
@ -1107,73 +1182,48 @@ static struct btf *btf_parse_elf(const char *path, struct btf *base_btf,
return ERR_PTR(err);
}
err = -LIBBPF_ERRNO__FORMAT;
elf = elf_begin(fd, ELF_C_READ, NULL);
if (!elf) {
pr_warn("failed to open %s as ELF file\n", path);
goto done;
}
if (!gelf_getehdr(elf, &ehdr)) {
pr_warn("failed to get EHDR from %s\n", path);
err = btf_find_elf_sections(elf, path, &secs);
if (err)
goto done;
}
if (elf_getshdrstrndx(elf, &shstrndx)) {
pr_warn("failed to get section names section index for %s\n",
path);
goto done;
}
if (!elf_rawdata(elf_getscn(elf, shstrndx), NULL)) {
pr_warn("failed to get e_shstrndx from %s\n", path);
goto done;
}
while ((scn = elf_nextscn(elf, scn)) != NULL) {
GElf_Shdr sh;
char *name;
idx++;
if (gelf_getshdr(scn, &sh) != &sh) {
pr_warn("failed to get section(%d) header from %s\n",
idx, path);
goto done;
}
name = elf_strptr(elf, shstrndx, sh.sh_name);
if (!name) {
pr_warn("failed to get section(%d) name from %s\n",
idx, path);
goto done;
}
if (strcmp(name, BTF_ELF_SEC) == 0) {
btf_data = elf_getdata(scn, 0);
if (!btf_data) {
pr_warn("failed to get section(%d, %s) data from %s\n",
idx, name, path);
goto done;
}
continue;
} else if (btf_ext && strcmp(name, BTF_EXT_ELF_SEC) == 0) {
btf_ext_data = elf_getdata(scn, 0);
if (!btf_ext_data) {
pr_warn("failed to get section(%d, %s) data from %s\n",
idx, name, path);
goto done;
}
continue;
}
}
if (!btf_data) {
if (!secs.btf_data) {
pr_warn("failed to find '%s' ELF section in %s\n", BTF_ELF_SEC, path);
err = -ENODATA;
goto done;
}
btf = btf_new(btf_data->d_buf, btf_data->d_size, base_btf);
err = libbpf_get_error(btf);
if (err)
if (secs.btf_base_data) {
dist_base_btf = btf_new(secs.btf_base_data->d_buf, secs.btf_base_data->d_size,
NULL);
if (IS_ERR(dist_base_btf)) {
err = PTR_ERR(dist_base_btf);
dist_base_btf = NULL;
goto done;
}
}
btf = btf_new(secs.btf_data->d_buf, secs.btf_data->d_size,
dist_base_btf ?: base_btf);
if (IS_ERR(btf)) {
err = PTR_ERR(btf);
goto done;
}
if (dist_base_btf && base_btf) {
err = btf__relocate(btf, base_btf);
if (err)
goto done;
btf__free(dist_base_btf);
dist_base_btf = NULL;
}
if (dist_base_btf)
btf->owns_base = true;
switch (gelf_getclass(elf)) {
case ELFCLASS32:
@ -1187,11 +1237,12 @@ static struct btf *btf_parse_elf(const char *path, struct btf *base_btf,
break;
}
if (btf_ext && btf_ext_data) {
*btf_ext = btf_ext__new(btf_ext_data->d_buf, btf_ext_data->d_size);
err = libbpf_get_error(*btf_ext);
if (err)
if (btf_ext && secs.btf_ext_data) {
*btf_ext = btf_ext__new(secs.btf_ext_data->d_buf, secs.btf_ext_data->d_size);
if (IS_ERR(*btf_ext)) {
err = PTR_ERR(*btf_ext);
goto done;
}
} else if (btf_ext) {
*btf_ext = NULL;
}
@ -1205,6 +1256,7 @@ done:
if (btf_ext)
btf_ext__free(*btf_ext);
btf__free(dist_base_btf);
btf__free(btf);
return ERR_PTR(err);
@ -1770,9 +1822,8 @@ static int btf_rewrite_str(struct btf_pipe *p, __u32 *str_off)
return 0;
}
int btf__add_type(struct btf *btf, const struct btf *src_btf, const struct btf_type *src_type)
static int btf_add_type(struct btf_pipe *p, const struct btf_type *src_type)
{
struct btf_pipe p = { .src = src_btf, .dst = btf };
struct btf_field_iter it;
struct btf_type *t;
__u32 *str_off;
@ -1783,10 +1834,10 @@ int btf__add_type(struct btf *btf, const struct btf *src_btf, const struct btf_t
return libbpf_err(sz);
/* deconstruct BTF, if necessary, and invalidate raw_data */
if (btf_ensure_modifiable(btf))
if (btf_ensure_modifiable(p->dst))
return libbpf_err(-ENOMEM);
t = btf_add_type_mem(btf, sz);
t = btf_add_type_mem(p->dst, sz);
if (!t)
return libbpf_err(-ENOMEM);
@ -1797,12 +1848,19 @@ int btf__add_type(struct btf *btf, const struct btf *src_btf, const struct btf_t
return libbpf_err(err);
while ((str_off = btf_field_iter_next(&it))) {
err = btf_rewrite_str(&p, str_off);
err = btf_rewrite_str(p, str_off);
if (err)
return libbpf_err(err);
}
return btf_commit_type(btf, sz);
return btf_commit_type(p->dst, sz);
}
int btf__add_type(struct btf *btf, const struct btf *src_btf, const struct btf_type *src_type)
{
struct btf_pipe p = { .src = src_btf, .dst = btf };
return btf_add_type(&p, src_type);
}
static size_t btf_dedup_identity_hash_fn(long key, void *ctx);
@ -5035,168 +5093,6 @@ struct btf *btf__load_module_btf(const char *module_name, struct btf *vmlinux_bt
return btf__parse_split(path, vmlinux_btf);
}
int btf_field_iter_init(struct btf_field_iter *it, struct btf_type *t, enum btf_field_iter_kind iter_kind)
{
it->p = NULL;
it->m_idx = -1;
it->off_idx = 0;
it->vlen = 0;
switch (iter_kind) {
case BTF_FIELD_ITER_IDS:
switch (btf_kind(t)) {
case BTF_KIND_UNKN:
case BTF_KIND_INT:
case BTF_KIND_FLOAT:
case BTF_KIND_ENUM:
case BTF_KIND_ENUM64:
it->desc = (struct btf_field_desc) {};
break;
case BTF_KIND_FWD:
case BTF_KIND_CONST:
case BTF_KIND_VOLATILE:
case BTF_KIND_RESTRICT:
case BTF_KIND_PTR:
case BTF_KIND_TYPEDEF:
case BTF_KIND_FUNC:
case BTF_KIND_VAR:
case BTF_KIND_DECL_TAG:
case BTF_KIND_TYPE_TAG:
it->desc = (struct btf_field_desc) { 1, {offsetof(struct btf_type, type)} };
break;
case BTF_KIND_ARRAY:
it->desc = (struct btf_field_desc) {
2, {sizeof(struct btf_type) + offsetof(struct btf_array, type),
sizeof(struct btf_type) + offsetof(struct btf_array, index_type)}
};
break;
case BTF_KIND_STRUCT:
case BTF_KIND_UNION:
it->desc = (struct btf_field_desc) {
0, {},
sizeof(struct btf_member),
1, {offsetof(struct btf_member, type)}
};
break;
case BTF_KIND_FUNC_PROTO:
it->desc = (struct btf_field_desc) {
1, {offsetof(struct btf_type, type)},
sizeof(struct btf_param),
1, {offsetof(struct btf_param, type)}
};
break;
case BTF_KIND_DATASEC:
it->desc = (struct btf_field_desc) {
0, {},
sizeof(struct btf_var_secinfo),
1, {offsetof(struct btf_var_secinfo, type)}
};
break;
default:
return -EINVAL;
}
break;
case BTF_FIELD_ITER_STRS:
switch (btf_kind(t)) {
case BTF_KIND_UNKN:
it->desc = (struct btf_field_desc) {};
break;
case BTF_KIND_INT:
case BTF_KIND_FLOAT:
case BTF_KIND_FWD:
case BTF_KIND_ARRAY:
case BTF_KIND_CONST:
case BTF_KIND_VOLATILE:
case BTF_KIND_RESTRICT:
case BTF_KIND_PTR:
case BTF_KIND_TYPEDEF:
case BTF_KIND_FUNC:
case BTF_KIND_VAR:
case BTF_KIND_DECL_TAG:
case BTF_KIND_TYPE_TAG:
case BTF_KIND_DATASEC:
it->desc = (struct btf_field_desc) {
1, {offsetof(struct btf_type, name_off)}
};
break;
case BTF_KIND_ENUM:
it->desc = (struct btf_field_desc) {
1, {offsetof(struct btf_type, name_off)},
sizeof(struct btf_enum),
1, {offsetof(struct btf_enum, name_off)}
};
break;
case BTF_KIND_ENUM64:
it->desc = (struct btf_field_desc) {
1, {offsetof(struct btf_type, name_off)},
sizeof(struct btf_enum64),
1, {offsetof(struct btf_enum64, name_off)}
};
break;
case BTF_KIND_STRUCT:
case BTF_KIND_UNION:
it->desc = (struct btf_field_desc) {
1, {offsetof(struct btf_type, name_off)},
sizeof(struct btf_member),
1, {offsetof(struct btf_member, name_off)}
};
break;
case BTF_KIND_FUNC_PROTO:
it->desc = (struct btf_field_desc) {
1, {offsetof(struct btf_type, name_off)},
sizeof(struct btf_param),
1, {offsetof(struct btf_param, name_off)}
};
break;
default:
return -EINVAL;
}
break;
default:
return -EINVAL;
}
if (it->desc.m_sz)
it->vlen = btf_vlen(t);
it->p = t;
return 0;
}
__u32 *btf_field_iter_next(struct btf_field_iter *it)
{
if (!it->p)
return NULL;
if (it->m_idx < 0) {
if (it->off_idx < it->desc.t_off_cnt)
return it->p + it->desc.t_offs[it->off_idx++];
/* move to per-member iteration */
it->m_idx = 0;
it->p += sizeof(struct btf_type);
it->off_idx = 0;
}
/* if type doesn't have members, stop */
if (it->desc.m_sz == 0) {
it->p = NULL;
return NULL;
}
if (it->off_idx >= it->desc.m_off_cnt) {
/* exhausted this member's fields, go to the next member */
it->m_idx++;
it->p += it->desc.m_sz;
it->off_idx = 0;
}
if (it->m_idx < it->vlen)
return it->p + it->desc.m_offs[it->off_idx++];
it->p = NULL;
return NULL;
}
int btf_ext_visit_type_ids(struct btf_ext *btf_ext, type_id_visit_fn visit, void *ctx)
{
const struct btf_ext_info *seg;
@ -5276,3 +5172,325 @@ int btf_ext_visit_str_offs(struct btf_ext *btf_ext, str_off_visit_fn visit, void
return 0;
}
struct btf_distill {
struct btf_pipe pipe;
int *id_map;
unsigned int split_start_id;
unsigned int split_start_str;
int diff_id;
};
static int btf_add_distilled_type_ids(struct btf_distill *dist, __u32 i)
{
struct btf_type *split_t = btf_type_by_id(dist->pipe.src, i);
struct btf_field_iter it;
__u32 *id;
int err;
err = btf_field_iter_init(&it, split_t, BTF_FIELD_ITER_IDS);
if (err)
return err;
while ((id = btf_field_iter_next(&it))) {
struct btf_type *base_t;
if (!*id)
continue;
/* split BTF id, not needed */
if (*id >= dist->split_start_id)
continue;
/* already added ? */
if (dist->id_map[*id] > 0)
continue;
/* only a subset of base BTF types should be referenced from
* split BTF; ensure nothing unexpected is referenced.
*/
base_t = btf_type_by_id(dist->pipe.src, *id);
switch (btf_kind(base_t)) {
case BTF_KIND_INT:
case BTF_KIND_FLOAT:
case BTF_KIND_FWD:
case BTF_KIND_ARRAY:
case BTF_KIND_STRUCT:
case BTF_KIND_UNION:
case BTF_KIND_TYPEDEF:
case BTF_KIND_ENUM:
case BTF_KIND_ENUM64:
case BTF_KIND_PTR:
case BTF_KIND_CONST:
case BTF_KIND_RESTRICT:
case BTF_KIND_VOLATILE:
case BTF_KIND_FUNC_PROTO:
case BTF_KIND_TYPE_TAG:
dist->id_map[*id] = *id;
break;
default:
pr_warn("unexpected reference to base type[%u] of kind [%u] when creating distilled base BTF.\n",
*id, btf_kind(base_t));
return -EINVAL;
}
/* If a base type is used, ensure types it refers to are
* marked as used also; so for example if we find a PTR to INT
* we need both the PTR and INT.
*
* The only exception is named struct/unions, since distilled
* base BTF composite types have no members.
*/
if (btf_is_composite(base_t) && base_t->name_off)
continue;
err = btf_add_distilled_type_ids(dist, *id);
if (err)
return err;
}
return 0;
}
static int btf_add_distilled_types(struct btf_distill *dist)
{
bool adding_to_base = dist->pipe.dst->start_id == 1;
int id = btf__type_cnt(dist->pipe.dst);
struct btf_type *t;
int i, err = 0;
/* Add types for each of the required references to either distilled
* base or split BTF, depending on type characteristics.
*/
for (i = 1; i < dist->split_start_id; i++) {
const char *name;
int kind;
if (!dist->id_map[i])
continue;
t = btf_type_by_id(dist->pipe.src, i);
kind = btf_kind(t);
name = btf__name_by_offset(dist->pipe.src, t->name_off);
switch (kind) {
case BTF_KIND_INT:
case BTF_KIND_FLOAT:
case BTF_KIND_FWD:
/* Named int, float, fwd are added to base. */
if (!adding_to_base)
continue;
err = btf_add_type(&dist->pipe, t);
break;
case BTF_KIND_STRUCT:
case BTF_KIND_UNION:
/* Named struct/union are added to base as 0-vlen
* struct/union of same size. Anonymous struct/unions
* are added to split BTF as-is.
*/
if (adding_to_base) {
if (!t->name_off)
continue;
err = btf_add_composite(dist->pipe.dst, kind, name, t->size);
} else {
if (t->name_off)
continue;
err = btf_add_type(&dist->pipe, t);
}
break;
case BTF_KIND_ENUM:
case BTF_KIND_ENUM64:
/* Named enum[64]s are added to base as a sized
* enum; relocation will match with appropriately-named
* and sized enum or enum64.
*
* Anonymous enums are added to split BTF as-is.
*/
if (adding_to_base) {
if (!t->name_off)
continue;
err = btf__add_enum(dist->pipe.dst, name, t->size);
} else {
if (t->name_off)
continue;
err = btf_add_type(&dist->pipe, t);
}
break;
case BTF_KIND_ARRAY:
case BTF_KIND_TYPEDEF:
case BTF_KIND_PTR:
case BTF_KIND_CONST:
case BTF_KIND_RESTRICT:
case BTF_KIND_VOLATILE:
case BTF_KIND_FUNC_PROTO:
case BTF_KIND_TYPE_TAG:
/* All other types are added to split BTF. */
if (adding_to_base)
continue;
err = btf_add_type(&dist->pipe, t);
break;
default:
pr_warn("unexpected kind when adding base type '%s'[%u] of kind [%u] to distilled base BTF.\n",
name, i, kind);
return -EINVAL;
}
if (err < 0)
break;
dist->id_map[i] = id++;
}
return err;
}
/* Split BTF ids without a mapping will be shifted downwards since distilled
* base BTF is smaller than the original base BTF. For those that have a
* mapping (either to base or updated split BTF), update the id based on
* that mapping.
*/
static int btf_update_distilled_type_ids(struct btf_distill *dist, __u32 i)
{
struct btf_type *t = btf_type_by_id(dist->pipe.dst, i);
struct btf_field_iter it;
__u32 *id;
int err;
err = btf_field_iter_init(&it, t, BTF_FIELD_ITER_IDS);
if (err)
return err;
while ((id = btf_field_iter_next(&it))) {
if (dist->id_map[*id])
*id = dist->id_map[*id];
else if (*id >= dist->split_start_id)
*id -= dist->diff_id;
}
return 0;
}
/* Create updated split BTF with distilled base BTF; distilled base BTF
* consists of BTF information required to clarify the types that split
* BTF refers to, omitting unneeded details. Specifically it will contain
* base types and memberless definitions of named structs, unions and enumerated
* types. Associated reference types like pointers, arrays and anonymous
* structs, unions and enumerated types will be added to split BTF.
* Size is recorded for named struct/unions to help guide matching to the
* target base BTF during later relocation.
*
* The only case where structs, unions or enumerated types are fully represented
* is when they are anonymous; in such cases, the anonymous type is added to
* split BTF in full.
*
* We return newly-created split BTF where the split BTF refers to a newly-created
* distilled base BTF. Both must be freed separately by the caller.
*/
int btf__distill_base(const struct btf *src_btf, struct btf **new_base_btf,
struct btf **new_split_btf)
{
struct btf *new_base = NULL, *new_split = NULL;
const struct btf *old_base;
unsigned int n = btf__type_cnt(src_btf);
struct btf_distill dist = {};
struct btf_type *t;
int i, err = 0;
/* src BTF must be split BTF. */
old_base = btf__base_btf(src_btf);
if (!new_base_btf || !new_split_btf || !old_base)
return libbpf_err(-EINVAL);
new_base = btf__new_empty();
if (!new_base)
return libbpf_err(-ENOMEM);
dist.id_map = calloc(n, sizeof(*dist.id_map));
if (!dist.id_map) {
err = -ENOMEM;
goto done;
}
dist.pipe.src = src_btf;
dist.pipe.dst = new_base;
dist.pipe.str_off_map = hashmap__new(btf_dedup_identity_hash_fn, btf_dedup_equal_fn, NULL);
if (IS_ERR(dist.pipe.str_off_map)) {
err = -ENOMEM;
goto done;
}
dist.split_start_id = btf__type_cnt(old_base);
dist.split_start_str = old_base->hdr->str_len;
/* Pass over src split BTF; generate the list of base BTF type ids it
* references; these will constitute our distilled BTF set to be
* distributed over base and split BTF as appropriate.
*/
for (i = src_btf->start_id; i < n; i++) {
err = btf_add_distilled_type_ids(&dist, i);
if (err < 0)
goto done;
}
/* Next add types for each of the required references to base BTF and split BTF
* in turn.
*/
err = btf_add_distilled_types(&dist);
if (err < 0)
goto done;
/* Create new split BTF with distilled base BTF as its base; the final
* state is split BTF with distilled base BTF that represents enough
* about its base references to allow it to be relocated with the base
* BTF available.
*/
new_split = btf__new_empty_split(new_base);
if (!new_split) {
err = -errno;
goto done;
}
dist.pipe.dst = new_split;
/* First add all split types */
for (i = src_btf->start_id; i < n; i++) {
t = btf_type_by_id(src_btf, i);
err = btf_add_type(&dist.pipe, t);
if (err < 0)
goto done;
}
/* Now add distilled types to split BTF that are not added to base. */
err = btf_add_distilled_types(&dist);
if (err < 0)
goto done;
/* All split BTF ids will be shifted downwards since there are less base
* BTF ids in distilled base BTF.
*/
dist.diff_id = dist.split_start_id - btf__type_cnt(new_base);
n = btf__type_cnt(new_split);
/* Now update base/split BTF ids. */
for (i = 1; i < n; i++) {
err = btf_update_distilled_type_ids(&dist, i);
if (err < 0)
break;
}
done:
free(dist.id_map);
hashmap__free(dist.pipe.str_off_map);
if (err) {
btf__free(new_split);
btf__free(new_base);
return libbpf_err(err);
}
*new_base_btf = new_base;
*new_split_btf = new_split;
return 0;
}
const struct btf_header *btf_header(const struct btf *btf)
{
return btf->hdr;
}
void btf_set_base_btf(struct btf *btf, const struct btf *base_btf)
{
btf->base_btf = (struct btf *)base_btf;
btf->start_id = btf__type_cnt(base_btf);
btf->start_str_off = base_btf->hdr->str_len;
}
int btf__relocate(struct btf *btf, const struct btf *base_btf)
{
int err = btf_relocate(btf, base_btf, NULL);
if (!err)
btf->owns_base = false;
return libbpf_err(err);
}

36
tools/lib/bpf/btf.h
View File

@ -18,6 +18,7 @@ extern "C" {
#define BTF_ELF_SEC ".BTF"
#define BTF_EXT_ELF_SEC ".BTF.ext"
#define BTF_BASE_ELF_SEC ".BTF.base"
#define MAPS_ELF_SEC ".maps"
struct btf;
@ -107,6 +108,27 @@ LIBBPF_API struct btf *btf__new_empty(void);
*/
LIBBPF_API struct btf *btf__new_empty_split(struct btf *base_btf);
/**
* @brief **btf__distill_base()** creates new versions of the split BTF
* *src_btf* and its base BTF. The new base BTF will only contain the types
* needed to improve robustness of the split BTF to small changes in base BTF.
* When that split BTF is loaded against a (possibly changed) base, this
* distilled base BTF will help update references to that (possibly changed)
* base BTF.
*
* Both the new split and its associated new base BTF must be freed by
* the caller.
*
* If successful, 0 is returned and **new_base_btf** and **new_split_btf**
* will point at new base/split BTF. Both the new split and its associated
* new base BTF must be freed by the caller.
*
* A negative value is returned on error and the thread-local `errno` variable
* is set to the error code as well.
*/
LIBBPF_API int btf__distill_base(const struct btf *src_btf, struct btf **new_base_btf,
struct btf **new_split_btf);
LIBBPF_API struct btf *btf__parse(const char *path, struct btf_ext **btf_ext);
LIBBPF_API struct btf *btf__parse_split(const char *path, struct btf *base_btf);
LIBBPF_API struct btf *btf__parse_elf(const char *path, struct btf_ext **btf_ext);
@ -231,6 +253,20 @@ struct btf_dedup_opts {
LIBBPF_API int btf__dedup(struct btf *btf, const struct btf_dedup_opts *opts);
/**
* @brief **btf__relocate()** will check the split BTF *btf* for references
* to base BTF kinds, and verify those references are compatible with
* *base_btf*; if they are, *btf* is adjusted such that is re-parented to
* *base_btf* and type ids and strings are adjusted to accommodate this.
*
* If successful, 0 is returned and **btf** now has **base_btf** as its
* base.
*
* A negative value is returned on error and the thread-local `errno` variable
* is set to the error code as well.
*/
LIBBPF_API int btf__relocate(struct btf *btf, const struct btf *base_btf);
struct btf_dump;
struct btf_dump_opts {

177
tools/lib/bpf/btf_iter.c Normal file
View File

@ -0,0 +1,177 @@
// SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause)
/* Copyright (c) 2021 Facebook */
/* Copyright (c) 2024, Oracle and/or its affiliates. */
#ifdef __KERNEL__
#include <linux/bpf.h>
#include <linux/btf.h>
#define btf_var_secinfos(t) (struct btf_var_secinfo *)btf_type_var_secinfo(t)
#else
#include "btf.h"
#include "libbpf_internal.h"
#endif
int btf_field_iter_init(struct btf_field_iter *it, struct btf_type *t,
enum btf_field_iter_kind iter_kind)
{
it->p = NULL;
it->m_idx = -1;
it->off_idx = 0;
it->vlen = 0;
switch (iter_kind) {
case BTF_FIELD_ITER_IDS:
switch (btf_kind(t)) {
case BTF_KIND_UNKN:
case BTF_KIND_INT:
case BTF_KIND_FLOAT:
case BTF_KIND_ENUM:
case BTF_KIND_ENUM64:
it->desc = (struct btf_field_desc) {};
break;
case BTF_KIND_FWD:
case BTF_KIND_CONST:
case BTF_KIND_VOLATILE:
case BTF_KIND_RESTRICT:
case BTF_KIND_PTR:
case BTF_KIND_TYPEDEF:
case BTF_KIND_FUNC:
case BTF_KIND_VAR:
case BTF_KIND_DECL_TAG:
case BTF_KIND_TYPE_TAG:
it->desc = (struct btf_field_desc) { 1, {offsetof(struct btf_type, type)} };
break;
case BTF_KIND_ARRAY:
it->desc = (struct btf_field_desc) {
2, {sizeof(struct btf_type) + offsetof(struct btf_array, type),
sizeof(struct btf_type) + offsetof(struct btf_array, index_type)}
};
break;
case BTF_KIND_STRUCT:
case BTF_KIND_UNION:
it->desc = (struct btf_field_desc) {
0, {},
sizeof(struct btf_member),
1, {offsetof(struct btf_member, type)}
};
break;
case BTF_KIND_FUNC_PROTO:
it->desc = (struct btf_field_desc) {
1, {offsetof(struct btf_type, type)},
sizeof(struct btf_param),
1, {offsetof(struct btf_param, type)}
};
break;
case BTF_KIND_DATASEC:
it->desc = (struct btf_field_desc) {
0, {},
sizeof(struct btf_var_secinfo),
1, {offsetof(struct btf_var_secinfo, type)}
};
break;
default:
return -EINVAL;
}
break;
case BTF_FIELD_ITER_STRS:
switch (btf_kind(t)) {
case BTF_KIND_UNKN:
it->desc = (struct btf_field_desc) {};
break;
case BTF_KIND_INT:
case BTF_KIND_FLOAT:
case BTF_KIND_FWD:
case BTF_KIND_ARRAY:
case BTF_KIND_CONST:
case BTF_KIND_VOLATILE:
case BTF_KIND_RESTRICT:
case BTF_KIND_PTR:
case BTF_KIND_TYPEDEF:
case BTF_KIND_FUNC:
case BTF_KIND_VAR:
case BTF_KIND_DECL_TAG:
case BTF_KIND_TYPE_TAG:
case BTF_KIND_DATASEC:
it->desc = (struct btf_field_desc) {
1, {offsetof(struct btf_type, name_off)}
};
break;
case BTF_KIND_ENUM:
it->desc = (struct btf_field_desc) {
1, {offsetof(struct btf_type, name_off)},
sizeof(struct btf_enum),
1, {offsetof(struct btf_enum, name_off)}
};
break;
case BTF_KIND_ENUM64:
it->desc = (struct btf_field_desc) {
1, {offsetof(struct btf_type, name_off)},
sizeof(struct btf_enum64),
1, {offsetof(struct btf_enum64, name_off)}
};
break;
case BTF_KIND_STRUCT:
case BTF_KIND_UNION:
it->desc = (struct btf_field_desc) {
1, {offsetof(struct btf_type, name_off)},
sizeof(struct btf_member),
1, {offsetof(struct btf_member, name_off)}
};
break;
case BTF_KIND_FUNC_PROTO:
it->desc = (struct btf_field_desc) {
1, {offsetof(struct btf_type, name_off)},
sizeof(struct btf_param),
1, {offsetof(struct btf_param, name_off)}
};
break;
default:
return -EINVAL;
}
break;
default:
return -EINVAL;
}
if (it->desc.m_sz)
it->vlen = btf_vlen(t);
it->p = t;
return 0;
}
__u32 *btf_field_iter_next(struct btf_field_iter *it)
{
if (!it->p)
return NULL;
if (it->m_idx < 0) {
if (it->off_idx < it->desc.t_off_cnt)
return it->p + it->desc.t_offs[it->off_idx++];
/* move to per-member iteration */
it->m_idx = 0;
it->p += sizeof(struct btf_type);
it->off_idx = 0;
}
/* if type doesn't have members, stop */
if (it->desc.m_sz == 0) {
it->p = NULL;
return NULL;
}
if (it->off_idx >= it->desc.m_off_cnt) {
/* exhausted this member's fields, go to the next member */
it->m_idx++;
it->p += it->desc.m_sz;
it->off_idx = 0;
}
if (it->m_idx < it->vlen)
return it->p + it->desc.m_offs[it->off_idx++];
it->p = NULL;
return NULL;
}

519
tools/lib/bpf/btf_relocate.c Normal file
View File

@ -0,0 +1,519 @@
// SPDX-License-Identifier: GPL-2.0
/* Copyright (c) 2024, Oracle and/or its affiliates. */
#ifndef _GNU_SOURCE
#define _GNU_SOURCE
#endif
#ifdef __KERNEL__
#include <linux/bpf.h>
#include <linux/bsearch.h>
#include <linux/btf.h>
#include <linux/sort.h>
#include <linux/string.h>
#include <linux/bpf_verifier.h>
#define btf_type_by_id (struct btf_type *)btf_type_by_id
#define btf__type_cnt btf_nr_types
#define btf__base_btf btf_base_btf
#define btf__name_by_offset btf_name_by_offset
#define btf__str_by_offset btf_str_by_offset
#define btf_kflag btf_type_kflag
#define calloc(nmemb, sz) kvcalloc(nmemb, sz, GFP_KERNEL | __GFP_NOWARN)
#define free(ptr) kvfree(ptr)
#define qsort(base, num, sz, cmp) sort(base, num, sz, cmp, NULL)
#else
#include "btf.h"
#include "bpf.h"
#include "libbpf.h"
#include "libbpf_internal.h"
#endif /* __KERNEL__ */
struct btf;
struct btf_relocate {
struct btf *btf;
const struct btf *base_btf;
const struct btf *dist_base_btf;
unsigned int nr_base_types;
unsigned int nr_split_types;
unsigned int nr_dist_base_types;
int dist_str_len;
int base_str_len;
__u32 *id_map;
__u32 *str_map;
};
/* Set temporarily in relocation id_map if distilled base struct/union is
* embedded in a split BTF struct/union; in such a case, size information must
* match between distilled base BTF and base BTF representation of type.
*/
#define BTF_IS_EMBEDDED ((__u32)-1)
/* <name, size, id> triple used in sorting/searching distilled base BTF. */
struct btf_name_info {
const char *name;
/* set when search requires a size match */
bool needs_size: 1;
unsigned int size: 31;
__u32 id;
};
static int btf_relocate_rewrite_type_id(struct btf_relocate *r, __u32 i)
{
struct btf_type *t = btf_type_by_id(r->btf, i);
struct btf_field_iter it;
__u32 *id;
int err;
err = btf_field_iter_init(&it, t, BTF_FIELD_ITER_IDS);
if (err)
return err;
while ((id = btf_field_iter_next(&it)))
*id = r->id_map[*id];
return 0;
}
/* Simple string comparison used for sorting within BTF, since all distilled
* types are named. If strings match, and size is non-zero for both elements
* fall back to using size for ordering.
*/
static int cmp_btf_name_size(const void *n1, const void *n2)
{
const struct btf_name_info *ni1 = n1;
const struct btf_name_info *ni2 = n2;
int name_diff = strcmp(ni1->name, ni2->name);
if (!name_diff && ni1->needs_size && ni2->needs_size)
return ni2->size - ni1->size;
return name_diff;
}
/* Binary search with a small twist; find leftmost element that matches
* so that we can then iterate through all exact matches. So for example
* searching { "a", "bb", "bb", "c" } we would always match on the
* leftmost "bb".
*/
static struct btf_name_info *search_btf_name_size(struct btf_name_info *key,
struct btf_name_info *vals,
int nelems)
{
struct btf_name_info *ret = NULL;
int high = nelems - 1;
int low = 0;
while (low <= high) {
int mid = (low + high)/2;
struct btf_name_info *val = &vals[mid];
int diff = cmp_btf_name_size(key, val);
if (diff == 0)
ret = val;
/* even if found, keep searching for leftmost match */
if (diff <= 0)
high = mid - 1;
else
low = mid + 1;
}
return ret;
}
/* If a member of a split BTF struct/union refers to a base BTF
* struct/union, mark that struct/union id temporarily in the id_map
* with BTF_IS_EMBEDDED. Members can be const/restrict/volatile/typedef
* reference types, but if a pointer is encountered, the type is no longer
* considered embedded.
*/
static int btf_mark_embedded_composite_type_ids(struct btf_relocate *r, __u32 i)
{
struct btf_type *t = btf_type_by_id(r->btf, i);
struct btf_field_iter it;
__u32 *id;
int err;
if (!btf_is_composite(t))
return 0;
err = btf_field_iter_init(&it, t, BTF_FIELD_ITER_IDS);
if (err)
return err;
while ((id = btf_field_iter_next(&it))) {
__u32 next_id = *id;
while (next_id) {
t = btf_type_by_id(r->btf, next_id);
switch (btf_kind(t)) {
case BTF_KIND_CONST:
case BTF_KIND_RESTRICT:
case BTF_KIND_VOLATILE:
case BTF_KIND_TYPEDEF:
case BTF_KIND_TYPE_TAG:
next_id = t->type;
break;
case BTF_KIND_ARRAY: {
struct btf_array *a = btf_array(t);
next_id = a->type;
break;
}
case BTF_KIND_STRUCT:
case BTF_KIND_UNION:
if (next_id < r->nr_dist_base_types)
r->id_map[next_id] = BTF_IS_EMBEDDED;
next_id = 0;
break;
default:
next_id = 0;
break;
}
}
}
return 0;
}
/* Build a map from distilled base BTF ids to base BTF ids. To do so, iterate
* through base BTF looking up distilled type (using binary search) equivalents.
*/
static int btf_relocate_map_distilled_base(struct btf_relocate *r)
{
struct btf_name_info *info, *info_end;
struct btf_type *base_t, *dist_t;
__u8 *base_name_cnt = NULL;
int err = 0;
__u32 id;
/* generate a sort index array of name/type ids sorted by name for
* distilled base BTF to speed name-based lookups.
*/
info = calloc(r->nr_dist_base_types, sizeof(*info));
if (!info) {
err = -ENOMEM;
goto done;
}
info_end = info + r->nr_dist_base_types;
for (id = 0; id < r->nr_dist_base_types; id++) {
dist_t = btf_type_by_id(r->dist_base_btf, id);
info[id].name = btf__name_by_offset(r->dist_base_btf, dist_t->name_off);
info[id].id = id;
info[id].size = dist_t->size;
info[id].needs_size = true;
}
qsort(info, r->nr_dist_base_types, sizeof(*info), cmp_btf_name_size);
/* Mark distilled base struct/union members of split BTF structs/unions
* in id_map with BTF_IS_EMBEDDED; this signals that these types
* need to match both name and size, otherwise embedding the base
* struct/union in the split type is invalid.
*/
for (id = r->nr_dist_base_types; id < r->nr_split_types; id++) {
err = btf_mark_embedded_composite_type_ids(r, id);
if (err)
goto done;
}
/* Collect name counts for composite types in base BTF. If multiple
* instances of a struct/union of the same name exist, we need to use
* size to determine which to map to since name alone is ambiguous.
*/
base_name_cnt = calloc(r->base_str_len, sizeof(*base_name_cnt));
if (!base_name_cnt) {
err = -ENOMEM;
goto done;
}
for (id = 1; id < r->nr_base_types; id++) {
base_t = btf_type_by_id(r->base_btf, id);
if (!btf_is_composite(base_t) || !base_t->name_off)
continue;
if (base_name_cnt[base_t->name_off] < 255)
base_name_cnt[base_t->name_off]++;
}
/* Now search base BTF for matching distilled base BTF types. */
for (id = 1; id < r->nr_base_types; id++) {
struct btf_name_info *dist_info, base_info = {};
int dist_kind, base_kind;
base_t = btf_type_by_id(r->base_btf, id);
/* distilled base consists of named types only. */
if (!base_t->name_off)
continue;
base_kind = btf_kind(base_t);
base_info.id = id;
base_info.name = btf__name_by_offset(r->base_btf, base_t->name_off);
switch (base_kind) {
case BTF_KIND_INT:
case BTF_KIND_FLOAT:
case BTF_KIND_ENUM:
case BTF_KIND_ENUM64:
/* These types should match both name and size */
base_info.needs_size = true;
base_info.size = base_t->size;
break;
case BTF_KIND_FWD:
/* No size considerations for fwds. */
break;
case BTF_KIND_STRUCT:
case BTF_KIND_UNION:
/* Size only needs to be used for struct/union if there
* are multiple types in base BTF with the same name.
* If there are multiple _distilled_ types with the same
* name (a very unlikely scenario), that doesn't matter
* unless corresponding _base_ types to match them are
* missing.
*/
base_info.needs_size = base_name_cnt[base_t->name_off] > 1;
base_info.size = base_t->size;
break;
default:
continue;
}
/* iterate over all matching distilled base types */
for (dist_info = search_btf_name_size(&base_info, info, r->nr_dist_base_types);
dist_info != NULL && dist_info < info_end &&
cmp_btf_name_size(&base_info, dist_info) == 0;
dist_info++) {
if (!dist_info->id || dist_info->id >= r->nr_dist_base_types) {
pr_warn("base BTF id [%d] maps to invalid distilled base BTF id [%d]\n",
id, dist_info->id);
err = -EINVAL;
goto done;
}
dist_t = btf_type_by_id(r->dist_base_btf, dist_info->id);
dist_kind = btf_kind(dist_t);
/* Validate that the found distilled type is compatible.
* Do not error out on mismatch as another match may
* occur for an identically-named type.
*/
switch (dist_kind) {
case BTF_KIND_FWD:
switch (base_kind) {
case BTF_KIND_FWD:
if (btf_kflag(dist_t) != btf_kflag(base_t))
continue;
break;
case BTF_KIND_STRUCT:
if (btf_kflag(base_t))
continue;
break;
case BTF_KIND_UNION:
if (!btf_kflag(base_t))
continue;
break;
default:
continue;
}
break;
case BTF_KIND_INT:
if (dist_kind != base_kind ||
btf_int_encoding(base_t) != btf_int_encoding(dist_t))
continue;
break;
case BTF_KIND_FLOAT:
if (dist_kind != base_kind)
continue;
break;
case BTF_KIND_ENUM:
/* ENUM and ENUM64 are encoded as sized ENUM in
* distilled base BTF.
*/
if (base_kind != dist_kind && base_kind != BTF_KIND_ENUM64)
continue;
break;
case BTF_KIND_STRUCT:
case BTF_KIND_UNION:
/* size verification is required for embedded
* struct/unions.
*/
if (r->id_map[dist_info->id] == BTF_IS_EMBEDDED &&
base_t->size != dist_t->size)
continue;
break;
default:
continue;
}
if (r->id_map[dist_info->id] &&
r->id_map[dist_info->id] != BTF_IS_EMBEDDED) {
/* we already have a match; this tells us that
* multiple base types of the same name
* have the same size, since for cases where
* multiple types have the same name we match
* on name and size. In this case, we have
* no way of determining which to relocate
* to in base BTF, so error out.
*/
pr_warn("distilled base BTF type '%s' [%u], size %u has multiple candidates of the same size (ids [%u, %u]) in base BTF\n",
base_info.name, dist_info->id,
base_t->size, id, r->id_map[dist_info->id]);
err = -EINVAL;
goto done;
}
/* map id and name */
r->id_map[dist_info->id] = id;
r->str_map[dist_t->name_off] = base_t->name_off;
}
}
/* ensure all distilled BTF ids now have a mapping... */
for (id = 1; id < r->nr_dist_base_types; id++) {
const char *name;
if (r->id_map[id] && r->id_map[id] != BTF_IS_EMBEDDED)
continue;
dist_t = btf_type_by_id(r->dist_base_btf, id);
name = btf__name_by_offset(r->dist_base_btf, dist_t->name_off);
pr_warn("distilled base BTF type '%s' [%d] is not mapped to base BTF id\n",
name, id);
err = -EINVAL;
break;
}
done:
free(base_name_cnt);
free(info);
return err;
}
/* distilled base should only have named int/float/enum/fwd/struct/union types. */
static int btf_relocate_validate_distilled_base(struct btf_relocate *r)
{
unsigned int i;
for (i = 1; i < r->nr_dist_base_types; i++) {
struct btf_type *t = btf_type_by_id(r->dist_base_btf, i);
int kind = btf_kind(t);
switch (kind) {
case BTF_KIND_INT:
case BTF_KIND_FLOAT:
case BTF_KIND_ENUM:
case BTF_KIND_STRUCT:
case BTF_KIND_UNION:
case BTF_KIND_FWD:
if (t->name_off)
break;
pr_warn("type [%d], kind [%d] is invalid for distilled base BTF; it is anonymous\n",
i, kind);
return -EINVAL;
default:
pr_warn("type [%d] in distilled based BTF has unexpected kind [%d]\n",
i, kind);
return -EINVAL;
}
}
return 0;
}
static int btf_relocate_rewrite_strs(struct btf_relocate *r, __u32 i)
{
struct btf_type *t = btf_type_by_id(r->btf, i);
struct btf_field_iter it;
__u32 *str_off;
int off, err;
err = btf_field_iter_init(&it, t, BTF_FIELD_ITER_STRS);
if (err)
return err;
while ((str_off = btf_field_iter_next(&it))) {
if (!*str_off)
continue;
if (*str_off >= r->dist_str_len) {
*str_off += r->base_str_len - r->dist_str_len;
} else {
off = r->str_map[*str_off];
if (!off) {
pr_warn("string '%s' [offset %u] is not mapped to base BTF",
btf__str_by_offset(r->btf, off), *str_off);
return -ENOENT;
}
*str_off = off;
}
}
return 0;
}
/* If successful, output of relocation is updated BTF with base BTF pointing
* at base_btf, and type ids, strings adjusted accordingly.
*/
int btf_relocate(struct btf *btf, const struct btf *base_btf, __u32 **id_map)
{
unsigned int nr_types = btf__type_cnt(btf);
const struct btf_header *dist_base_hdr;
const struct btf_header *base_hdr;
struct btf_relocate r = {};
int err = 0;
__u32 id, i;
r.dist_base_btf = btf__base_btf(btf);
if (!base_btf || r.dist_base_btf == base_btf)
return -EINVAL;
r.nr_dist_base_types = btf__type_cnt(r.dist_base_btf);
r.nr_base_types = btf__type_cnt(base_btf);
r.nr_split_types = nr_types - r.nr_dist_base_types;
r.btf = btf;
r.base_btf = base_btf;
r.id_map = calloc(nr_types, sizeof(*r.id_map));
r.str_map = calloc(btf_header(r.dist_base_btf)->str_len, sizeof(*r.str_map));
dist_base_hdr = btf_header(r.dist_base_btf);
base_hdr = btf_header(r.base_btf);
r.dist_str_len = dist_base_hdr->str_len;
r.base_str_len = base_hdr->str_len;
if (!r.id_map || !r.str_map) {
err = -ENOMEM;
goto err_out;
}
err = btf_relocate_validate_distilled_base(&r);
if (err)
goto err_out;
/* Split BTF ids need to be adjusted as base and distilled base
* have different numbers of types, changing the start id of split
* BTF.
*/
for (id = r.nr_dist_base_types; id < nr_types; id++)
r.id_map[id] = id + r.nr_base_types - r.nr_dist_base_types;
/* Build a map from distilled base ids to actual base BTF ids; it is used
* to update split BTF id references. Also build a str_map mapping from
* distilled base BTF names to base BTF names.
*/
err = btf_relocate_map_distilled_base(&r);
if (err)
goto err_out;
/* Next, rewrite type ids in split BTF, replacing split ids with updated
* ids based on number of types in base BTF, and base ids with
* relocated ids from base_btf.
*/
for (i = 0, id = r.nr_dist_base_types; i < r.nr_split_types; i++, id++) {
err = btf_relocate_rewrite_type_id(&r, id);
if (err)
goto err_out;
}
/* String offsets now need to be updated using the str_map. */
for (i = 0; i < r.nr_split_types; i++) {
err = btf_relocate_rewrite_strs(&r, i + r.nr_dist_base_types);
if (err)
goto err_out;
}
/* Finally reset base BTF to be base_btf */
btf_set_base_btf(btf, base_btf);
if (id_map) {
*id_map = r.id_map;
r.id_map = NULL;
}
err_out:
free(r.id_map);
free(r.str_map);
return err;
}

4
tools/lib/bpf/libbpf.c
View File

@ -10375,7 +10375,7 @@ __bpf_map__iter(const struct bpf_map *m, const struct bpf_object *obj, int i)
struct bpf_map *
bpf_object__next_map(const struct bpf_object *obj, const struct bpf_map *prev)
{
if (prev == NULL)
if (prev == NULL && obj != NULL)
return obj->maps;
return __bpf_map__iter(prev, obj, 1);
@ -10384,7 +10384,7 @@ bpf_object__next_map(const struct bpf_object *obj, const struct bpf_map *prev)
struct bpf_map *
bpf_object__prev_map(const struct bpf_object *obj, const struct bpf_map *next)
{
if (next == NULL) {
if (next == NULL && obj != NULL) {
if (!obj->nr_maps)
return NULL;
return obj->maps + obj->nr_maps - 1;

2
tools/lib/bpf/libbpf.map
View File

@ -419,6 +419,8 @@ LIBBPF_1.4.0 {
LIBBPF_1.5.0 {
global:
btf__distill_base;
btf__relocate;
bpf_map__autoattach;
bpf_map__set_autoattach;
bpf_program__attach_sockmap;

3
tools/lib/bpf/libbpf_internal.h
View File

@ -234,6 +234,9 @@ struct btf_type;
struct btf_type *btf_type_by_id(const struct btf *btf, __u32 type_id);
const char *btf_kind_str(const struct btf_type *t);
const struct btf_type *skip_mods_and_typedefs(const struct btf *btf, __u32 id, __u32 *res_id);
const struct btf_header *btf_header(const struct btf *btf);
void btf_set_base_btf(struct btf *btf, const struct btf *base_btf);
int btf_relocate(struct btf *btf, const struct btf *base_btf, __u32 **id_map);
static inline enum btf_func_linkage btf_func_linkage(const struct btf_type *t)
{

11
tools/lib/bpf/linker.c
View File

@ -2227,10 +2227,17 @@ static int linker_fixup_btf(struct src_obj *obj)
vi = btf_var_secinfos(t);
for (j = 0, m = btf_vlen(t); j < m; j++, vi++) {
const struct btf_type *vt = btf__type_by_id(obj->btf, vi->type);
const char *var_name = btf__str_by_offset(obj->btf, vt->name_off);
int var_linkage = btf_var(vt)->linkage;
const char *var_name;
int var_linkage;
Elf64_Sym *sym;
/* could be a variable or function */
if (!btf_is_var(vt))
continue;
var_name = btf__str_by_offset(obj->btf, vt->name_off);
var_linkage = btf_var(vt)->linkage;
/* no need to patch up static or extern vars */
if (var_linkage != BTF_VAR_GLOBAL_ALLOCATED)
continue;

2
tools/testing/selftests/bpf/DENYLIST.aarch64
View File

@ -1,11 +1,11 @@
bpf_cookie/multi_kprobe_attach_api # kprobe_multi_link_api_subtest:FAIL:fentry_raw_skel_load unexpected error: -3
bpf_cookie/multi_kprobe_link_api # kprobe_multi_link_api_subtest:FAIL:fentry_raw_skel_load unexpected error: -3
fexit_sleep # The test never returns. The remaining tests cannot start.
kprobe_multi_bench_attach # needs CONFIG_FPROBE
kprobe_multi_test # needs CONFIG_FPROBE
module_attach # prog 'kprobe_multi': failed to auto-attach: -95
fentry_test/fentry_many_args # fentry_many_args:FAIL:fentry_many_args_attach unexpected error: -524
fexit_test/fexit_many_args # fexit_many_args:FAIL:fexit_many_args_attach unexpected error: -524
tracing_struct/struct_many_args # struct_many_args:FAIL:tracing_struct_many_args__attach unexpected error: -524
fill_link_info/kprobe_multi_link_info # bpf_program__attach_kprobe_multi_opts unexpected error: -95
fill_link_info/kretprobe_multi_link_info # bpf_program__attach_kprobe_multi_opts unexpected error: -95
fill_link_info/kprobe_multi_invalid_ubuff # bpf_program__attach_kprobe_multi_opts unexpected error: -95

4
tools/testing/selftests/bpf/DENYLIST.s390x
View File

@ -1,9 +1,5 @@
# TEMPORARY
# Alphabetical order
exceptions # JIT does not support calling kfunc bpf_throw (exceptions)
get_stack_raw_tp # user_stack corrupted user stack (no backchain userspace)
stacktrace_build_id # compare_map_keys stackid_hmap vs. stackmap err -2 errno 2 (?)
verifier_iterating_callbacks
verifier_arena # JIT does not support arena
arena_htab # JIT does not support arena
arena_atomics

2
tools/testing/selftests/bpf/bpf_arena_common.h
View File

@ -34,10 +34,12 @@
#if defined(__BPF_FEATURE_ADDR_SPACE_CAST) && !defined(BPF_ARENA_FORCE_ASM)
#define __arena __attribute__((address_space(1)))
#define __arena_global __attribute__((address_space(1)))
#define cast_kern(ptr) /* nop for bpf prog. emitted by LLVM */
#define cast_user(ptr) /* nop for bpf prog. emitted by LLVM */
#else
#define __arena
#define __arena_global SEC(".addr_space.1")
#define cast_kern(ptr) bpf_addr_space_cast(ptr, 0, 1)
#define cast_user(ptr) bpf_addr_space_cast(ptr, 1, 0)
#endif

32
tools/testing/selftests/bpf/bpf_experimental.h
View File

@ -163,7 +163,7 @@ struct bpf_iter_task_vma;
extern int bpf_iter_task_vma_new(struct bpf_iter_task_vma *it,
struct task_struct *task,
unsigned long addr) __ksym;
__u64 addr) __ksym;
extern struct vm_area_struct *bpf_iter_task_vma_next(struct bpf_iter_task_vma *it) __ksym;
extern void bpf_iter_task_vma_destroy(struct bpf_iter_task_vma *it) __ksym;
@ -351,6 +351,7 @@ l_true: \
l_continue:; \
})
#else
#if __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__
#define can_loop \
({ __label__ l_break, l_continue; \
bool ret = true; \
@ -376,6 +377,33 @@ l_true: \
l_break: break; \
l_continue:; \
})
#else
#define can_loop \
({ __label__ l_break, l_continue; \
bool ret = true; \
asm volatile goto("1:.byte 0xe5; \
.byte 0; \
.long (((%l[l_break] - 1b - 8) / 8) & 0xffff) << 16; \
.short 0" \
:::: l_break); \
goto l_continue; \
l_break: ret = false; \
l_continue:; \
ret; \
})
#define cond_break \
({ __label__ l_break, l_continue; \
asm volatile goto("1:.byte 0xe5; \
.byte 0; \
.long (((%l[l_break] - 1b - 8) / 8) & 0xffff) << 16; \
.short 0" \
:::: l_break); \
goto l_continue; \
l_break: break; \
l_continue:; \
})
#endif
#endif
#ifndef bpf_nop_mov
@ -524,7 +552,7 @@ extern void bpf_iter_css_destroy(struct bpf_iter_css *it) __weak __ksym;
extern int bpf_wq_init(struct bpf_wq *wq, void *p__map, unsigned int flags) __weak __ksym;
extern int bpf_wq_start(struct bpf_wq *wq, unsigned int flags) __weak __ksym;
extern int bpf_wq_set_callback_impl(struct bpf_wq *wq,
int (callback_fn)(void *map, int *key, struct bpf_wq *wq),
int (callback_fn)(void *map, int *key, void *value),
unsigned int flags__k, void *aux__ign) __ksym;
#define bpf_wq_set_callback(timer, cb, flags) \
bpf_wq_set_callback_impl(timer, cb, flags, NULL)

2
tools/testing/selftests/bpf/bpf_kfuncs.h
View File

@ -77,5 +77,5 @@ extern int bpf_verify_pkcs7_signature(struct bpf_dynptr *data_ptr,
struct bpf_key *trusted_keyring) __ksym;
extern bool bpf_session_is_return(void) __ksym __weak;
extern long *bpf_session_cookie(void) __ksym __weak;
extern __u64 *bpf_session_cookie(void) __ksym __weak;
#endif

71
tools/testing/selftests/bpf/bpf_testmod/bpf_testmod.c
View File

@ -53,6 +53,13 @@ struct bpf_testmod_struct_arg_4 {
int b;
};
struct bpf_testmod_struct_arg_5 {
char a;
short b;
int c;
long d;
};
__bpf_hook_start();
noinline int
@ -110,6 +117,15 @@ bpf_testmod_test_struct_arg_8(u64 a, void *b, short c, int d, void *e,
return bpf_testmod_test_struct_arg_result;
}
noinline int
bpf_testmod_test_struct_arg_9(u64 a, void *b, short c, int d, void *e, char f,
short g, struct bpf_testmod_struct_arg_5 h, long i)
{
bpf_testmod_test_struct_arg_result = a + (long)b + c + d + (long)e +
f + g + h.a + h.b + h.c + h.d + i;
return bpf_testmod_test_struct_arg_result;
}
noinline int
bpf_testmod_test_arg_ptr_to_struct(struct bpf_testmod_struct_arg_1 *a) {
bpf_testmod_test_struct_arg_result = a->a;
@ -154,6 +170,42 @@ __bpf_kfunc void bpf_kfunc_common_test(void)
{
}
__bpf_kfunc void bpf_kfunc_dynptr_test(struct bpf_dynptr *ptr,
struct bpf_dynptr *ptr__nullable)
{
}
__bpf_kfunc struct bpf_testmod_ctx *
bpf_testmod_ctx_create(int *err)
{
struct bpf_testmod_ctx *ctx;
ctx = kzalloc(sizeof(*ctx), GFP_ATOMIC);
if (!ctx) {
*err = -ENOMEM;
return NULL;
}
refcount_set(&ctx->usage, 1);
return ctx;
}
static void testmod_free_cb(struct rcu_head *head)
{
struct bpf_testmod_ctx *ctx;
ctx = container_of(head, struct bpf_testmod_ctx, rcu);
kfree(ctx);
}
__bpf_kfunc void bpf_testmod_ctx_release(struct bpf_testmod_ctx *ctx)
{
if (!ctx)
return;
if (refcount_dec_and_test(&ctx->usage))
call_rcu(&ctx->rcu, testmod_free_cb);
}
struct bpf_testmod_btf_type_tag_1 {
int a;
};
@ -269,6 +321,7 @@ bpf_testmod_test_read(struct file *file, struct kobject *kobj,
struct bpf_testmod_struct_arg_2 struct_arg2 = {2, 3};
struct bpf_testmod_struct_arg_3 *struct_arg3;
struct bpf_testmod_struct_arg_4 struct_arg4 = {21, 22};
struct bpf_testmod_struct_arg_5 struct_arg5 = {23, 24, 25, 26};
int i = 1;
while (bpf_testmod_return_ptr(i))
@ -283,6 +336,8 @@ bpf_testmod_test_read(struct file *file, struct kobject *kobj,
(void *)20, struct_arg4);
(void)bpf_testmod_test_struct_arg_8(16, (void *)17, 18, 19,
(void *)20, struct_arg4, 23);
(void)bpf_testmod_test_struct_arg_9(16, (void *)17, 18, 19, (void *)20,
21, 22, struct_arg5, 27);
(void)bpf_testmod_test_arg_ptr_to_struct(&struct_arg1_2);
@ -363,8 +418,15 @@ BTF_ID_FLAGS(func, bpf_iter_testmod_seq_new, KF_ITER_NEW)
BTF_ID_FLAGS(func, bpf_iter_testmod_seq_next, KF_ITER_NEXT | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_iter_testmod_seq_destroy, KF_ITER_DESTROY)
BTF_ID_FLAGS(func, bpf_kfunc_common_test)
BTF_ID_FLAGS(func, bpf_kfunc_dynptr_test)
BTF_ID_FLAGS(func, bpf_testmod_ctx_create, KF_ACQUIRE | KF_RET_NULL)
BTF_ID_FLAGS(func, bpf_testmod_ctx_release, KF_RELEASE)
BTF_KFUNCS_END(bpf_testmod_common_kfunc_ids)
BTF_ID_LIST(bpf_testmod_dtor_ids)
BTF_ID(struct, bpf_testmod_ctx)
BTF_ID(func, bpf_testmod_ctx_release)
static const struct btf_kfunc_id_set bpf_testmod_common_kfunc_set = {
.owner = THIS_MODULE,
.set = &bpf_testmod_common_kfunc_ids,
@ -898,6 +960,12 @@ extern int bpf_fentry_test1(int a);
static int bpf_testmod_init(void)
{
const struct btf_id_dtor_kfunc bpf_testmod_dtors[] = {
{
.btf_id = bpf_testmod_dtor_ids[0],
.kfunc_btf_id = bpf_testmod_dtor_ids[1]
},
};
int ret;
ret = register_btf_kfunc_id_set(BPF_PROG_TYPE_UNSPEC, &bpf_testmod_common_kfunc_set);
@ -906,6 +974,9 @@ static int bpf_testmod_init(void)
ret = ret ?: register_btf_kfunc_id_set(BPF_PROG_TYPE_SYSCALL, &bpf_testmod_kfunc_set);
ret = ret ?: register_bpf_struct_ops(&bpf_bpf_testmod_ops, bpf_testmod_ops);
ret = ret ?: register_bpf_struct_ops(&bpf_testmod_ops2, bpf_testmod_ops2);
ret = ret ?: register_btf_id_dtor_kfuncs(bpf_testmod_dtors,
ARRAY_SIZE(bpf_testmod_dtors),
THIS_MODULE);
if (ret < 0)
return ret;
if (bpf_fentry_test1(0) < 0)

10
tools/testing/selftests/bpf/bpf_testmod/bpf_testmod_kfunc.h
View File

@ -80,6 +80,11 @@ struct sendmsg_args {
int msglen;
};
struct bpf_testmod_ctx {
struct callback_head rcu;
refcount_t usage;
};
struct prog_test_ref_kfunc *
bpf_kfunc_call_test_acquire(unsigned long *scalar_ptr) __ksym;
void bpf_kfunc_call_test_release(struct prog_test_ref_kfunc *p) __ksym;
@ -134,4 +139,9 @@ int bpf_kfunc_call_sock_sendmsg(struct sendmsg_args *args) __ksym;
int bpf_kfunc_call_kernel_getsockname(struct addr_args *args) __ksym;
int bpf_kfunc_call_kernel_getpeername(struct addr_args *args) __ksym;
void bpf_kfunc_dynptr_test(struct bpf_dynptr *ptr, struct bpf_dynptr *ptr__nullable) __ksym;
struct bpf_testmod_ctx *bpf_testmod_ctx_create(int *err) __ksym;
void bpf_testmod_ctx_release(struct bpf_testmod_ctx *ctx) __ksym;
#endif /* _BPF_TESTMOD_KFUNC_H */

13
tools/testing/selftests/bpf/config
View File

@ -83,6 +83,19 @@ CONFIG_NF_CONNTRACK_MARK=y
CONFIG_NF_CONNTRACK_ZONES=y
CONFIG_NF_DEFRAG_IPV4=y
CONFIG_NF_DEFRAG_IPV6=y
CONFIG_NF_TABLES=y
CONFIG_NF_TABLES_INET=y
CONFIG_NF_TABLES_NETDEV=y
CONFIG_NF_TABLES_IPV4=y
CONFIG_NF_TABLES_IPV6=y
CONFIG_NETFILTER_INGRESS=y
CONFIG_NF_FLOW_TABLE=y
CONFIG_NF_FLOW_TABLE_INET=y
CONFIG_NETFILTER_NETLINK=y
CONFIG_NFT_FLOW_OFFLOAD=y
CONFIG_IP_NF_IPTABLES=y
CONFIG_IP6_NF_IPTABLES=y
CONFIG_IP6_NF_FILTER=y
CONFIG_NF_NAT=y
CONFIG_RC_CORE=y
CONFIG_SECURITY=y

100
tools/testing/selftests/bpf/network_helpers.c
View File

@ -249,6 +249,34 @@ error_close:
return -1;
}
int client_socket(int family, int type,
const struct network_helper_opts *opts)
{
int fd;
if (!opts)
opts = &default_opts;
fd = socket(family, type, opts->proto);
if (fd < 0) {
log_err("Failed to create client socket");
return -1;
}
if (settimeo(fd, opts->timeout_ms))
goto error_close;
if (opts->post_socket_cb &&
opts->post_socket_cb(fd, opts->cb_opts))
goto error_close;
return fd;
error_close:
save_errno_close(fd);
return -1;
}
static int connect_fd_to_addr(int fd,
const struct sockaddr_storage *addr,
socklen_t addrlen, const bool must_fail)
@ -284,15 +312,12 @@ int connect_to_addr(int type, const struct sockaddr_storage *addr, socklen_t add
if (!opts)
opts = &default_opts;
fd = socket(addr->ss_family, type, opts->proto);
fd = client_socket(addr->ss_family, type, opts);
if (fd < 0) {
log_err("Failed to create client socket");
return -1;
}
if (settimeo(fd, opts->timeout_ms))
goto error_close;
if (connect_fd_to_addr(fd, addr, addrlen, opts->must_fail))
goto error_close;
@ -303,65 +328,21 @@ error_close:
return -1;
}
int connect_to_fd_opts(int server_fd, const struct network_helper_opts *opts)
int connect_to_fd_opts(int server_fd, int type, const struct network_helper_opts *opts)
{
struct sockaddr_storage addr;
struct sockaddr_in *addr_in;
socklen_t addrlen, optlen;
int fd, type, protocol;
socklen_t addrlen;
if (!opts)
opts = &default_opts;
optlen = sizeof(type);
if (opts->type) {
type = opts->type;
} else {
if (getsockopt(server_fd, SOL_SOCKET, SO_TYPE, &type, &optlen)) {
log_err("getsockopt(SOL_TYPE)");
return -1;
}
}
if (opts->proto) {
protocol = opts->proto;
} else {
if (getsockopt(server_fd, SOL_SOCKET, SO_PROTOCOL, &protocol, &optlen)) {
log_err("getsockopt(SOL_PROTOCOL)");
return -1;
}
}
addrlen = sizeof(addr);
if (getsockname(server_fd, (struct sockaddr *)&addr, &addrlen)) {
log_err("Failed to get server addr");
return -1;
}
addr_in = (struct sockaddr_in *)&addr;
fd = socket(addr_in->sin_family, type, protocol);
if (fd < 0) {
log_err("Failed to create client socket");
return -1;
}
if (settimeo(fd, opts->timeout_ms))
goto error_close;
if (opts->post_socket_cb &&
opts->post_socket_cb(fd, opts->cb_opts))
goto error_close;
if (!opts->noconnect)
if (connect_fd_to_addr(fd, &addr, addrlen, opts->must_fail))
goto error_close;
return fd;
error_close:
save_errno_close(fd);
return -1;
return connect_to_addr(type, &addr, addrlen, opts);
}
int connect_to_fd(int server_fd, int timeout_ms)
@ -369,8 +350,23 @@ int connect_to_fd(int server_fd, int timeout_ms)
struct network_helper_opts opts = {
.timeout_ms = timeout_ms,
};
int type, protocol;
socklen_t optlen;
return connect_to_fd_opts(server_fd, &opts);
optlen = sizeof(type);
if (getsockopt(server_fd, SOL_SOCKET, SO_TYPE, &type, &optlen)) {
log_err("getsockopt(SOL_TYPE)");
return -1;
}
optlen = sizeof(protocol);
if (getsockopt(server_fd, SOL_SOCKET, SO_PROTOCOL, &protocol, &optlen)) {
log_err("getsockopt(SOL_PROTOCOL)");
return -1;
}
opts.proto = protocol;
return connect_to_fd_opts(server_fd, type, &opts);
}
int connect_fd_to_fd(int client_fd, int server_fd, int timeout_ms)

6
tools/testing/selftests/bpf/network_helpers.h
View File

@ -24,8 +24,6 @@ typedef __u16 __sum16;
struct network_helper_opts {
int timeout_ms;
bool must_fail;
bool noconnect;
int type;
int proto;
int (*post_socket_cb)(int fd, void *opts);
void *cb_opts;
@ -58,10 +56,12 @@ int *start_reuseport_server(int family, int type, const char *addr_str,
int start_server_addr(int type, const struct sockaddr_storage *addr, socklen_t len,
const struct network_helper_opts *opts);
void free_fds(int *fds, unsigned int nr_close_fds);
int client_socket(int family, int type,
const struct network_helper_opts *opts);
int connect_to_addr(int type, const struct sockaddr_storage *addr, socklen_t len,
const struct network_helper_opts *opts);
int connect_to_fd(int server_fd, int timeout_ms);
int connect_to_fd_opts(int server_fd, const struct network_helper_opts *opts);
int connect_to_fd_opts(int server_fd, int type, const struct network_helper_opts *opts);
int connect_fd_to_fd(int client_fd, int server_fd, int timeout_ms);
int fastopen_connect(int server_fd, const char *data, unsigned int data_len,
int timeout_ms);

18
tools/testing/selftests/bpf/prog_tests/arena_atomics.c
View File

@ -146,6 +146,22 @@ static void test_xchg(struct arena_atomics *skel)
ASSERT_EQ(skel->arena->xchg32_result, 1, "xchg32_result");
}
static void test_uaf(struct arena_atomics *skel)
{
LIBBPF_OPTS(bpf_test_run_opts, topts);
int err, prog_fd;
/* No need to attach it, just run it directly */
prog_fd = bpf_program__fd(skel->progs.uaf);
err = bpf_prog_test_run_opts(prog_fd, &topts);
if (!ASSERT_OK(err, "test_run_opts err"))
return;
if (!ASSERT_OK(topts.retval, "test_run_opts retval"))
return;
ASSERT_EQ(skel->arena->uaf_recovery_fails, 0, "uaf_recovery_fails");
}
void test_arena_atomics(void)
{
struct arena_atomics *skel;
@ -180,6 +196,8 @@ void test_arena_atomics(void)
test_cmpxchg(skel);
if (test__start_subtest("xchg"))
test_xchg(skel);
if (test__start_subtest("uaf"))
test_uaf(skel);
cleanup:
arena_atomics__destroy(skel);

37
tools/testing/selftests/bpf/prog_tests/bpf_tcp_ca.c
View File

@ -49,7 +49,7 @@ static bool start_test(char *addr_str,
goto err;
/* connect to server */
*cli_fd = connect_to_fd_opts(*srv_fd, cli_opts);
*cli_fd = connect_to_fd_opts(*srv_fd, SOCK_STREAM, cli_opts);
if (!ASSERT_NEQ(*cli_fd, -1, "connect_to_fd_opts"))
goto err;
@ -185,6 +185,39 @@ done:
close(fd);
}
static void test_dctcp_autoattach_map(void)
{
struct cb_opts cb_opts = {
.cc = "bpf_dctcp",
};
struct network_helper_opts opts = {
.post_socket_cb = cc_cb,
.cb_opts = &cb_opts,
};
struct bpf_dctcp *dctcp_skel;
struct bpf_link *link;
dctcp_skel = bpf_dctcp__open_and_load();
if (!ASSERT_OK_PTR(dctcp_skel, "bpf_dctcp__open_and_load"))
return;
bpf_map__set_autoattach(dctcp_skel->maps.dctcp, true);
bpf_map__set_autoattach(dctcp_skel->maps.dctcp_nouse, false);
if (!ASSERT_OK(bpf_dctcp__attach(dctcp_skel), "bpf_dctcp__attach"))
goto destroy;
/* struct_ops is auto-attached */
link = dctcp_skel->links.dctcp;
if (!ASSERT_OK_PTR(link, "link"))
goto destroy;
do_test(&opts);
destroy:
bpf_dctcp__destroy(dctcp_skel);
}
static char *err_str;
static bool found;
@ -598,4 +631,6 @@ void test_bpf_tcp_ca(void)
test_tcp_ca_kfunc();
if (test__start_subtest("cc_cubic"))
test_cc_cubic();
if (test__start_subtest("dctcp_autoattach_map"))
test_dctcp_autoattach_map();
}

552
tools/testing/selftests/bpf/prog_tests/btf_distill.c Normal file
View File

@ -0,0 +1,552 @@
// SPDX-License-Identifier: GPL-2.0
/* Copyright (c) 2024, Oracle and/or its affiliates. */
#include <test_progs.h>
#include <bpf/btf.h>
#include "btf_helpers.h"
/* Fabricate base, split BTF with references to base types needed; then create
* split BTF with distilled base BTF and ensure expectations are met:
* - only referenced base types from split BTF are present
* - struct/union/enum are represented as empty unless anonymous, when they
* are represented in full in split BTF
*/
static void test_distilled_base(void)
{
struct btf *btf1 = NULL, *btf2 = NULL, *btf3 = NULL, *btf4 = NULL;
btf1 = btf__new_empty();
if (!ASSERT_OK_PTR(btf1, "empty_main_btf"))
return;
btf__add_int(btf1, "int", 4, BTF_INT_SIGNED); /* [1] int */
btf__add_ptr(btf1, 1); /* [2] ptr to int */
btf__add_struct(btf1, "s1", 8); /* [3] struct s1 { */
btf__add_field(btf1, "f1", 2, 0, 0); /* int *f1; */
/* } */
btf__add_struct(btf1, "", 12); /* [4] struct { */
btf__add_field(btf1, "f1", 1, 0, 0); /* int f1; */
btf__add_field(btf1, "f2", 3, 32, 0); /* struct s1 f2; */
/* } */
btf__add_int(btf1, "unsigned int", 4, 0); /* [5] unsigned int */
btf__add_union(btf1, "u1", 12); /* [6] union u1 { */
btf__add_field(btf1, "f1", 1, 0, 0); /* int f1; */
btf__add_field(btf1, "f2", 2, 0, 0); /* int *f2; */
/* } */
btf__add_union(btf1, "", 4); /* [7] union { */
btf__add_field(btf1, "f1", 1, 0, 0); /* int f1; */
/* } */
btf__add_enum(btf1, "e1", 4); /* [8] enum e1 { */
btf__add_enum_value(btf1, "v1", 1); /* v1 = 1; */
/* } */
btf__add_enum(btf1, "", 4); /* [9] enum { */
btf__add_enum_value(btf1, "av1", 2); /* av1 = 2; */
/* } */
btf__add_enum64(btf1, "e641", 8, true); /* [10] enum64 { */
btf__add_enum64_value(btf1, "v1", 1024); /* v1 = 1024; */
/* } */
btf__add_enum64(btf1, "", 8, true); /* [11] enum64 { */
btf__add_enum64_value(btf1, "v1", 1025); /* v1 = 1025; */
/* } */
btf__add_struct(btf1, "unneeded", 4); /* [12] struct unneeded { */
btf__add_field(btf1, "f1", 1, 0, 0); /* int f1; */
/* } */
btf__add_struct(btf1, "embedded", 4); /* [13] struct embedded { */
btf__add_field(btf1, "f1", 1, 0, 0); /* int f1; */
/* } */
btf__add_func_proto(btf1, 1); /* [14] int (*)(int *p1); */
btf__add_func_param(btf1, "p1", 1);
btf__add_array(btf1, 1, 1, 3); /* [15] int [3]; */
btf__add_struct(btf1, "from_proto", 4); /* [16] struct from_proto { */
btf__add_field(btf1, "f1", 1, 0, 0); /* int f1; */
/* } */
btf__add_union(btf1, "u1", 4); /* [17] union u1 { */
btf__add_field(btf1, "f1", 1, 0, 0); /* int f1; */
/* } */
VALIDATE_RAW_BTF(
btf1,
"[1] INT 'int' size=4 bits_offset=0 nr_bits=32 encoding=SIGNED",
"[2] PTR '(anon)' type_id=1",
"[3] STRUCT 's1' size=8 vlen=1\n"
"\t'f1' type_id=2 bits_offset=0",
"[4] STRUCT '(anon)' size=12 vlen=2\n"
"\t'f1' type_id=1 bits_offset=0\n"
"\t'f2' type_id=3 bits_offset=32",
"[5] INT 'unsigned int' size=4 bits_offset=0 nr_bits=32 encoding=(none)",
"[6] UNION 'u1' size=12 vlen=2\n"
"\t'f1' type_id=1 bits_offset=0\n"
"\t'f2' type_id=2 bits_offset=0",
"[7] UNION '(anon)' size=4 vlen=1\n"
"\t'f1' type_id=1 bits_offset=0",
"[8] ENUM 'e1' encoding=UNSIGNED size=4 vlen=1\n"
"\t'v1' val=1",
"[9] ENUM '(anon)' encoding=UNSIGNED size=4 vlen=1\n"
"\t'av1' val=2",
"[10] ENUM64 'e641' encoding=SIGNED size=8 vlen=1\n"
"\t'v1' val=1024",
"[11] ENUM64 '(anon)' encoding=SIGNED size=8 vlen=1\n"
"\t'v1' val=1025",
"[12] STRUCT 'unneeded' size=4 vlen=1\n"
"\t'f1' type_id=1 bits_offset=0",
"[13] STRUCT 'embedded' size=4 vlen=1\n"
"\t'f1' type_id=1 bits_offset=0",
"[14] FUNC_PROTO '(anon)' ret_type_id=1 vlen=1\n"
"\t'p1' type_id=1",
"[15] ARRAY '(anon)' type_id=1 index_type_id=1 nr_elems=3",
"[16] STRUCT 'from_proto' size=4 vlen=1\n"
"\t'f1' type_id=1 bits_offset=0",
"[17] UNION 'u1' size=4 vlen=1\n"
"\t'f1' type_id=1 bits_offset=0");
btf2 = btf__new_empty_split(btf1);
if (!ASSERT_OK_PTR(btf2, "empty_split_btf"))
goto cleanup;
btf__add_ptr(btf2, 3); /* [18] ptr to struct s1 */
/* add ptr to struct anon */
btf__add_ptr(btf2, 4); /* [19] ptr to struct (anon) */
btf__add_const(btf2, 6); /* [20] const union u1 */
btf__add_restrict(btf2, 7); /* [21] restrict union (anon) */
btf__add_volatile(btf2, 8); /* [22] volatile enum e1 */
btf__add_typedef(btf2, "et", 9); /* [23] typedef enum (anon) */
btf__add_const(btf2, 10); /* [24] const enum64 e641 */
btf__add_ptr(btf2, 11); /* [25] restrict enum64 (anon) */
btf__add_struct(btf2, "with_embedded", 4); /* [26] struct with_embedded { */
btf__add_field(btf2, "f1", 13, 0, 0); /* struct embedded f1; */
/* } */
btf__add_func(btf2, "fn", BTF_FUNC_STATIC, 14); /* [27] int fn(int p1); */
btf__add_typedef(btf2, "arraytype", 15); /* [28] typedef int[3] foo; */
btf__add_func_proto(btf2, 1); /* [29] int (*)(struct from proto p1); */
btf__add_func_param(btf2, "p1", 16);
VALIDATE_RAW_BTF(
btf2,
"[1] INT 'int' size=4 bits_offset=0 nr_bits=32 encoding=SIGNED",
"[2] PTR '(anon)' type_id=1",
"[3] STRUCT 's1' size=8 vlen=1\n"
"\t'f1' type_id=2 bits_offset=0",
"[4] STRUCT '(anon)' size=12 vlen=2\n"
"\t'f1' type_id=1 bits_offset=0\n"
"\t'f2' type_id=3 bits_offset=32",
"[5] INT 'unsigned int' size=4 bits_offset=0 nr_bits=32 encoding=(none)",
"[6] UNION 'u1' size=12 vlen=2\n"
"\t'f1' type_id=1 bits_offset=0\n"
"\t'f2' type_id=2 bits_offset=0",
"[7] UNION '(anon)' size=4 vlen=1\n"
"\t'f1' type_id=1 bits_offset=0",
"[8] ENUM 'e1' encoding=UNSIGNED size=4 vlen=1\n"
"\t'v1' val=1",
"[9] ENUM '(anon)' encoding=UNSIGNED size=4 vlen=1\n"
"\t'av1' val=2",
"[10] ENUM64 'e641' encoding=SIGNED size=8 vlen=1\n"
"\t'v1' val=1024",
"[11] ENUM64 '(anon)' encoding=SIGNED size=8 vlen=1\n"
"\t'v1' val=1025",
"[12] STRUCT 'unneeded' size=4 vlen=1\n"
"\t'f1' type_id=1 bits_offset=0",
"[13] STRUCT 'embedded' size=4 vlen=1\n"
"\t'f1' type_id=1 bits_offset=0",
"[14] FUNC_PROTO '(anon)' ret_type_id=1 vlen=1\n"
"\t'p1' type_id=1",
"[15] ARRAY '(anon)' type_id=1 index_type_id=1 nr_elems=3",
"[16] STRUCT 'from_proto' size=4 vlen=1\n"
"\t'f1' type_id=1 bits_offset=0",
"[17] UNION 'u1' size=4 vlen=1\n"
"\t'f1' type_id=1 bits_offset=0",
"[18] PTR '(anon)' type_id=3",
"[19] PTR '(anon)' type_id=4",
"[20] CONST '(anon)' type_id=6",
"[21] RESTRICT '(anon)' type_id=7",
"[22] VOLATILE '(anon)' type_id=8",
"[23] TYPEDEF 'et' type_id=9",
"[24] CONST '(anon)' type_id=10",
"[25] PTR '(anon)' type_id=11",
"[26] STRUCT 'with_embedded' size=4 vlen=1\n"
"\t'f1' type_id=13 bits_offset=0",
"[27] FUNC 'fn' type_id=14 linkage=static",
"[28] TYPEDEF 'arraytype' type_id=15",
"[29] FUNC_PROTO '(anon)' ret_type_id=1 vlen=1\n"
"\t'p1' type_id=16");
if (!ASSERT_EQ(0, btf__distill_base(btf2, &btf3, &btf4),
"distilled_base") ||
!ASSERT_OK_PTR(btf3, "distilled_base") ||
!ASSERT_OK_PTR(btf4, "distilled_split") ||
!ASSERT_EQ(8, btf__type_cnt(btf3), "distilled_base_type_cnt"))
goto cleanup;
VALIDATE_RAW_BTF(
btf4,
"[1] INT 'int' size=4 bits_offset=0 nr_bits=32 encoding=SIGNED",
"[2] STRUCT 's1' size=8 vlen=0",
"[3] UNION 'u1' size=12 vlen=0",
"[4] ENUM 'e1' encoding=UNSIGNED size=4 vlen=0",
"[5] ENUM 'e641' encoding=UNSIGNED size=8 vlen=0",
"[6] STRUCT 'embedded' size=4 vlen=0",
"[7] STRUCT 'from_proto' size=4 vlen=0",
/* split BTF; these types should match split BTF above from 17-28, with
* updated type id references
*/
"[8] PTR '(anon)' type_id=2",
"[9] PTR '(anon)' type_id=20",
"[10] CONST '(anon)' type_id=3",
"[11] RESTRICT '(anon)' type_id=21",
"[12] VOLATILE '(anon)' type_id=4",
"[13] TYPEDEF 'et' type_id=22",
"[14] CONST '(anon)' type_id=5",
"[15] PTR '(anon)' type_id=23",
"[16] STRUCT 'with_embedded' size=4 vlen=1\n"
"\t'f1' type_id=6 bits_offset=0",
"[17] FUNC 'fn' type_id=24 linkage=static",
"[18] TYPEDEF 'arraytype' type_id=25",
"[19] FUNC_PROTO '(anon)' ret_type_id=1 vlen=1\n"
"\t'p1' type_id=7",
/* split BTF types added from original base BTF below */
"[20] STRUCT '(anon)' size=12 vlen=2\n"
"\t'f1' type_id=1 bits_offset=0\n"
"\t'f2' type_id=2 bits_offset=32",
"[21] UNION '(anon)' size=4 vlen=1\n"
"\t'f1' type_id=1 bits_offset=0",
"[22] ENUM '(anon)' encoding=UNSIGNED size=4 vlen=1\n"
"\t'av1' val=2",
"[23] ENUM64 '(anon)' encoding=SIGNED size=8 vlen=1\n"
"\t'v1' val=1025",
"[24] FUNC_PROTO '(anon)' ret_type_id=1 vlen=1\n"
"\t'p1' type_id=1",
"[25] ARRAY '(anon)' type_id=1 index_type_id=1 nr_elems=3");
if (!ASSERT_EQ(btf__relocate(btf4, btf1), 0, "relocate_split"))
goto cleanup;
VALIDATE_RAW_BTF(
btf4,
"[1] INT 'int' size=4 bits_offset=0 nr_bits=32 encoding=SIGNED",
"[2] PTR '(anon)' type_id=1",
"[3] STRUCT 's1' size=8 vlen=1\n"
"\t'f1' type_id=2 bits_offset=0",
"[4] STRUCT '(anon)' size=12 vlen=2\n"
"\t'f1' type_id=1 bits_offset=0\n"
"\t'f2' type_id=3 bits_offset=32",
"[5] INT 'unsigned int' size=4 bits_offset=0 nr_bits=32 encoding=(none)",
"[6] UNION 'u1' size=12 vlen=2\n"
"\t'f1' type_id=1 bits_offset=0\n"
"\t'f2' type_id=2 bits_offset=0",
"[7] UNION '(anon)' size=4 vlen=1\n"
"\t'f1' type_id=1 bits_offset=0",
"[8] ENUM 'e1' encoding=UNSIGNED size=4 vlen=1\n"
"\t'v1' val=1",
"[9] ENUM '(anon)' encoding=UNSIGNED size=4 vlen=1\n"
"\t'av1' val=2",
"[10] ENUM64 'e641' encoding=SIGNED size=8 vlen=1\n"
"\t'v1' val=1024",
"[11] ENUM64 '(anon)' encoding=SIGNED size=8 vlen=1\n"
"\t'v1' val=1025",
"[12] STRUCT 'unneeded' size=4 vlen=1\n"
"\t'f1' type_id=1 bits_offset=0",
"[13] STRUCT 'embedded' size=4 vlen=1\n"
"\t'f1' type_id=1 bits_offset=0",
"[14] FUNC_PROTO '(anon)' ret_type_id=1 vlen=1\n"
"\t'p1' type_id=1",
"[15] ARRAY '(anon)' type_id=1 index_type_id=1 nr_elems=3",
"[16] STRUCT 'from_proto' size=4 vlen=1\n"
"\t'f1' type_id=1 bits_offset=0",
"[17] UNION 'u1' size=4 vlen=1\n"
"\t'f1' type_id=1 bits_offset=0",
"[18] PTR '(anon)' type_id=3",
"[19] PTR '(anon)' type_id=30",
"[20] CONST '(anon)' type_id=6",
"[21] RESTRICT '(anon)' type_id=31",
"[22] VOLATILE '(anon)' type_id=8",
"[23] TYPEDEF 'et' type_id=32",
"[24] CONST '(anon)' type_id=10",
"[25] PTR '(anon)' type_id=33",
"[26] STRUCT 'with_embedded' size=4 vlen=1\n"
"\t'f1' type_id=13 bits_offset=0",
"[27] FUNC 'fn' type_id=34 linkage=static",
"[28] TYPEDEF 'arraytype' type_id=35",
"[29] FUNC_PROTO '(anon)' ret_type_id=1 vlen=1\n"
"\t'p1' type_id=16",
/* below here are (duplicate) anon base types added by distill
* process to split BTF.
*/
"[30] STRUCT '(anon)' size=12 vlen=2\n"
"\t'f1' type_id=1 bits_offset=0\n"
"\t'f2' type_id=3 bits_offset=32",
"[31] UNION '(anon)' size=4 vlen=1\n"
"\t'f1' type_id=1 bits_offset=0",
"[32] ENUM '(anon)' encoding=UNSIGNED size=4 vlen=1\n"
"\t'av1' val=2",
"[33] ENUM64 '(anon)' encoding=SIGNED size=8 vlen=1\n"
"\t'v1' val=1025",
"[34] FUNC_PROTO '(anon)' ret_type_id=1 vlen=1\n"
"\t'p1' type_id=1",
"[35] ARRAY '(anon)' type_id=1 index_type_id=1 nr_elems=3");
cleanup:
btf__free(btf4);
btf__free(btf3);
btf__free(btf2);
btf__free(btf1);
}
/* ensure we can cope with multiple types with the same name in
* distilled base BTF. In this case because sizes are different,
* we can still disambiguate them.
*/
static void test_distilled_base_multi(void)
{
struct btf *btf1 = NULL, *btf2 = NULL, *btf3 = NULL, *btf4 = NULL;
btf1 = btf__new_empty();
if (!ASSERT_OK_PTR(btf1, "empty_main_btf"))
return;
btf__add_int(btf1, "int", 4, BTF_INT_SIGNED); /* [1] int */
btf__add_int(btf1, "int", 8, BTF_INT_SIGNED); /* [2] int */
VALIDATE_RAW_BTF(
btf1,
"[1] INT 'int' size=4 bits_offset=0 nr_bits=32 encoding=SIGNED",
"[2] INT 'int' size=8 bits_offset=0 nr_bits=64 encoding=SIGNED");
btf2 = btf__new_empty_split(btf1);
if (!ASSERT_OK_PTR(btf2, "empty_split_btf"))
goto cleanup;
btf__add_ptr(btf2, 1);
btf__add_const(btf2, 2);
VALIDATE_RAW_BTF(
btf2,
"[1] INT 'int' size=4 bits_offset=0 nr_bits=32 encoding=SIGNED",
"[2] INT 'int' size=8 bits_offset=0 nr_bits=64 encoding=SIGNED",
"[3] PTR '(anon)' type_id=1",
"[4] CONST '(anon)' type_id=2");
if (!ASSERT_EQ(0, btf__distill_base(btf2, &btf3, &btf4),
"distilled_base") ||
!ASSERT_OK_PTR(btf3, "distilled_base") ||
!ASSERT_OK_PTR(btf4, "distilled_split") ||
!ASSERT_EQ(3, btf__type_cnt(btf3), "distilled_base_type_cnt"))
goto cleanup;
VALIDATE_RAW_BTF(
btf3,
"[1] INT 'int' size=4 bits_offset=0 nr_bits=32 encoding=SIGNED",
"[2] INT 'int' size=8 bits_offset=0 nr_bits=64 encoding=SIGNED");
if (!ASSERT_EQ(btf__relocate(btf4, btf1), 0, "relocate_split"))
goto cleanup;
VALIDATE_RAW_BTF(
btf4,
"[1] INT 'int' size=4 bits_offset=0 nr_bits=32 encoding=SIGNED",
"[2] INT 'int' size=8 bits_offset=0 nr_bits=64 encoding=SIGNED",
"[3] PTR '(anon)' type_id=1",
"[4] CONST '(anon)' type_id=2");
cleanup:
btf__free(btf4);
btf__free(btf3);
btf__free(btf2);
btf__free(btf1);
}
/* If a needed type is not present in the base BTF we wish to relocate
* with, btf__relocate() should error our.
*/
static void test_distilled_base_missing_err(void)
{
struct btf *btf1 = NULL, *btf2 = NULL, *btf3 = NULL, *btf4 = NULL, *btf5 = NULL;
btf1 = btf__new_empty();
if (!ASSERT_OK_PTR(btf1, "empty_main_btf"))
return;
btf__add_int(btf1, "int", 4, BTF_INT_SIGNED); /* [1] int */
btf__add_int(btf1, "int", 8, BTF_INT_SIGNED); /* [2] int */
VALIDATE_RAW_BTF(
btf1,
"[1] INT 'int' size=4 bits_offset=0 nr_bits=32 encoding=SIGNED",
"[2] INT 'int' size=8 bits_offset=0 nr_bits=64 encoding=SIGNED");
btf2 = btf__new_empty_split(btf1);
if (!ASSERT_OK_PTR(btf2, "empty_split_btf"))
goto cleanup;
btf__add_ptr(btf2, 1);
btf__add_const(btf2, 2);
VALIDATE_RAW_BTF(
btf2,
"[1] INT 'int' size=4 bits_offset=0 nr_bits=32 encoding=SIGNED",
"[2] INT 'int' size=8 bits_offset=0 nr_bits=64 encoding=SIGNED",
"[3] PTR '(anon)' type_id=1",
"[4] CONST '(anon)' type_id=2");
if (!ASSERT_EQ(0, btf__distill_base(btf2, &btf3, &btf4),
"distilled_base") ||
!ASSERT_OK_PTR(btf3, "distilled_base") ||
!ASSERT_OK_PTR(btf4, "distilled_split") ||
!ASSERT_EQ(3, btf__type_cnt(btf3), "distilled_base_type_cnt"))
goto cleanup;
VALIDATE_RAW_BTF(
btf3,
"[1] INT 'int' size=4 bits_offset=0 nr_bits=32 encoding=SIGNED",
"[2] INT 'int' size=8 bits_offset=0 nr_bits=64 encoding=SIGNED");
btf5 = btf__new_empty();
if (!ASSERT_OK_PTR(btf5, "empty_reloc_btf"))
return;
btf__add_int(btf5, "int", 4, BTF_INT_SIGNED); /* [1] int */
VALIDATE_RAW_BTF(
btf5,
"[1] INT 'int' size=4 bits_offset=0 nr_bits=32 encoding=SIGNED");
ASSERT_EQ(btf__relocate(btf4, btf5), -EINVAL, "relocate_split");
cleanup:
btf__free(btf5);
btf__free(btf4);
btf__free(btf3);
btf__free(btf2);
btf__free(btf1);
}
/* With 2 types of same size in distilled base BTF, relocation should
* fail as we have no means to choose between them.
*/
static void test_distilled_base_multi_err(void)
{
struct btf *btf1 = NULL, *btf2 = NULL, *btf3 = NULL, *btf4 = NULL;
btf1 = btf__new_empty();
if (!ASSERT_OK_PTR(btf1, "empty_main_btf"))
return;
btf__add_int(btf1, "int", 4, BTF_INT_SIGNED); /* [1] int */
btf__add_int(btf1, "int", 4, BTF_INT_SIGNED); /* [2] int */
VALIDATE_RAW_BTF(
btf1,
"[1] INT 'int' size=4 bits_offset=0 nr_bits=32 encoding=SIGNED",
"[2] INT 'int' size=4 bits_offset=0 nr_bits=32 encoding=SIGNED");
btf2 = btf__new_empty_split(btf1);
if (!ASSERT_OK_PTR(btf2, "empty_split_btf"))
goto cleanup;
btf__add_ptr(btf2, 1);
btf__add_const(btf2, 2);
VALIDATE_RAW_BTF(
btf2,
"[1] INT 'int' size=4 bits_offset=0 nr_bits=32 encoding=SIGNED",
"[2] INT 'int' size=4 bits_offset=0 nr_bits=32 encoding=SIGNED",
"[3] PTR '(anon)' type_id=1",
"[4] CONST '(anon)' type_id=2");
if (!ASSERT_EQ(0, btf__distill_base(btf2, &btf3, &btf4),
"distilled_base") ||
!ASSERT_OK_PTR(btf3, "distilled_base") ||
!ASSERT_OK_PTR(btf4, "distilled_split") ||
!ASSERT_EQ(3, btf__type_cnt(btf3), "distilled_base_type_cnt"))
goto cleanup;
VALIDATE_RAW_BTF(
btf3,
"[1] INT 'int' size=4 bits_offset=0 nr_bits=32 encoding=SIGNED",
"[2] INT 'int' size=4 bits_offset=0 nr_bits=32 encoding=SIGNED");
ASSERT_EQ(btf__relocate(btf4, btf1), -EINVAL, "relocate_split");
cleanup:
btf__free(btf4);
btf__free(btf3);
btf__free(btf2);
btf__free(btf1);
}
/* With 2 types of same size in base BTF, relocation should
* fail as we have no means to choose between them.
*/
static void test_distilled_base_multi_err2(void)
{
struct btf *btf1 = NULL, *btf2 = NULL, *btf3 = NULL, *btf4 = NULL, *btf5 = NULL;
btf1 = btf__new_empty();
if (!ASSERT_OK_PTR(btf1, "empty_main_btf"))
return;
btf__add_int(btf1, "int", 4, BTF_INT_SIGNED); /* [1] int */
VALIDATE_RAW_BTF(
btf1,
"[1] INT 'int' size=4 bits_offset=0 nr_bits=32 encoding=SIGNED");
btf2 = btf__new_empty_split(btf1);
if (!ASSERT_OK_PTR(btf2, "empty_split_btf"))
goto cleanup;
btf__add_ptr(btf2, 1);
VALIDATE_RAW_BTF(
btf2,
"[1] INT 'int' size=4 bits_offset=0 nr_bits=32 encoding=SIGNED",
"[2] PTR '(anon)' type_id=1");
if (!ASSERT_EQ(0, btf__distill_base(btf2, &btf3, &btf4),
"distilled_base") ||
!ASSERT_OK_PTR(btf3, "distilled_base") ||
!ASSERT_OK_PTR(btf4, "distilled_split") ||
!ASSERT_EQ(2, btf__type_cnt(btf3), "distilled_base_type_cnt"))
goto cleanup;
VALIDATE_RAW_BTF(
btf3,
"[1] INT 'int' size=4 bits_offset=0 nr_bits=32 encoding=SIGNED");
btf5 = btf__new_empty();
if (!ASSERT_OK_PTR(btf5, "empty_reloc_btf"))
return;
btf__add_int(btf5, "int", 4, BTF_INT_SIGNED); /* [1] int */
btf__add_int(btf5, "int", 4, BTF_INT_SIGNED); /* [2] int */
VALIDATE_RAW_BTF(
btf5,
"[1] INT 'int' size=4 bits_offset=0 nr_bits=32 encoding=SIGNED",
"[2] INT 'int' size=4 bits_offset=0 nr_bits=32 encoding=SIGNED");
ASSERT_EQ(btf__relocate(btf4, btf5), -EINVAL, "relocate_split");
cleanup:
btf__free(btf5);
btf__free(btf4);
btf__free(btf3);
btf__free(btf2);
btf__free(btf1);
}
/* create split reference BTF from vmlinux + split BTF with a few type references;
* ensure the resultant split reference BTF is as expected, containing only types
* needed to disambiguate references from split BTF.
*/
static void test_distilled_base_vmlinux(void)
{
struct btf *split_btf = NULL, *vmlinux_btf = btf__load_vmlinux_btf();
struct btf *split_dist = NULL, *base_dist = NULL;
__s32 int_id, myint_id;
if (!ASSERT_OK_PTR(vmlinux_btf, "load_vmlinux"))
return;
int_id = btf__find_by_name_kind(vmlinux_btf, "int", BTF_KIND_INT);
if (!ASSERT_GT(int_id, 0, "find_int"))
goto cleanup;
split_btf = btf__new_empty_split(vmlinux_btf);
if (!ASSERT_OK_PTR(split_btf, "new_split"))
goto cleanup;
myint_id = btf__add_typedef(split_btf, "myint", int_id);
btf__add_ptr(split_btf, myint_id);
if (!ASSERT_EQ(btf__distill_base(split_btf, &base_dist, &split_dist), 0,
"distill_vmlinux_base"))
goto cleanup;
if (!ASSERT_OK_PTR(split_dist, "split_distilled") ||
!ASSERT_OK_PTR(base_dist, "base_dist"))
goto cleanup;
VALIDATE_RAW_BTF(
split_dist,
"[1] INT 'int' size=4 bits_offset=0 nr_bits=32 encoding=SIGNED",
"[2] TYPEDEF 'myint' type_id=1",
"[3] PTR '(anon)' type_id=2");
cleanup:
btf__free(split_dist);
btf__free(base_dist);
btf__free(split_btf);
btf__free(vmlinux_btf);
}
void test_btf_distill(void)
{
if (test__start_subtest("distilled_base"))
test_distilled_base();
if (test__start_subtest("distilled_base_multi"))
test_distilled_base_multi();
if (test__start_subtest("distilled_base_missing_err"))
test_distilled_base_missing_err();
if (test__start_subtest("distilled_base_multi_err"))
test_distilled_base_multi_err();
if (test__start_subtest("distilled_base_multi_err2"))
test_distilled_base_multi_err2();
if (test__start_subtest("distilled_base_vmlinux"))
test_distilled_base_vmlinux();
}

4
tools/testing/selftests/bpf/prog_tests/cgroup_v1v2.c
View File

@ -32,7 +32,7 @@ static int run_test(int cgroup_fd, int server_fd, bool classid)
goto out;
}
fd = connect_to_fd_opts(server_fd, &opts);
fd = connect_to_fd_opts(server_fd, SOCK_STREAM, &opts);
if (fd < 0)
err = -1;
else
@ -52,7 +52,7 @@ void test_cgroup_v1v2(void)
server_fd = start_server(AF_INET, SOCK_STREAM, NULL, port, 0);
if (!ASSERT_GE(server_fd, 0, "server_fd"))
return;
client_fd = connect_to_fd_opts(server_fd, &opts);
client_fd = connect_to_fd_opts(server_fd, SOCK_STREAM, &opts);
if (!ASSERT_GE(client_fd, 0, "client_fd")) {
close(server_fd);
return;

4
tools/testing/selftests/bpf/prog_tests/fexit_stress.c
View File

@ -50,9 +50,9 @@ void serial_test_fexit_stress(void)
out:
for (i = 0; i < bpf_max_tramp_links; i++) {
if (link_fd[i])
if (link_fd[i] > 0)
close(link_fd[i]);
if (fexit_fd[i])
if (fexit_fd[i] > 0)
close(fexit_fd[i]);
}
free(fd);

14
tools/testing/selftests/bpf/prog_tests/ip_check_defrag.c
View File

@ -158,15 +158,13 @@ static int send_frags6(int client)
void test_bpf_ip_check_defrag_ok(bool ipv6)
{
int family = ipv6 ? AF_INET6 : AF_INET;
struct network_helper_opts rx_opts = {
.timeout_ms = 1000,
.noconnect = true,
};
struct network_helper_opts tx_ops = {
.timeout_ms = 1000,
.type = SOCK_RAW,
.proto = IPPROTO_RAW,
.noconnect = true,
};
struct sockaddr_storage caddr;
struct ip_check_defrag *skel;
@ -192,7 +190,7 @@ void test_bpf_ip_check_defrag_ok(bool ipv6)
nstoken = open_netns(NS1);
if (!ASSERT_OK_PTR(nstoken, "setns ns1"))
goto out;
srv_fd = start_server(ipv6 ? AF_INET6 : AF_INET, SOCK_DGRAM, NULL, SERVER_PORT, 0);
srv_fd = start_server(family, SOCK_DGRAM, NULL, SERVER_PORT, 0);
close_netns(nstoken);
if (!ASSERT_GE(srv_fd, 0, "start_server"))
goto out;
@ -201,18 +199,18 @@ void test_bpf_ip_check_defrag_ok(bool ipv6)
nstoken = open_netns(NS0);
if (!ASSERT_OK_PTR(nstoken, "setns ns0"))
goto out;
client_tx_fd = connect_to_fd_opts(srv_fd, &tx_ops);
client_tx_fd = client_socket(family, SOCK_RAW, &tx_ops);
close_netns(nstoken);
if (!ASSERT_GE(client_tx_fd, 0, "connect_to_fd_opts"))
if (!ASSERT_GE(client_tx_fd, 0, "client_socket"))
goto out;
/* Open rx socket in ns0 */
nstoken = open_netns(NS0);
if (!ASSERT_OK_PTR(nstoken, "setns ns0"))
goto out;
client_rx_fd = connect_to_fd_opts(srv_fd, &rx_opts);
client_rx_fd = client_socket(family, SOCK_DGRAM, &rx_opts);
close_netns(nstoken);
if (!ASSERT_GE(client_rx_fd, 0, "connect_to_fd_opts"))
if (!ASSERT_GE(client_rx_fd, 0, "client_socket"))
goto out;
/* Bind rx socket to a premeditated port */

1
tools/testing/selftests/bpf/prog_tests/kfunc_call.c
View File

@ -78,6 +78,7 @@ static struct kfunc_test_params kfunc_tests[] = {
SYSCALL_TEST(kfunc_syscall_test, 0),
SYSCALL_NULL_CTX_TEST(kfunc_syscall_test_null, 0),
TC_TEST(kfunc_call_test_static_unused_arg, 0),
TC_TEST(kfunc_call_ctx, 0),
};
struct syscall_test_args {

11
tools/testing/selftests/bpf/prog_tests/kfunc_param_nullable.c Normal file
View File

@ -0,0 +1,11 @@
// SPDX-License-Identifier: GPL-2.0
/* Copyright (c) 2024 Meta Platforms, Inc */
#include <test_progs.h>
#include "test_kfunc_param_nullable.skel.h"
void test_kfunc_param_nullable(void)
{
RUN_TESTS(test_kfunc_param_nullable);
}

7
tools/testing/selftests/bpf/prog_tests/mptcp.c
View File

@ -89,13 +89,8 @@ static int start_mptcp_server(int family, const char *addr_str, __u16 port,
.timeout_ms = timeout_ms,
.proto = IPPROTO_MPTCP,
};
struct sockaddr_storage addr;
socklen_t addrlen;
if (make_sockaddr(family, addr_str, port, &addr, &addrlen))
return -1;
return start_server_addr(SOCK_STREAM, &addr, addrlen, &opts);
return start_server_str(family, SOCK_STREAM, addr_str, port, &opts);
}
static int verify_tsk(int map_fd, int client_fd)

1
tools/testing/selftests/bpf/prog_tests/test_skb_pkt_end.c
View File

@ -11,6 +11,7 @@ static int sanity_run(struct bpf_program *prog)
.data_in = &pkt_v4,
.data_size_in = sizeof(pkt_v4),
.repeat = 1,
.flags = BPF_F_TEST_SKB_CHECKSUM_COMPLETE,
);
prog_fd = bpf_program__fd(prog);

44
tools/testing/selftests/bpf/prog_tests/tracing_struct.c
View File

@ -3,8 +3,9 @@
#include <test_progs.h>
#include "tracing_struct.skel.h"
#include "tracing_struct_many_args.skel.h"
static void test_fentry(void)
static void test_struct_args(void)
{
struct tracing_struct *skel;
int err;
@ -55,6 +56,25 @@ static void test_fentry(void)
ASSERT_EQ(skel->bss->t6, 1, "t6 ret");
destroy_skel:
tracing_struct__destroy(skel);
}
static void test_struct_many_args(void)
{
struct tracing_struct_many_args *skel;
int err;
skel = tracing_struct_many_args__open_and_load();
if (!ASSERT_OK_PTR(skel, "tracing_struct_many_args__open_and_load"))
return;
err = tracing_struct_many_args__attach(skel);
if (!ASSERT_OK(err, "tracing_struct_many_args__attach"))
goto destroy_skel;
ASSERT_OK(trigger_module_test_read(256), "trigger_read");
ASSERT_EQ(skel->bss->t7_a, 16, "t7:a");
ASSERT_EQ(skel->bss->t7_b, 17, "t7:b");
ASSERT_EQ(skel->bss->t7_c, 18, "t7:c");
@ -74,12 +94,28 @@ static void test_fentry(void)
ASSERT_EQ(skel->bss->t8_g, 23, "t8:g");
ASSERT_EQ(skel->bss->t8_ret, 156, "t8 ret");
tracing_struct__detach(skel);
ASSERT_EQ(skel->bss->t9_a, 16, "t9:a");
ASSERT_EQ(skel->bss->t9_b, 17, "t9:b");
ASSERT_EQ(skel->bss->t9_c, 18, "t9:c");
ASSERT_EQ(skel->bss->t9_d, 19, "t9:d");
ASSERT_EQ(skel->bss->t9_e, 20, "t9:e");
ASSERT_EQ(skel->bss->t9_f, 21, "t9:f");
ASSERT_EQ(skel->bss->t9_g, 22, "t9:f");
ASSERT_EQ(skel->bss->t9_h_a, 23, "t9:h.a");
ASSERT_EQ(skel->bss->t9_h_b, 24, "t9:h.b");
ASSERT_EQ(skel->bss->t9_h_c, 25, "t9:h.c");
ASSERT_EQ(skel->bss->t9_h_d, 26, "t9:h.d");
ASSERT_EQ(skel->bss->t9_i, 27, "t9:i");
ASSERT_EQ(skel->bss->t9_ret, 258, "t9 ret");
destroy_skel:
tracing_struct__destroy(skel);
tracing_struct_many_args__destroy(skel);
}
void test_tracing_struct(void)
{
test_fentry();
if (test__start_subtest("struct_args"))
test_struct_args();
if (test__start_subtest("struct_many_args"))
test_struct_many_args();
}

168
tools/testing/selftests/bpf/prog_tests/xdp_flowtable.c Normal file
View File

@ -0,0 +1,168 @@
// SPDX-License-Identifier: GPL-2.0
#include <test_progs.h>
#include <network_helpers.h>
#include <bpf/btf.h>
#include <linux/if_link.h>
#include <linux/udp.h>
#include <net/if.h>
#include <unistd.h>
#include "xdp_flowtable.skel.h"
#define TX_NETNS_NAME "ns0"
#define RX_NETNS_NAME "ns1"
#define TX_NAME "v0"
#define FORWARD_NAME "v1"
#define RX_NAME "d0"
#define TX_MAC "00:00:00:00:00:01"
#define FORWARD_MAC "00:00:00:00:00:02"
#define RX_MAC "00:00:00:00:00:03"
#define DST_MAC "00:00:00:00:00:04"
#define TX_ADDR "10.0.0.1"
#define FORWARD_ADDR "10.0.0.2"
#define RX_ADDR "20.0.0.1"
#define DST_ADDR "20.0.0.2"
#define PREFIX_LEN "8"
#define N_PACKETS 10
#define UDP_PORT 12345
#define UDP_PORT_STR "12345"
static int send_udp_traffic(void)
{
struct sockaddr_storage addr;
int i, sock;
if (make_sockaddr(AF_INET, DST_ADDR, UDP_PORT, &addr, NULL))
return -EINVAL;
sock = socket(AF_INET, SOCK_DGRAM, 0);
if (sock < 0)
return sock;
for (i = 0; i < N_PACKETS; i++) {
unsigned char buf[] = { 0xaa, 0xbb, 0xcc };
int n;
n = sendto(sock, buf, sizeof(buf), MSG_NOSIGNAL | MSG_CONFIRM,
(struct sockaddr *)&addr, sizeof(addr));
if (n != sizeof(buf)) {
close(sock);
return -EINVAL;
}
usleep(50000); /* 50ms */
}
close(sock);
return 0;
}
void test_xdp_flowtable(void)
{
struct xdp_flowtable *skel = NULL;
struct nstoken *tok = NULL;
int iifindex, stats_fd;
__u32 value, key = 0;
struct bpf_link *link;
if (SYS_NOFAIL("nft -v")) {
fprintf(stdout, "Missing required nft tool\n");
test__skip();
return;
}
SYS(out, "ip netns add " TX_NETNS_NAME);
SYS(out, "ip netns add " RX_NETNS_NAME);
tok = open_netns(RX_NETNS_NAME);
if (!ASSERT_OK_PTR(tok, "setns"))
goto out;
SYS(out, "sysctl -qw net.ipv4.conf.all.forwarding=1");
SYS(out, "ip link add " TX_NAME " type veth peer " FORWARD_NAME);
SYS(out, "ip link set " TX_NAME " netns " TX_NETNS_NAME);
SYS(out, "ip link set dev " FORWARD_NAME " address " FORWARD_MAC);
SYS(out,
"ip addr add " FORWARD_ADDR "/" PREFIX_LEN " dev " FORWARD_NAME);
SYS(out, "ip link set dev " FORWARD_NAME " up");
SYS(out, "ip link add " RX_NAME " type dummy");
SYS(out, "ip link set dev " RX_NAME " address " RX_MAC);
SYS(out, "ip addr add " RX_ADDR "/" PREFIX_LEN " dev " RX_NAME);
SYS(out, "ip link set dev " RX_NAME " up");
/* configure the flowtable */
SYS(out, "nft add table ip filter");
SYS(out,
"nft add flowtable ip filter f { hook ingress priority 0\\; "
"devices = { " FORWARD_NAME ", " RX_NAME " }\\; }");
SYS(out,
"nft add chain ip filter forward "
"{ type filter hook forward priority 0\\; }");
SYS(out,
"nft add rule ip filter forward ip protocol udp th dport "
UDP_PORT_STR " flow add @f");
/* Avoid ARP calls */
SYS(out,
"ip -4 neigh add " DST_ADDR " lladdr " DST_MAC " dev " RX_NAME);
close_netns(tok);
tok = open_netns(TX_NETNS_NAME);
if (!ASSERT_OK_PTR(tok, "setns"))
goto out;
SYS(out, "ip addr add " TX_ADDR "/" PREFIX_LEN " dev " TX_NAME);
SYS(out, "ip link set dev " TX_NAME " address " TX_MAC);
SYS(out, "ip link set dev " TX_NAME " up");
SYS(out, "ip route add default via " FORWARD_ADDR);
close_netns(tok);
tok = open_netns(RX_NETNS_NAME);
if (!ASSERT_OK_PTR(tok, "setns"))
goto out;
iifindex = if_nametoindex(FORWARD_NAME);
if (!ASSERT_NEQ(iifindex, 0, "iifindex"))
goto out;
skel = xdp_flowtable__open_and_load();
if (!ASSERT_OK_PTR(skel, "skel"))
goto out;
link = bpf_program__attach_xdp(skel->progs.xdp_flowtable_do_lookup,
iifindex);
if (!ASSERT_OK_PTR(link, "prog_attach"))
goto out;
close_netns(tok);
tok = open_netns(TX_NETNS_NAME);
if (!ASSERT_OK_PTR(tok, "setns"))
goto out;
if (!ASSERT_OK(send_udp_traffic(), "send udp"))
goto out;
close_netns(tok);
tok = open_netns(RX_NETNS_NAME);
if (!ASSERT_OK_PTR(tok, "setns"))
goto out;
stats_fd = bpf_map__fd(skel->maps.stats);
if (!ASSERT_OK(bpf_map_lookup_elem(stats_fd, &key, &value),
"bpf_map_update_elem stats"))
goto out;
ASSERT_GE(value, N_PACKETS - 2, "bpf_xdp_flow_lookup failed");
out:
xdp_flowtable__destroy(skel);
if (tok)
close_netns(tok);
SYS_NOFAIL("ip netns del " TX_NETNS_NAME);
SYS_NOFAIL("ip netns del " RX_NETNS_NAME);
}

143
tools/testing/selftests/bpf/progs/arena_atomics.c
View File

@ -25,20 +25,13 @@ bool skip_tests = true;
__u32 pid = 0;
#undef __arena
#if defined(__BPF_FEATURE_ADDR_SPACE_CAST)
#define __arena __attribute__((address_space(1)))
#else
#define __arena SEC(".addr_space.1")
#endif
__u64 __arena add64_value = 1;
__u64 __arena add64_result = 0;
__u32 __arena add32_value = 1;
__u32 __arena add32_result = 0;
__u64 __arena add_stack_value_copy = 0;
__u64 __arena add_stack_result = 0;
__u64 __arena add_noreturn_value = 1;
__u64 __arena_global add64_value = 1;
__u64 __arena_global add64_result = 0;
__u32 __arena_global add32_value = 1;
__u32 __arena_global add32_result = 0;
__u64 __arena_global add_stack_value_copy = 0;
__u64 __arena_global add_stack_result = 0;
__u64 __arena_global add_noreturn_value = 1;
SEC("raw_tp/sys_enter")
int add(const void *ctx)
@ -58,13 +51,13 @@ int add(const void *ctx)
return 0;
}
__s64 __arena sub64_value = 1;
__s64 __arena sub64_result = 0;
__s32 __arena sub32_value = 1;
__s32 __arena sub32_result = 0;
__s64 __arena sub_stack_value_copy = 0;
__s64 __arena sub_stack_result = 0;
__s64 __arena sub_noreturn_value = 1;
__s64 __arena_global sub64_value = 1;
__s64 __arena_global sub64_result = 0;
__s32 __arena_global sub32_value = 1;
__s32 __arena_global sub32_result = 0;
__s64 __arena_global sub_stack_value_copy = 0;
__s64 __arena_global sub_stack_result = 0;
__s64 __arena_global sub_noreturn_value = 1;
SEC("raw_tp/sys_enter")
int sub(const void *ctx)
@ -84,8 +77,8 @@ int sub(const void *ctx)
return 0;
}
__u64 __arena and64_value = (0x110ull << 32);
__u32 __arena and32_value = 0x110;
__u64 __arena_global and64_value = (0x110ull << 32);
__u32 __arena_global and32_value = 0x110;
SEC("raw_tp/sys_enter")
int and(const void *ctx)
@ -101,8 +94,8 @@ int and(const void *ctx)
return 0;
}
__u32 __arena or32_value = 0x110;
__u64 __arena or64_value = (0x110ull << 32);
__u32 __arena_global or32_value = 0x110;
__u64 __arena_global or64_value = (0x110ull << 32);
SEC("raw_tp/sys_enter")
int or(const void *ctx)
@ -117,8 +110,8 @@ int or(const void *ctx)
return 0;
}
__u64 __arena xor64_value = (0x110ull << 32);
__u32 __arena xor32_value = 0x110;
__u64 __arena_global xor64_value = (0x110ull << 32);
__u32 __arena_global xor32_value = 0x110;
SEC("raw_tp/sys_enter")
int xor(const void *ctx)
@ -133,12 +126,12 @@ int xor(const void *ctx)
return 0;
}
__u32 __arena cmpxchg32_value = 1;
__u32 __arena cmpxchg32_result_fail = 0;
__u32 __arena cmpxchg32_result_succeed = 0;
__u64 __arena cmpxchg64_value = 1;
__u64 __arena cmpxchg64_result_fail = 0;
__u64 __arena cmpxchg64_result_succeed = 0;
__u32 __arena_global cmpxchg32_value = 1;
__u32 __arena_global cmpxchg32_result_fail = 0;
__u32 __arena_global cmpxchg32_result_succeed = 0;
__u64 __arena_global cmpxchg64_value = 1;
__u64 __arena_global cmpxchg64_result_fail = 0;
__u64 __arena_global cmpxchg64_result_succeed = 0;
SEC("raw_tp/sys_enter")
int cmpxchg(const void *ctx)
@ -156,10 +149,10 @@ int cmpxchg(const void *ctx)
return 0;
}
__u64 __arena xchg64_value = 1;
__u64 __arena xchg64_result = 0;
__u32 __arena xchg32_value = 1;
__u32 __arena xchg32_result = 0;
__u64 __arena_global xchg64_value = 1;
__u64 __arena_global xchg64_result = 0;
__u32 __arena_global xchg32_value = 1;
__u32 __arena_global xchg32_result = 0;
SEC("raw_tp/sys_enter")
int xchg(const void *ctx)
@ -176,3 +169,79 @@ int xchg(const void *ctx)
return 0;
}
__u64 __arena_global uaf_sink;
volatile __u64 __arena_global uaf_recovery_fails;
SEC("syscall")
int uaf(const void *ctx)
{
if (pid != (bpf_get_current_pid_tgid() >> 32))
return 0;
#if defined(ENABLE_ATOMICS_TESTS) && !defined(__TARGET_ARCH_arm64) && \
!defined(__TARGET_ARCH_x86)
__u32 __arena *page32;
__u64 __arena *page64;
void __arena *page;
page = bpf_arena_alloc_pages(&arena, NULL, 1, NUMA_NO_NODE, 0);
bpf_arena_free_pages(&arena, page, 1);
uaf_recovery_fails = 24;
page32 = (__u32 __arena *)page;
uaf_sink += __sync_fetch_and_add(page32, 1);
uaf_recovery_fails -= 1;
__sync_add_and_fetch(page32, 1);
uaf_recovery_fails -= 1;
uaf_sink += __sync_fetch_and_sub(page32, 1);
uaf_recovery_fails -= 1;
__sync_sub_and_fetch(page32, 1);
uaf_recovery_fails -= 1;
uaf_sink += __sync_fetch_and_and(page32, 1);
uaf_recovery_fails -= 1;
__sync_and_and_fetch(page32, 1);
uaf_recovery_fails -= 1;
uaf_sink += __sync_fetch_and_or(page32, 1);
uaf_recovery_fails -= 1;
__sync_or_and_fetch(page32, 1);
uaf_recovery_fails -= 1;
uaf_sink += __sync_fetch_and_xor(page32, 1);
uaf_recovery_fails -= 1;
__sync_xor_and_fetch(page32, 1);
uaf_recovery_fails -= 1;
uaf_sink += __sync_val_compare_and_swap(page32, 0, 1);
uaf_recovery_fails -= 1;
uaf_sink += __sync_lock_test_and_set(page32, 1);
uaf_recovery_fails -= 1;
page64 = (__u64 __arena *)page;
uaf_sink += __sync_fetch_and_add(page64, 1);
uaf_recovery_fails -= 1;
__sync_add_and_fetch(page64, 1);
uaf_recovery_fails -= 1;
uaf_sink += __sync_fetch_and_sub(page64, 1);
uaf_recovery_fails -= 1;
__sync_sub_and_fetch(page64, 1);
uaf_recovery_fails -= 1;
uaf_sink += __sync_fetch_and_and(page64, 1);
uaf_recovery_fails -= 1;
__sync_and_and_fetch(page64, 1);
uaf_recovery_fails -= 1;
uaf_sink += __sync_fetch_and_or(page64, 1);
uaf_recovery_fails -= 1;
__sync_or_and_fetch(page64, 1);
uaf_recovery_fails -= 1;
uaf_sink += __sync_fetch_and_xor(page64, 1);
uaf_recovery_fails -= 1;
__sync_xor_and_fetch(page64, 1);
uaf_recovery_fails -= 1;
uaf_sink += __sync_val_compare_and_swap(page64, 0, 1);
uaf_recovery_fails -= 1;
uaf_sink += __sync_lock_test_and_set(page64, 1);
uaf_recovery_fails -= 1;
#endif
return 0;
}
char _license[] SEC("license") = "GPL";

21
tools/testing/selftests/bpf/progs/arena_htab.c
View File

@ -1,5 +1,6 @@
// SPDX-License-Identifier: GPL-2.0
/* Copyright (c) 2024 Meta Platforms, Inc. and affiliates. */
#define BPF_NO_KFUNC_PROTOTYPES
#include <vmlinux.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>
@ -18,25 +19,35 @@ void __arena *htab_for_user;
bool skip = false;
int zero = 0;
char __arena arr1[100000];
char arr2[1000];
SEC("syscall")
int arena_htab_llvm(void *ctx)
{
#if defined(__BPF_FEATURE_ADDR_SPACE_CAST) || defined(BPF_ARENA_FORCE_ASM)
struct htab __arena *htab;
char __arena *arr = arr1;
__u64 i;
htab = bpf_alloc(sizeof(*htab));
cast_kern(htab);
htab_init(htab);
/* first run. No old elems in the table */
for (i = zero; i < 1000; i++)
htab_update_elem(htab, i, i);
cast_kern(arr);
/* should replace all elems with new ones */
for (i = zero; i < 1000; i++)
/* first run. No old elems in the table */
for (i = zero; i < 100000 && can_loop; i++) {
htab_update_elem(htab, i, i);
arr[i] = i;
}
/* should replace some elems with new ones */
for (i = zero; i < 1000 && can_loop; i++) {
htab_update_elem(htab, i, i);
/* Access mem to make the verifier use bounded loop logic */
arr2[i] = i;
}
cast_user(htab);
htab_for_user = htab;
#else

1
tools/testing/selftests/bpf/progs/arena_list.c
View File

@ -1,5 +1,6 @@
// SPDX-License-Identifier: GPL-2.0
/* Copyright (c) 2024 Meta Platforms, Inc. and affiliates. */
#define BPF_NO_KFUNC_PROTOTYPES
#include <vmlinux.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>

36
tools/testing/selftests/bpf/progs/bpf_dctcp.c
View File

@ -65,7 +65,7 @@ static void dctcp_reset(const struct tcp_sock *tp, struct bpf_dctcp *ca)
}
SEC("struct_ops")
void BPF_PROG(dctcp_init, struct sock *sk)
void BPF_PROG(bpf_dctcp_init, struct sock *sk)
{
const struct tcp_sock *tp = tcp_sk(sk);
struct bpf_dctcp *ca = inet_csk_ca(sk);
@ -77,7 +77,7 @@ void BPF_PROG(dctcp_init, struct sock *sk)
(void *)fallback, sizeof(fallback)) == -EBUSY)
ebusy_cnt++;
/* Switch back to myself and the recurred dctcp_init()
/* Switch back to myself and the recurred bpf_dctcp_init()
* will get -EBUSY for all bpf_setsockopt(TCP_CONGESTION),
* except the last "cdg" one.
*/
@ -112,7 +112,7 @@ void BPF_PROG(dctcp_init, struct sock *sk)
}
SEC("struct_ops")
__u32 BPF_PROG(dctcp_ssthresh, struct sock *sk)
__u32 BPF_PROG(bpf_dctcp_ssthresh, struct sock *sk)
{
struct bpf_dctcp *ca = inet_csk_ca(sk);
struct tcp_sock *tp = tcp_sk(sk);
@ -122,7 +122,7 @@ __u32 BPF_PROG(dctcp_ssthresh, struct sock *sk)
}
SEC("struct_ops")
void BPF_PROG(dctcp_update_alpha, struct sock *sk, __u32 flags)
void BPF_PROG(bpf_dctcp_update_alpha, struct sock *sk, __u32 flags)
{
const struct tcp_sock *tp = tcp_sk(sk);
struct bpf_dctcp *ca = inet_csk_ca(sk);
@ -161,12 +161,12 @@ static void dctcp_react_to_loss(struct sock *sk)
}
SEC("struct_ops")
void BPF_PROG(dctcp_state, struct sock *sk, __u8 new_state)
void BPF_PROG(bpf_dctcp_state, struct sock *sk, __u8 new_state)
{
if (new_state == TCP_CA_Recovery &&
new_state != BPF_CORE_READ_BITFIELD(inet_csk(sk), icsk_ca_state))
dctcp_react_to_loss(sk);
/* We handle RTO in dctcp_cwnd_event to ensure that we perform only
/* We handle RTO in bpf_dctcp_cwnd_event to ensure that we perform only
* one loss-adjustment per RTT.
*/
}
@ -208,7 +208,7 @@ static void dctcp_ece_ack_update(struct sock *sk, enum tcp_ca_event evt,
}
SEC("struct_ops")
void BPF_PROG(dctcp_cwnd_event, struct sock *sk, enum tcp_ca_event ev)
void BPF_PROG(bpf_dctcp_cwnd_event, struct sock *sk, enum tcp_ca_event ev)
{
struct bpf_dctcp *ca = inet_csk_ca(sk);
@ -227,7 +227,7 @@ void BPF_PROG(dctcp_cwnd_event, struct sock *sk, enum tcp_ca_event ev)
}
SEC("struct_ops")
__u32 BPF_PROG(dctcp_cwnd_undo, struct sock *sk)
__u32 BPF_PROG(bpf_dctcp_cwnd_undo, struct sock *sk)
{
const struct bpf_dctcp *ca = inet_csk_ca(sk);
@ -237,28 +237,28 @@ __u32 BPF_PROG(dctcp_cwnd_undo, struct sock *sk)
extern void tcp_reno_cong_avoid(struct sock *sk, __u32 ack, __u32 acked) __ksym;
SEC("struct_ops")
void BPF_PROG(dctcp_cong_avoid, struct sock *sk, __u32 ack, __u32 acked)
void BPF_PROG(bpf_dctcp_cong_avoid, struct sock *sk, __u32 ack, __u32 acked)
{
tcp_reno_cong_avoid(sk, ack, acked);
}
SEC(".struct_ops")
struct tcp_congestion_ops dctcp_nouse = {
.init = (void *)dctcp_init,
.set_state = (void *)dctcp_state,
.init = (void *)bpf_dctcp_init,
.set_state = (void *)bpf_dctcp_state,
.flags = TCP_CONG_NEEDS_ECN,
.name = "bpf_dctcp_nouse",
};
SEC(".struct_ops")
struct tcp_congestion_ops dctcp = {
.init = (void *)dctcp_init,
.in_ack_event = (void *)dctcp_update_alpha,
.cwnd_event = (void *)dctcp_cwnd_event,
.ssthresh = (void *)dctcp_ssthresh,
.cong_avoid = (void *)dctcp_cong_avoid,
.undo_cwnd = (void *)dctcp_cwnd_undo,
.set_state = (void *)dctcp_state,
.init = (void *)bpf_dctcp_init,
.in_ack_event = (void *)bpf_dctcp_update_alpha,
.cwnd_event = (void *)bpf_dctcp_cwnd_event,
.ssthresh = (void *)bpf_dctcp_ssthresh,
.cong_avoid = (void *)bpf_dctcp_cong_avoid,
.undo_cwnd = (void *)bpf_dctcp_cwnd_undo,
.set_state = (void *)bpf_dctcp_state,
.flags = TCP_CONG_NEEDS_ECN,
.name = "bpf_dctcp",
};

15
tools/testing/selftests/bpf/progs/bpf_misc.h
View File

@ -7,9 +7,9 @@
*
* The test_loader sequentially loads each program in a skeleton.
* Programs could be loaded in privileged and unprivileged modes.
* - __success, __failure, __msg imply privileged mode;
* - __success_unpriv, __failure_unpriv, __msg_unpriv imply
* unprivileged mode.
* - __success, __failure, __msg, __regex imply privileged mode;
* - __success_unpriv, __failure_unpriv, __msg_unpriv, __regex_unpriv
* imply unprivileged mode.
* If combination of privileged and unprivileged attributes is present
* both modes are used. If none are present privileged mode is implied.
*
@ -24,6 +24,9 @@
* Multiple __msg attributes could be specified.
* __msg_unpriv Same as __msg but for unprivileged mode.
*
* __regex Same as __msg, but using a regular expression.
* __regex_unpriv Same as __msg_unpriv but using a regular expression.
*
* __success Expect program load success in privileged mode.
* __success_unpriv Expect program load success in unprivileged mode.
*
@ -59,10 +62,12 @@
* __auxiliary_unpriv Same, but load program in unprivileged mode.
*/
#define __msg(msg) __attribute__((btf_decl_tag("comment:test_expect_msg=" msg)))
#define __regex(regex) __attribute__((btf_decl_tag("comment:test_expect_regex=" regex)))
#define __failure __attribute__((btf_decl_tag("comment:test_expect_failure")))
#define __success __attribute__((btf_decl_tag("comment:test_expect_success")))
#define __description(desc) __attribute__((btf_decl_tag("comment:test_description=" desc)))
#define __msg_unpriv(msg) __attribute__((btf_decl_tag("comment:test_expect_msg_unpriv=" msg)))
#define __regex_unpriv(regex) __attribute__((btf_decl_tag("comment:test_expect_regex_unpriv=" regex)))
#define __failure_unpriv __attribute__((btf_decl_tag("comment:test_expect_failure_unpriv")))
#define __success_unpriv __attribute__((btf_decl_tag("comment:test_expect_success_unpriv")))
#define __log_level(lvl) __attribute__((btf_decl_tag("comment:test_log_level="#lvl)))
@ -135,4 +140,8 @@
/* make it look to compiler like value is read and written */
#define __sink(expr) asm volatile("" : "+g"(expr))
#ifndef ARRAY_SIZE
#define ARRAY_SIZE(x) (sizeof(x) / sizeof((x)[0]))
#endif
#endif

10
tools/testing/selftests/bpf/progs/crypto_bench.c
View File

@ -57,7 +57,7 @@ int crypto_encrypt(struct __sk_buff *skb)
{
struct __crypto_ctx_value *v;
struct bpf_crypto_ctx *ctx;
struct bpf_dynptr psrc, pdst, iv;
struct bpf_dynptr psrc, pdst;
v = crypto_ctx_value_lookup();
if (!v) {
@ -73,9 +73,8 @@ int crypto_encrypt(struct __sk_buff *skb)
bpf_dynptr_from_skb(skb, 0, &psrc);
bpf_dynptr_from_mem(dst, len, 0, &pdst);
bpf_dynptr_from_mem(dst, 0, 0, &iv);
status = bpf_crypto_encrypt(ctx, &psrc, &pdst, &iv);
status = bpf_crypto_encrypt(ctx, &psrc, &pdst, NULL);
__sync_add_and_fetch(&hits, 1);
return 0;
@ -84,7 +83,7 @@ int crypto_encrypt(struct __sk_buff *skb)
SEC("tc")
int crypto_decrypt(struct __sk_buff *skb)
{
struct bpf_dynptr psrc, pdst, iv;
struct bpf_dynptr psrc, pdst;
struct __crypto_ctx_value *v;
struct bpf_crypto_ctx *ctx;
@ -98,9 +97,8 @@ int crypto_decrypt(struct __sk_buff *skb)
bpf_dynptr_from_skb(skb, 0, &psrc);
bpf_dynptr_from_mem(dst, len, 0, &pdst);
bpf_dynptr_from_mem(dst, 0, 0, &iv);
status = bpf_crypto_decrypt(ctx, &psrc, &pdst, &iv);
status = bpf_crypto_decrypt(ctx, &psrc, &pdst, NULL);
__sync_add_and_fetch(&hits, 1);
return 0;

16
tools/testing/selftests/bpf/progs/crypto_sanity.c
View File

@ -89,7 +89,7 @@ int decrypt_sanity(struct __sk_buff *skb)
{
struct __crypto_ctx_value *v;
struct bpf_crypto_ctx *ctx;
struct bpf_dynptr psrc, pdst, iv;
struct bpf_dynptr psrc, pdst;
int err;
err = skb_dynptr_validate(skb, &psrc);
@ -114,12 +114,8 @@ int decrypt_sanity(struct __sk_buff *skb)
* production code, a percpu map should be used to store the result.
*/
bpf_dynptr_from_mem(dst, sizeof(dst), 0, &pdst);
/* iv dynptr has to be initialized with 0 size, but proper memory region
* has to be provided anyway
*/
bpf_dynptr_from_mem(dst, 0, 0, &iv);
status = bpf_crypto_decrypt(ctx, &psrc, &pdst, &iv);
status = bpf_crypto_decrypt(ctx, &psrc, &pdst, NULL);
return TC_ACT_SHOT;
}
@ -129,7 +125,7 @@ int encrypt_sanity(struct __sk_buff *skb)
{
struct __crypto_ctx_value *v;
struct bpf_crypto_ctx *ctx;
struct bpf_dynptr psrc, pdst, iv;
struct bpf_dynptr psrc, pdst;
int err;
status = 0;
@ -156,12 +152,8 @@ int encrypt_sanity(struct __sk_buff *skb)
* production code, a percpu map should be used to store the result.
*/
bpf_dynptr_from_mem(dst, sizeof(dst), 0, &pdst);
/* iv dynptr has to be initialized with 0 size, but proper memory region
* has to be provided anyway
*/
bpf_dynptr_from_mem(dst, 0, 0, &iv);
status = bpf_crypto_encrypt(ctx, &psrc, &pdst, &iv);
status = bpf_crypto_encrypt(ctx, &psrc, &pdst, NULL);
return TC_ACT_SHOT;
}

30
tools/testing/selftests/bpf/progs/dynptr_fail.c
View File

@ -964,7 +964,7 @@ int dynptr_invalidate_slice_reinit(void *ctx)
* mem_or_null pointers.
*/
SEC("?raw_tp")
__failure __msg("R1 type=scalar expected=percpu_ptr_")
__failure __regex("R[0-9]+ type=scalar expected=percpu_ptr_")
int dynptr_invalidate_slice_or_null(void *ctx)
{
struct bpf_dynptr ptr;
@ -982,7 +982,7 @@ int dynptr_invalidate_slice_or_null(void *ctx)
/* Destruction of dynptr should also any slices obtained from it */
SEC("?raw_tp")
__failure __msg("R7 invalid mem access 'scalar'")
__failure __regex("R[0-9]+ invalid mem access 'scalar'")
int dynptr_invalidate_slice_failure(void *ctx)
{
struct bpf_dynptr ptr1;
@ -1069,7 +1069,7 @@ int dynptr_read_into_slot(void *ctx)
/* bpf_dynptr_slice()s are read-only and cannot be written to */
SEC("?tc")
__failure __msg("R0 cannot write into rdonly_mem")
__failure __regex("R[0-9]+ cannot write into rdonly_mem")
int skb_invalid_slice_write(struct __sk_buff *skb)
{
struct bpf_dynptr ptr;
@ -1686,3 +1686,27 @@ int test_dynptr_skb_small_buff(struct __sk_buff *skb)
return !!data;
}
__noinline long global_call_bpf_dynptr(const struct bpf_dynptr *dynptr)
{
long ret = 0;
/* Avoid leaving this global function empty to avoid having the compiler
* optimize away the call to this global function.
*/
__sink(ret);
return ret;
}
SEC("?raw_tp")
__failure __msg("arg#1 expected pointer to stack or const struct bpf_dynptr")
int test_dynptr_reg_type(void *ctx)
{
struct task_struct *current = NULL;
/* R1 should be holding a PTR_TO_BTF_ID, so this shouldn't be a
* reg->type that can be passed to a function accepting a
* ARG_PTR_TO_DYNPTR | MEM_RDONLY. process_dynptr_func() should catch
* this.
*/
global_call_bpf_dynptr((const struct bpf_dynptr *)current);
return 0;
}

7
tools/testing/selftests/bpf/progs/get_func_ip_test.c
View File

@ -5,13 +5,12 @@
char _license[] SEC("license") = "GPL";
extern const void bpf_fentry_test1 __ksym;
extern int bpf_fentry_test1(int a) __ksym;
extern int bpf_modify_return_test(int a, int *b) __ksym;
extern const void bpf_fentry_test2 __ksym;
extern const void bpf_fentry_test3 __ksym;
extern const void bpf_fentry_test4 __ksym;
extern const void bpf_modify_return_test __ksym;
extern const void bpf_fentry_test6 __ksym;
extern const void bpf_fentry_test7 __ksym;
extern bool CONFIG_X86_KERNEL_IBT __kconfig __weak;

10
tools/testing/selftests/bpf/progs/ip_check_defrag.c
View File

@ -12,7 +12,7 @@
#define IP_OFFSET 0x1FFF
#define NEXTHDR_FRAGMENT 44
extern int bpf_dynptr_from_skb(struct sk_buff *skb, __u64 flags,
extern int bpf_dynptr_from_skb(struct __sk_buff *skb, __u64 flags,
struct bpf_dynptr *ptr__uninit) __ksym;
extern void *bpf_dynptr_slice(const struct bpf_dynptr *ptr, uint32_t offset,
void *buffer, uint32_t buffer__sz) __ksym;
@ -42,7 +42,7 @@ static bool is_frag_v6(struct ipv6hdr *ip6h)
return ip6h->nexthdr == NEXTHDR_FRAGMENT;
}
static int handle_v4(struct sk_buff *skb)
static int handle_v4(struct __sk_buff *skb)
{
struct bpf_dynptr ptr;
u8 iph_buf[20] = {};
@ -64,7 +64,7 @@ static int handle_v4(struct sk_buff *skb)
return NF_ACCEPT;
}
static int handle_v6(struct sk_buff *skb)
static int handle_v6(struct __sk_buff *skb)
{
struct bpf_dynptr ptr;
struct ipv6hdr *ip6h;
@ -89,9 +89,9 @@ static int handle_v6(struct sk_buff *skb)
SEC("netfilter")
int defrag(struct bpf_nf_ctx *ctx)
{
struct sk_buff *skb = ctx->skb;
struct __sk_buff *skb = (struct __sk_buff *)ctx->skb;
switch (bpf_ntohs(skb->protocol)) {
switch (bpf_ntohs(ctx->skb->protocol)) {
case ETH_P_IP:
return handle_v4(skb);
case ETH_P_IPV6:

2
tools/testing/selftests/bpf/progs/iters.c
View File

@ -7,8 +7,6 @@
#include "bpf_misc.h"
#include "bpf_compiler.h"
#define ARRAY_SIZE(x) (int)(sizeof(x) / sizeof((x)[0]))
static volatile int zero = 0;
int my_pid;

37
tools/testing/selftests/bpf/progs/kfunc_call_test.c
View File

@ -177,4 +177,41 @@ int kfunc_call_test_static_unused_arg(struct __sk_buff *skb)
return actual != expected ? -1 : 0;
}
struct ctx_val {
struct bpf_testmod_ctx __kptr *ctx;
};
struct {
__uint(type, BPF_MAP_TYPE_ARRAY);
__uint(max_entries, 1);
__type(key, int);
__type(value, struct ctx_val);
} ctx_map SEC(".maps");
SEC("tc")
int kfunc_call_ctx(struct __sk_buff *skb)
{
struct bpf_testmod_ctx *ctx;
int err = 0;
ctx = bpf_testmod_ctx_create(&err);
if (!ctx && !err)
err = -1;
if (ctx) {
int key = 0;
struct ctx_val *ctx_val = bpf_map_lookup_elem(&ctx_map, &key);
/* Transfer ctx to map to be freed via implicit dtor call
* on cleanup.
*/
if (ctx_val)
ctx = bpf_kptr_xchg(&ctx_val->ctx, ctx);
if (ctx) {
bpf_testmod_ctx_release(ctx);
err = -1;
}
}
return err;
}
char _license[] SEC("license") = "GPL";

3
tools/testing/selftests/bpf/progs/kprobe_multi_session.c
View File

@ -4,8 +4,7 @@
#include <bpf/bpf_tracing.h>
#include <stdbool.h>
#include "bpf_kfuncs.h"
#define ARRAY_SIZE(x) (int)(sizeof(x) / sizeof((x)[0]))
#include "bpf_misc.h"
char _license[] SEC("license") = "GPL";

2
tools/testing/selftests/bpf/progs/kprobe_multi_session_cookie.c
View File

@ -25,7 +25,7 @@ int BPF_PROG(trigger)
static int check_cookie(__u64 val, __u64 *result)
{
long *cookie;
__u64 *cookie;
if (bpf_get_current_pid_tgid() >> 32 != pid)
return 1;

5
tools/testing/selftests/bpf/progs/linked_list.c
View File

@ -4,10 +4,7 @@
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_core_read.h>
#include "bpf_experimental.h"
#ifndef ARRAY_SIZE
#define ARRAY_SIZE(x) (int)(sizeof(x) / sizeof((x)[0]))
#endif
#include "bpf_misc.h"
#include "linked_list.h"

2
tools/testing/selftests/bpf/progs/map_percpu_stats.c
View File

@ -7,7 +7,7 @@
__u32 target_id;
__s64 bpf_map_sum_elem_count(struct bpf_map *map) __ksym;
__s64 bpf_map_sum_elem_count(const struct bpf_map *map) __ksym;
SEC("iter/bpf_map")
int dump_bpf_map(struct bpf_iter__bpf_map *ctx)

2
tools/testing/selftests/bpf/progs/nested_trust_common.h
View File

@ -7,6 +7,6 @@
#include <stdbool.h>
bool bpf_cpumask_test_cpu(unsigned int cpu, const struct cpumask *cpumask) __ksym;
bool bpf_cpumask_first_zero(const struct cpumask *cpumask) __ksym;
__u32 bpf_cpumask_first_zero(const struct cpumask *cpumask) __ksym;
#endif /* _NESTED_TRUST_COMMON_H */

5
tools/testing/selftests/bpf/progs/netif_receive_skb.c
View File

@ -5,6 +5,7 @@
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>
#include <bpf/bpf_core_read.h>
#include "bpf_misc.h"
#include <errno.h>
@ -23,10 +24,6 @@ bool skip = false;
#define BADPTR 0
#endif
#ifndef ARRAY_SIZE
#define ARRAY_SIZE(x) (sizeof(x) / sizeof((x)[0]))
#endif
struct {
__uint(type, BPF_MAP_TYPE_PERCPU_ARRAY);
__uint(max_entries, 1);

5
tools/testing/selftests/bpf/progs/profiler.inc.h
View File

@ -9,6 +9,7 @@
#include "err.h"
#include "bpf_experimental.h"
#include "bpf_compiler.h"
#include "bpf_misc.h"
#ifndef NULL
#define NULL 0
@ -133,10 +134,6 @@ struct {
__uint(max_entries, 16);
} disallowed_exec_inodes SEC(".maps");
#ifndef ARRAY_SIZE
#define ARRAY_SIZE(arr) (int)(sizeof(arr) / sizeof(arr[0]))
#endif
static INLINE bool IS_ERR(const void* ptr)
{
return IS_ERR_VALUE((unsigned long)ptr);

2
tools/testing/selftests/bpf/progs/rbtree_fail.c
View File

@ -105,7 +105,7 @@ long rbtree_api_remove_unadded_node(void *ctx)
}
SEC("?tc")
__failure __msg("Unreleased reference id=3 alloc_insn=10")
__failure __regex("Unreleased reference id=3 alloc_insn=[0-9]+")
long rbtree_api_remove_no_drop(void *ctx)
{
struct bpf_rb_node *res;

4
tools/testing/selftests/bpf/progs/refcounted_kptr_fail.c
View File

@ -32,7 +32,7 @@ static bool less(struct bpf_rb_node *a, const struct bpf_rb_node *b)
}
SEC("?tc")
__failure __msg("Unreleased reference id=4 alloc_insn=21")
__failure __regex("Unreleased reference id=4 alloc_insn=[0-9]+")
long rbtree_refcounted_node_ref_escapes(void *ctx)
{
struct node_acquire *n, *m;
@ -73,7 +73,7 @@ long refcount_acquire_maybe_null(void *ctx)
}
SEC("?tc")
__failure __msg("Unreleased reference id=3 alloc_insn=9")
__failure __regex("Unreleased reference id=3 alloc_insn=[0-9]+")
long rbtree_refcounted_node_ref_escapes_owning_input(void *ctx)
{
struct node_acquire *n, *m;

5
tools/testing/selftests/bpf/progs/setget_sockopt.c
View File

@ -6,10 +6,7 @@
#include <bpf/bpf_core_read.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>
#ifndef ARRAY_SIZE
#define ARRAY_SIZE(x) (sizeof(x) / sizeof((x)[0]))
#endif
#include "bpf_misc.h"
extern unsigned long CONFIG_HZ __kconfig;

11
tools/testing/selftests/bpf/progs/skb_pkt_end.c
View File

@ -33,6 +33,8 @@ int main_prog(struct __sk_buff *skb)
struct iphdr *ip = NULL;
struct tcphdr *tcp;
__u8 proto = 0;
int urg_ptr;
u32 offset;
if (!(ip = get_iphdr(skb)))
goto out;
@ -48,7 +50,14 @@ int main_prog(struct __sk_buff *skb)
if (!tcp)
goto out;
return tcp->urg_ptr;
urg_ptr = tcp->urg_ptr;
/* Checksum validation part */
proto++;
offset = sizeof(struct ethhdr) + offsetof(struct iphdr, protocol);
bpf_skb_store_bytes(skb, offset, &proto, sizeof(proto), BPF_F_RECOMPUTE_CSUM);
return urg_ptr;
out:
return -1;
}

4
tools/testing/selftests/bpf/progs/test_bpf_ma.c
View File

@ -7,10 +7,6 @@
#include "bpf_experimental.h"
#include "bpf_misc.h"
#ifndef ARRAY_SIZE
#define ARRAY_SIZE(x) (sizeof(x) / sizeof((x)[0]))
#endif
struct generic_map_value {
void *data;
};

1
tools/testing/selftests/bpf/progs/test_bpf_nf.c
View File

@ -1,4 +1,5 @@
// SPDX-License-Identifier: GPL-2.0
#define BPF_NO_KFUNC_PROTOTYPES
#include <vmlinux.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_endian.h>

Some files were not shown because too many files have changed in this diff Show More