Merge git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next

Signed-off-by: David S. Miller <davem@davemloft.net>
This commit is contained in:
David S. Miller 2020-03-30 19:52:37 -07:00
commit ed52f2c608
107 changed files with 6098 additions and 1740 deletions

View File

@ -0,0 +1,142 @@
.. SPDX-License-Identifier: GPL-2.0+
.. Copyright (C) 2020 Google LLC.
================
LSM BPF Programs
================
These BPF programs allow runtime instrumentation of the LSM hooks by privileged
users to implement system-wide MAC (Mandatory Access Control) and Audit
policies using eBPF.
Structure
---------
The example shows an eBPF program that can be attached to the ``file_mprotect``
LSM hook:
.. c:function:: int file_mprotect(struct vm_area_struct *vma, unsigned long reqprot, unsigned long prot);
Other LSM hooks which can be instrumented can be found in
``include/linux/lsm_hooks.h``.
eBPF programs that use :doc:`/bpf/btf` do not need to include kernel headers
for accessing information from the attached eBPF program's context. They can
simply declare the structures in the eBPF program and only specify the fields
that need to be accessed.
.. code-block:: c
struct mm_struct {
unsigned long start_brk, brk, start_stack;
} __attribute__((preserve_access_index));
struct vm_area_struct {
unsigned long start_brk, brk, start_stack;
unsigned long vm_start, vm_end;
struct mm_struct *vm_mm;
} __attribute__((preserve_access_index));
.. note:: The order of the fields is irrelevant.
This can be further simplified (if one has access to the BTF information at
build time) by generating the ``vmlinux.h`` with:
.. code-block:: console
# bpftool btf dump file <path-to-btf-vmlinux> format c > vmlinux.h
.. note:: ``path-to-btf-vmlinux`` can be ``/sys/kernel/btf/vmlinux`` if the
build environment matches the environment the BPF programs are
deployed in.
The ``vmlinux.h`` can then simply be included in the BPF programs without
requiring the definition of the types.
The eBPF programs can be declared using the``BPF_PROG``
macros defined in `tools/lib/bpf/bpf_tracing.h`_. In this
example:
* ``"lsm/file_mprotect"`` indicates the LSM hook that the program must
be attached to
* ``mprotect_audit`` is the name of the eBPF program
.. code-block:: c
SEC("lsm/file_mprotect")
int BPF_PROG(mprotect_audit, struct vm_area_struct *vma,
unsigned long reqprot, unsigned long prot, int ret)
{
/* ret is the return value from the previous BPF program
* or 0 if it's the first hook.
*/
if (ret != 0)
return ret;
int is_heap;
is_heap = (vma->vm_start >= vma->vm_mm->start_brk &&
vma->vm_end <= vma->vm_mm->brk);
/* Return an -EPERM or write information to the perf events buffer
* for auditing
*/
if (is_heap)
return -EPERM;
}
The ``__attribute__((preserve_access_index))`` is a clang feature that allows
the BPF verifier to update the offsets for the access at runtime using the
:doc:`/bpf/btf` information. Since the BPF verifier is aware of the types, it
also validates all the accesses made to the various types in the eBPF program.
Loading
-------
eBPF programs can be loaded with the :manpage:`bpf(2)` syscall's
``BPF_PROG_LOAD`` operation:
.. code-block:: c
struct bpf_object *obj;
obj = bpf_object__open("./my_prog.o");
bpf_object__load(obj);
This can be simplified by using a skeleton header generated by ``bpftool``:
.. code-block:: console
# bpftool gen skeleton my_prog.o > my_prog.skel.h
and the program can be loaded by including ``my_prog.skel.h`` and using
the generated helper, ``my_prog__open_and_load``.
Attachment to LSM Hooks
-----------------------
The LSM allows attachment of eBPF programs as LSM hooks using :manpage:`bpf(2)`
syscall's ``BPF_RAW_TRACEPOINT_OPEN`` operation or more simply by
using the libbpf helper ``bpf_program__attach_lsm``.
The program can be detached from the LSM hook by *destroying* the ``link``
link returned by ``bpf_program__attach_lsm`` using ``bpf_link__destroy``.
One can also use the helpers generated in ``my_prog.skel.h`` i.e.
``my_prog__attach`` for attachment and ``my_prog__destroy`` for cleaning up.
Examples
--------
An example eBPF program can be found in
`tools/testing/selftests/bpf/progs/lsm.c`_ and the corresponding
userspace code in `tools/testing/selftests/bpf/prog_tests/test_lsm.c`_
.. Links
.. _tools/lib/bpf/bpf_tracing.h:
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/tools/lib/bpf/bpf_tracing.h
.. _tools/testing/selftests/bpf/progs/lsm.c:
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/tools/testing/selftests/bpf/progs/lsm.c
.. _tools/testing/selftests/bpf/prog_tests/test_lsm.c:
https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/tools/testing/selftests/bpf/prog_tests/test_lsm.c

213
Documentation/bpf/drgn.rst Normal file
View File

@ -0,0 +1,213 @@
.. SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause)
==============
BPF drgn tools
==============
drgn scripts is a convenient and easy to use mechanism to retrieve arbitrary
kernel data structures. drgn is not relying on kernel UAPI to read the data.
Instead it's reading directly from ``/proc/kcore`` or vmcore and pretty prints
the data based on DWARF debug information from vmlinux.
This document describes BPF related drgn tools.
See `drgn/tools`_ for all tools available at the moment and `drgn/doc`_ for
more details on drgn itself.
bpf_inspect.py
--------------
Description
===========
`bpf_inspect.py`_ is a tool intended to inspect BPF programs and maps. It can
iterate over all programs and maps in the system and print basic information
about these objects, including id, type and name.
The main use-case `bpf_inspect.py`_ covers is to show BPF programs of types
``BPF_PROG_TYPE_EXT`` and ``BPF_PROG_TYPE_TRACING`` attached to other BPF
programs via ``freplace``/``fentry``/``fexit`` mechanisms, since there is no
user-space API to get this information.
Getting started
===============
List BPF programs (full names are obtained from BTF)::
% sudo bpf_inspect.py prog
27: BPF_PROG_TYPE_TRACEPOINT tracepoint__tcp__tcp_send_reset
4632: BPF_PROG_TYPE_CGROUP_SOCK_ADDR tw_ipt_bind
49464: BPF_PROG_TYPE_RAW_TRACEPOINT raw_tracepoint__sched_process_exit
List BPF maps::
% sudo bpf_inspect.py map
2577: BPF_MAP_TYPE_HASH tw_ipt_vips
4050: BPF_MAP_TYPE_STACK_TRACE stack_traces
4069: BPF_MAP_TYPE_PERCPU_ARRAY ned_dctcp_cntr
Find BPF programs attached to BPF program ``test_pkt_access``::
% sudo bpf_inspect.py p | grep test_pkt_access
650: BPF_PROG_TYPE_SCHED_CLS test_pkt_access
654: BPF_PROG_TYPE_TRACING test_main linked:[650->25: BPF_TRAMP_FEXIT test_pkt_access->test_pkt_access()]
655: BPF_PROG_TYPE_TRACING test_subprog1 linked:[650->29: BPF_TRAMP_FEXIT test_pkt_access->test_pkt_access_subprog1()]
656: BPF_PROG_TYPE_TRACING test_subprog2 linked:[650->31: BPF_TRAMP_FEXIT test_pkt_access->test_pkt_access_subprog2()]
657: BPF_PROG_TYPE_TRACING test_subprog3 linked:[650->21: BPF_TRAMP_FEXIT test_pkt_access->test_pkt_access_subprog3()]
658: BPF_PROG_TYPE_EXT new_get_skb_len linked:[650->16: BPF_TRAMP_REPLACE test_pkt_access->get_skb_len()]
659: BPF_PROG_TYPE_EXT new_get_skb_ifindex linked:[650->23: BPF_TRAMP_REPLACE test_pkt_access->get_skb_ifindex()]
660: BPF_PROG_TYPE_EXT new_get_constant linked:[650->19: BPF_TRAMP_REPLACE test_pkt_access->get_constant()]
It can be seen that there is a program ``test_pkt_access``, id 650 and there
are multiple other tracing and ext programs attached to functions in
``test_pkt_access``.
For example the line::
658: BPF_PROG_TYPE_EXT new_get_skb_len linked:[650->16: BPF_TRAMP_REPLACE test_pkt_access->get_skb_len()]
, means that BPF program id 658, type ``BPF_PROG_TYPE_EXT``, name
``new_get_skb_len`` replaces (``BPF_TRAMP_REPLACE``) function ``get_skb_len()``
that has BTF id 16 in BPF program id 650, name ``test_pkt_access``.
Getting help:
.. code-block:: none
% sudo bpf_inspect.py
usage: bpf_inspect.py [-h] {prog,p,map,m} ...
drgn script to list BPF programs or maps and their properties
unavailable via kernel API.
See https://github.com/osandov/drgn/ for more details on drgn.
optional arguments:
-h, --help show this help message and exit
subcommands:
{prog,p,map,m}
prog (p) list BPF programs
map (m) list BPF maps
Customization
=============
The script is intended to be customized by developers to print relevant
information about BPF programs, maps and other objects.
For example, to print ``struct bpf_prog_aux`` for BPF program id 53077:
.. code-block:: none
% git diff
diff --git a/tools/bpf_inspect.py b/tools/bpf_inspect.py
index 650e228..aea2357 100755
--- a/tools/bpf_inspect.py
+++ b/tools/bpf_inspect.py
@@ -112,7 +112,9 @@ def list_bpf_progs(args):
if linked:
linked = f" linked:[{linked}]"
- print(f"{id_:>6}: {type_:32} {name:32} {linked}")
+ if id_ == 53077:
+ print(f"{id_:>6}: {type_:32} {name:32}")
+ print(f"{bpf_prog.aux}")
def list_bpf_maps(args):
It produces the output::
% sudo bpf_inspect.py p
53077: BPF_PROG_TYPE_XDP tw_xdp_policer
*(struct bpf_prog_aux *)0xffff8893fad4b400 = {
.refcnt = (atomic64_t){
.counter = (long)58,
},
.used_map_cnt = (u32)1,
.max_ctx_offset = (u32)8,
.max_pkt_offset = (u32)15,
.max_tp_access = (u32)0,
.stack_depth = (u32)8,
.id = (u32)53077,
.func_cnt = (u32)0,
.func_idx = (u32)0,
.attach_btf_id = (u32)0,
.linked_prog = (struct bpf_prog *)0x0,
.verifier_zext = (bool)0,
.offload_requested = (bool)0,
.attach_btf_trace = (bool)0,
.func_proto_unreliable = (bool)0,
.trampoline_prog_type = (enum bpf_tramp_prog_type)BPF_TRAMP_FENTRY,
.trampoline = (struct bpf_trampoline *)0x0,
.tramp_hlist = (struct hlist_node){
.next = (struct hlist_node *)0x0,
.pprev = (struct hlist_node **)0x0,
},
.attach_func_proto = (const struct btf_type *)0x0,
.attach_func_name = (const char *)0x0,
.func = (struct bpf_prog **)0x0,
.jit_data = (void *)0x0,
.poke_tab = (struct bpf_jit_poke_descriptor *)0x0,
.size_poke_tab = (u32)0,
.ksym_tnode = (struct latch_tree_node){
.node = (struct rb_node [2]){
{
.__rb_parent_color = (unsigned long)18446612956263126665,
.rb_right = (struct rb_node *)0x0,
.rb_left = (struct rb_node *)0xffff88a0be3d0088,
},
{
.__rb_parent_color = (unsigned long)18446612956263126689,
.rb_right = (struct rb_node *)0x0,
.rb_left = (struct rb_node *)0xffff88a0be3d00a0,
},
},
},
.ksym_lnode = (struct list_head){
.next = (struct list_head *)0xffff88bf481830b8,
.prev = (struct list_head *)0xffff888309f536b8,
},
.ops = (const struct bpf_prog_ops *)xdp_prog_ops+0x0 = 0xffffffff820fa350,
.used_maps = (struct bpf_map **)0xffff889ff795de98,
.prog = (struct bpf_prog *)0xffffc9000cf2d000,
.user = (struct user_struct *)root_user+0x0 = 0xffffffff82444820,
.load_time = (u64)2408348759285319,
.cgroup_storage = (struct bpf_map *[2]){},
.name = (char [16])"tw_xdp_policer",
.security = (void *)0xffff889ff795d548,
.offload = (struct bpf_prog_offload *)0x0,
.btf = (struct btf *)0xffff8890ce6d0580,
.func_info = (struct bpf_func_info *)0xffff889ff795d240,
.func_info_aux = (struct bpf_func_info_aux *)0xffff889ff795de20,
.linfo = (struct bpf_line_info *)0xffff888a707afc00,
.jited_linfo = (void **)0xffff8893fad48600,
.func_info_cnt = (u32)1,
.nr_linfo = (u32)37,
.linfo_idx = (u32)0,
.num_exentries = (u32)0,
.extable = (struct exception_table_entry *)0xffffffffa032d950,
.stats = (struct bpf_prog_stats *)0x603fe3a1f6d0,
.work = (struct work_struct){
.data = (atomic_long_t){
.counter = (long)0,
},
.entry = (struct list_head){
.next = (struct list_head *)0x0,
.prev = (struct list_head *)0x0,
},
.func = (work_func_t)0x0,
},
.rcu = (struct callback_head){
.next = (struct callback_head *)0x0,
.func = (void (*)(struct callback_head *))0x0,
},
}
.. Links
.. _drgn/doc: https://drgn.readthedocs.io/en/latest/
.. _drgn/tools: https://github.com/osandov/drgn/tree/master/tools
.. _bpf_inspect.py:
https://github.com/osandov/drgn/blob/master/tools/bpf_inspect.py

View File

@ -45,14 +45,16 @@ Program types
prog_cgroup_sockopt
prog_cgroup_sysctl
prog_flow_dissector
bpf_lsm
Testing BPF
===========
Testing and debugging BPF
=========================
.. toctree::
:maxdepth: 1
drgn
s390

View File

@ -3147,6 +3147,8 @@ R: Martin KaFai Lau <kafai@fb.com>
R: Song Liu <songliubraving@fb.com>
R: Yonghong Song <yhs@fb.com>
R: Andrii Nakryiko <andriin@fb.com>
R: John Fastabend <john.fastabend@gmail.com>
R: KP Singh <kpsingh@chromium.org>
L: netdev@vger.kernel.org
L: bpf@vger.kernel.org
T: git git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf.git

View File

@ -303,12 +303,6 @@ SECTIONS
*(.branch_lt)
}
#ifdef CONFIG_DEBUG_INFO_BTF
.BTF : AT(ADDR(.BTF) - LOAD_OFFSET) {
*(.BTF)
}
#endif
.opd : AT(ADDR(.opd) - LOAD_OFFSET) {
__start_opd = .;
KEEP(*(.opd))

View File

@ -535,6 +535,7 @@
\
RO_EXCEPTION_TABLE \
NOTES \
BTF \
\
. = ALIGN((align)); \
__end_rodata = .;
@ -621,6 +622,20 @@
__stop___ex_table = .; \
}
/*
* .BTF
*/
#ifdef CONFIG_DEBUG_INFO_BTF
#define BTF \
.BTF : AT(ADDR(.BTF) - LOAD_OFFSET) { \
__start_BTF = .; \
*(.BTF) \
__stop_BTF = .; \
}
#else
#define BTF
#endif
/*
* Init task
*/

View File

@ -51,9 +51,18 @@ struct bpf_cgroup_storage {
struct rcu_head rcu;
};
struct bpf_cgroup_link {
struct bpf_link link;
struct cgroup *cgroup;
enum bpf_attach_type type;
};
extern const struct bpf_link_ops bpf_cgroup_link_lops;
struct bpf_prog_list {
struct list_head node;
struct bpf_prog *prog;
struct bpf_cgroup_link *link;
struct bpf_cgroup_storage *storage[MAX_BPF_CGROUP_STORAGE_TYPE];
};
@ -84,20 +93,27 @@ struct cgroup_bpf {
int cgroup_bpf_inherit(struct cgroup *cgrp);
void cgroup_bpf_offline(struct cgroup *cgrp);
int __cgroup_bpf_attach(struct cgroup *cgrp, struct bpf_prog *prog,
struct bpf_prog *replace_prog,
int __cgroup_bpf_attach(struct cgroup *cgrp,
struct bpf_prog *prog, struct bpf_prog *replace_prog,
struct bpf_cgroup_link *link,
enum bpf_attach_type type, u32 flags);
int __cgroup_bpf_detach(struct cgroup *cgrp, struct bpf_prog *prog,
struct bpf_cgroup_link *link,
enum bpf_attach_type type);
int __cgroup_bpf_replace(struct cgroup *cgrp, struct bpf_cgroup_link *link,
struct bpf_prog *new_prog);
int __cgroup_bpf_query(struct cgroup *cgrp, const union bpf_attr *attr,
union bpf_attr __user *uattr);
/* Wrapper for __cgroup_bpf_*() protected by cgroup_mutex */
int cgroup_bpf_attach(struct cgroup *cgrp, struct bpf_prog *prog,
struct bpf_prog *replace_prog, enum bpf_attach_type type,
int cgroup_bpf_attach(struct cgroup *cgrp,
struct bpf_prog *prog, struct bpf_prog *replace_prog,
struct bpf_cgroup_link *link, enum bpf_attach_type type,
u32 flags);
int cgroup_bpf_detach(struct cgroup *cgrp, struct bpf_prog *prog,
enum bpf_attach_type type, u32 flags);
enum bpf_attach_type type);
int cgroup_bpf_replace(struct bpf_link *link, struct bpf_prog *old_prog,
struct bpf_prog *new_prog);
int cgroup_bpf_query(struct cgroup *cgrp, const union bpf_attr *attr,
union bpf_attr __user *uattr);
@ -332,11 +348,13 @@ int cgroup_bpf_prog_attach(const union bpf_attr *attr,
enum bpf_prog_type ptype, struct bpf_prog *prog);
int cgroup_bpf_prog_detach(const union bpf_attr *attr,
enum bpf_prog_type ptype);
int cgroup_bpf_link_attach(const union bpf_attr *attr, struct bpf_prog *prog);
int cgroup_bpf_prog_query(const union bpf_attr *attr,
union bpf_attr __user *uattr);
#else
struct bpf_prog;
struct bpf_link;
struct cgroup_bpf {};
static inline int cgroup_bpf_inherit(struct cgroup *cgrp) { return 0; }
static inline void cgroup_bpf_offline(struct cgroup *cgrp) {}
@ -354,6 +372,19 @@ static inline int cgroup_bpf_prog_detach(const union bpf_attr *attr,
return -EINVAL;
}
static inline int cgroup_bpf_link_attach(const union bpf_attr *attr,
struct bpf_prog *prog)
{
return -EINVAL;
}
static inline int cgroup_bpf_replace(struct bpf_link *link,
struct bpf_prog *old_prog,
struct bpf_prog *new_prog)
{
return -EINVAL;
}
static inline int cgroup_bpf_prog_query(const union bpf_attr *attr,
union bpf_attr __user *uattr)
{

View File

@ -234,6 +234,7 @@ enum bpf_arg_type {
ARG_CONST_SIZE_OR_ZERO, /* number of bytes accessed from memory or 0 */
ARG_PTR_TO_CTX, /* pointer to context */
ARG_PTR_TO_CTX_OR_NULL, /* pointer to context or NULL */
ARG_ANYTHING, /* any (initialized) argument is ok */
ARG_PTR_TO_SPIN_LOCK, /* pointer to bpf_spin_lock */
ARG_PTR_TO_SOCK_COMMON, /* pointer to sock_common */
@ -1082,15 +1083,23 @@ extern int sysctl_unprivileged_bpf_disabled;
int bpf_map_new_fd(struct bpf_map *map, int flags);
int bpf_prog_new_fd(struct bpf_prog *prog);
struct bpf_link;
struct bpf_link {
atomic64_t refcnt;
const struct bpf_link_ops *ops;
struct bpf_prog *prog;
struct work_struct work;
};
struct bpf_link_ops {
void (*release)(struct bpf_link *link);
void (*dealloc)(struct bpf_link *link);
};
void bpf_link_init(struct bpf_link *link, const struct bpf_link_ops *ops,
struct bpf_prog *prog);
void bpf_link_cleanup(struct bpf_link *link, struct file *link_file,
int link_fd);
void bpf_link_inc(struct bpf_link *link);
void bpf_link_put(struct bpf_link *link);
int bpf_link_new_fd(struct bpf_link *link);
@ -1501,6 +1510,7 @@ extern const struct bpf_func_proto bpf_get_stack_proto;
extern const struct bpf_func_proto bpf_sock_map_update_proto;
extern const struct bpf_func_proto bpf_sock_hash_update_proto;
extern const struct bpf_func_proto bpf_get_current_cgroup_id_proto;
extern const struct bpf_func_proto bpf_get_current_ancestor_cgroup_id_proto;
extern const struct bpf_func_proto bpf_msg_redirect_hash_proto;
extern const struct bpf_func_proto bpf_msg_redirect_map_proto;
extern const struct bpf_func_proto bpf_sk_redirect_hash_proto;
@ -1514,6 +1524,9 @@ extern const struct bpf_func_proto bpf_tcp_sock_proto;
extern const struct bpf_func_proto bpf_jiffies64_proto;
extern const struct bpf_func_proto bpf_get_ns_current_pid_tgid_proto;
const struct bpf_func_proto *bpf_tracing_func_proto(
enum bpf_func_id func_id, const struct bpf_prog *prog);
/* Shared helpers among cBPF and eBPF. */
void bpf_user_rnd_init_once(void);
u64 bpf_user_rnd_u32(u64 r1, u64 r2, u64 r3, u64 r4, u64 r5);

33
include/linux/bpf_lsm.h Normal file
View File

@ -0,0 +1,33 @@
/* SPDX-License-Identifier: GPL-2.0 */
/*
* Copyright (C) 2020 Google LLC.
*/
#ifndef _LINUX_BPF_LSM_H
#define _LINUX_BPF_LSM_H
#include <linux/bpf.h>
#include <linux/lsm_hooks.h>
#ifdef CONFIG_BPF_LSM
#define LSM_HOOK(RET, DEFAULT, NAME, ...) \
RET bpf_lsm_##NAME(__VA_ARGS__);
#include <linux/lsm_hook_defs.h>
#undef LSM_HOOK
int bpf_lsm_verify_prog(struct bpf_verifier_log *vlog,
const struct bpf_prog *prog);
#else /* !CONFIG_BPF_LSM */
static inline int bpf_lsm_verify_prog(struct bpf_verifier_log *vlog,
const struct bpf_prog *prog)
{
return -EOPNOTSUPP;
}
#endif /* CONFIG_BPF_LSM */
#endif /* _LINUX_BPF_LSM_H */

View File

@ -70,6 +70,10 @@ BPF_PROG_TYPE(BPF_PROG_TYPE_STRUCT_OPS, bpf_struct_ops,
void *, void *)
BPF_PROG_TYPE(BPF_PROG_TYPE_EXT, bpf_extension,
void *, void *)
#ifdef CONFIG_BPF_LSM
BPF_PROG_TYPE(BPF_PROG_TYPE_LSM, lsm,
void *, void *)
#endif /* CONFIG_BPF_LSM */
#endif
BPF_MAP_TYPE(BPF_MAP_TYPE_ARRAY, array_map_ops)

View File

@ -123,6 +123,10 @@ struct bpf_reg_state {
s64 smax_value; /* maximum possible (s64)value */
u64 umin_value; /* minimum possible (u64)value */
u64 umax_value; /* maximum possible (u64)value */
s32 s32_min_value; /* minimum possible (s32)value */
s32 s32_max_value; /* maximum possible (s32)value */
u32 u32_min_value; /* minimum possible (u32)value */
u32 u32_max_value; /* maximum possible (u32)value */
/* parentage chain for liveness checking */
struct bpf_reg_state *parent;
/* Inside the callee two registers can be both PTR_TO_STACK like

View File

@ -27,6 +27,7 @@
#define S16_MAX ((s16)(U16_MAX >> 1))
#define S16_MIN ((s16)(-S16_MAX - 1))
#define U32_MAX ((u32)~0U)
#define U32_MIN ((u32)0)
#define S32_MAX ((s32)(U32_MAX >> 1))
#define S32_MIN ((s32)(-S32_MAX - 1))
#define U64_MAX ((u64)~0ULL)

View File

@ -0,0 +1,381 @@
/* SPDX-License-Identifier: GPL-2.0 */
/*
* Linux Security Module Hook declarations.
*
* Copyright (C) 2001 WireX Communications, Inc <chris@wirex.com>
* Copyright (C) 2001 Greg Kroah-Hartman <greg@kroah.com>
* Copyright (C) 2001 Networks Associates Technology, Inc <ssmalley@nai.com>
* Copyright (C) 2001 James Morris <jmorris@intercode.com.au>
* Copyright (C) 2001 Silicon Graphics, Inc. (Trust Technology Group)
* Copyright (C) 2015 Intel Corporation.
* Copyright (C) 2015 Casey Schaufler <casey@schaufler-ca.com>
* Copyright (C) 2016 Mellanox Techonologies
* Copyright (C) 2020 Google LLC.
*/
/*
* The macro LSM_HOOK is used to define the data structures required by the
* the LSM framework using the pattern:
*
* LSM_HOOK(<return_type>, <default_value>, <hook_name>, args...)
*
* struct security_hook_heads {
* #define LSM_HOOK(RET, DEFAULT, NAME, ...) struct hlist_head NAME;
* #include <linux/lsm_hook_defs.h>
* #undef LSM_HOOK
* };
*/
LSM_HOOK(int, 0, binder_set_context_mgr, struct task_struct *mgr)
LSM_HOOK(int, 0, binder_transaction, struct task_struct *from,
struct task_struct *to)
LSM_HOOK(int, 0, binder_transfer_binder, struct task_struct *from,
struct task_struct *to)
LSM_HOOK(int, 0, binder_transfer_file, struct task_struct *from,
struct task_struct *to, struct file *file)
LSM_HOOK(int, 0, ptrace_access_check, struct task_struct *child,
unsigned int mode)
LSM_HOOK(int, 0, ptrace_traceme, struct task_struct *parent)
LSM_HOOK(int, 0, capget, struct task_struct *target, kernel_cap_t *effective,
kernel_cap_t *inheritable, kernel_cap_t *permitted)
LSM_HOOK(int, 0, capset, struct cred *new, const struct cred *old,
const kernel_cap_t *effective, const kernel_cap_t *inheritable,
const kernel_cap_t *permitted)
LSM_HOOK(int, 0, capable, const struct cred *cred, struct user_namespace *ns,
int cap, unsigned int opts)
LSM_HOOK(int, 0, quotactl, int cmds, int type, int id, struct super_block *sb)
LSM_HOOK(int, 0, quota_on, struct dentry *dentry)
LSM_HOOK(int, 0, syslog, int type)
LSM_HOOK(int, 0, settime, const struct timespec64 *ts,
const struct timezone *tz)
LSM_HOOK(int, 0, vm_enough_memory, struct mm_struct *mm, long pages)
LSM_HOOK(int, 0, bprm_set_creds, struct linux_binprm *bprm)
LSM_HOOK(int, 0, bprm_check_security, struct linux_binprm *bprm)
LSM_HOOK(void, LSM_RET_VOID, bprm_committing_creds, struct linux_binprm *bprm)
LSM_HOOK(void, LSM_RET_VOID, bprm_committed_creds, struct linux_binprm *bprm)
LSM_HOOK(int, 0, fs_context_dup, struct fs_context *fc,
struct fs_context *src_sc)
LSM_HOOK(int, 0, fs_context_parse_param, struct fs_context *fc,
struct fs_parameter *param)
LSM_HOOK(int, 0, sb_alloc_security, struct super_block *sb)
LSM_HOOK(void, LSM_RET_VOID, sb_free_security, struct super_block *sb)
LSM_HOOK(void, LSM_RET_VOID, sb_free_mnt_opts, void *mnt_opts)
LSM_HOOK(int, 0, sb_eat_lsm_opts, char *orig, void **mnt_opts)
LSM_HOOK(int, 0, sb_remount, struct super_block *sb, void *mnt_opts)
LSM_HOOK(int, 0, sb_kern_mount, struct super_block *sb)
LSM_HOOK(int, 0, sb_show_options, struct seq_file *m, struct super_block *sb)
LSM_HOOK(int, 0, sb_statfs, struct dentry *dentry)
LSM_HOOK(int, 0, sb_mount, const char *dev_name, const struct path *path,
const char *type, unsigned long flags, void *data)
LSM_HOOK(int, 0, sb_umount, struct vfsmount *mnt, int flags)
LSM_HOOK(int, 0, sb_pivotroot, const struct path *old_path,
const struct path *new_path)
LSM_HOOK(int, 0, sb_set_mnt_opts, struct super_block *sb, void *mnt_opts,
unsigned long kern_flags, unsigned long *set_kern_flags)
LSM_HOOK(int, 0, sb_clone_mnt_opts, const struct super_block *oldsb,
struct super_block *newsb, unsigned long kern_flags,
unsigned long *set_kern_flags)
LSM_HOOK(int, 0, sb_add_mnt_opt, const char *option, const char *val,
int len, void **mnt_opts)
LSM_HOOK(int, 0, move_mount, const struct path *from_path,
const struct path *to_path)
LSM_HOOK(int, 0, dentry_init_security, struct dentry *dentry,
int mode, const struct qstr *name, void **ctx, u32 *ctxlen)
LSM_HOOK(int, 0, dentry_create_files_as, struct dentry *dentry, int mode,
struct qstr *name, const struct cred *old, struct cred *new)
#ifdef CONFIG_SECURITY_PATH
LSM_HOOK(int, 0, path_unlink, const struct path *dir, struct dentry *dentry)
LSM_HOOK(int, 0, path_mkdir, const struct path *dir, struct dentry *dentry,
umode_t mode)
LSM_HOOK(int, 0, path_rmdir, const struct path *dir, struct dentry *dentry)
LSM_HOOK(int, 0, path_mknod, const struct path *dir, struct dentry *dentry,
umode_t mode, unsigned int dev)
LSM_HOOK(int, 0, path_truncate, const struct path *path)
LSM_HOOK(int, 0, path_symlink, const struct path *dir, struct dentry *dentry,
const char *old_name)
LSM_HOOK(int, 0, path_link, struct dentry *old_dentry,
const struct path *new_dir, struct dentry *new_dentry)
LSM_HOOK(int, 0, path_rename, const struct path *old_dir,
struct dentry *old_dentry, const struct path *new_dir,
struct dentry *new_dentry)
LSM_HOOK(int, 0, path_chmod, const struct path *path, umode_t mode)
LSM_HOOK(int, 0, path_chown, const struct path *path, kuid_t uid, kgid_t gid)
LSM_HOOK(int, 0, path_chroot, const struct path *path)
#endif /* CONFIG_SECURITY_PATH */
/* Needed for inode based security check */
LSM_HOOK(int, 0, path_notify, const struct path *path, u64 mask,
unsigned int obj_type)
LSM_HOOK(int, 0, inode_alloc_security, struct inode *inode)
LSM_HOOK(void, LSM_RET_VOID, inode_free_security, struct inode *inode)
LSM_HOOK(int, 0, inode_init_security, struct inode *inode,
struct inode *dir, const struct qstr *qstr, const char **name,
void **value, size_t *len)
LSM_HOOK(int, 0, inode_create, struct inode *dir, struct dentry *dentry,
umode_t mode)
LSM_HOOK(int, 0, inode_link, struct dentry *old_dentry, struct inode *dir,
struct dentry *new_dentry)
LSM_HOOK(int, 0, inode_unlink, struct inode *dir, struct dentry *dentry)
LSM_HOOK(int, 0, inode_symlink, struct inode *dir, struct dentry *dentry,
const char *old_name)
LSM_HOOK(int, 0, inode_mkdir, struct inode *dir, struct dentry *dentry,
umode_t mode)
LSM_HOOK(int, 0, inode_rmdir, struct inode *dir, struct dentry *dentry)
LSM_HOOK(int, 0, inode_mknod, struct inode *dir, struct dentry *dentry,
umode_t mode, dev_t dev)
LSM_HOOK(int, 0, inode_rename, struct inode *old_dir, struct dentry *old_dentry,
struct inode *new_dir, struct dentry *new_dentry)
LSM_HOOK(int, 0, inode_readlink, struct dentry *dentry)
LSM_HOOK(int, 0, inode_follow_link, struct dentry *dentry, struct inode *inode,
bool rcu)
LSM_HOOK(int, 0, inode_permission, struct inode *inode, int mask)
LSM_HOOK(int, 0, inode_setattr, struct dentry *dentry, struct iattr *attr)
LSM_HOOK(int, 0, inode_getattr, const struct path *path)
LSM_HOOK(int, 0, inode_setxattr, struct dentry *dentry, const char *name,
const void *value, size_t size, int flags)
LSM_HOOK(void, LSM_RET_VOID, inode_post_setxattr, struct dentry *dentry,
const char *name, const void *value, size_t size, int flags)
LSM_HOOK(int, 0, inode_getxattr, struct dentry *dentry, const char *name)
LSM_HOOK(int, 0, inode_listxattr, struct dentry *dentry)
LSM_HOOK(int, 0, inode_removexattr, struct dentry *dentry, const char *name)
LSM_HOOK(int, 0, inode_need_killpriv, struct dentry *dentry)
LSM_HOOK(int, 0, inode_killpriv, struct dentry *dentry)
LSM_HOOK(int, -EOPNOTSUPP, inode_getsecurity, struct inode *inode,
const char *name, void **buffer, bool alloc)
LSM_HOOK(int, -EOPNOTSUPP, inode_setsecurity, struct inode *inode,
const char *name, const void *value, size_t size, int flags)
LSM_HOOK(int, 0, inode_listsecurity, struct inode *inode, char *buffer,
size_t buffer_size)
LSM_HOOK(void, LSM_RET_VOID, inode_getsecid, struct inode *inode, u32 *secid)
LSM_HOOK(int, 0, inode_copy_up, struct dentry *src, struct cred **new)
LSM_HOOK(int, 0, inode_copy_up_xattr, const char *name)
LSM_HOOK(int, 0, kernfs_init_security, struct kernfs_node *kn_dir,
struct kernfs_node *kn)
LSM_HOOK(int, 0, file_permission, struct file *file, int mask)
LSM_HOOK(int, 0, file_alloc_security, struct file *file)
LSM_HOOK(void, LSM_RET_VOID, file_free_security, struct file *file)
LSM_HOOK(int, 0, file_ioctl, struct file *file, unsigned int cmd,
unsigned long arg)
LSM_HOOK(int, 0, mmap_addr, unsigned long addr)
LSM_HOOK(int, 0, mmap_file, struct file *file, unsigned long reqprot,
unsigned long prot, unsigned long flags)
LSM_HOOK(int, 0, file_mprotect, struct vm_area_struct *vma,
unsigned long reqprot, unsigned long prot)
LSM_HOOK(int, 0, file_lock, struct file *file, unsigned int cmd)
LSM_HOOK(int, 0, file_fcntl, struct file *file, unsigned int cmd,
unsigned long arg)
LSM_HOOK(void, LSM_RET_VOID, file_set_fowner, struct file *file)
LSM_HOOK(int, 0, file_send_sigiotask, struct task_struct *tsk,
struct fown_struct *fown, int sig)
LSM_HOOK(int, 0, file_receive, struct file *file)
LSM_HOOK(int, 0, file_open, struct file *file)
LSM_HOOK(int, 0, task_alloc, struct task_struct *task,
unsigned long clone_flags)
LSM_HOOK(void, LSM_RET_VOID, task_free, struct task_struct *task)
LSM_HOOK(int, 0, cred_alloc_blank, struct cred *cred, gfp_t gfp)
LSM_HOOK(void, LSM_RET_VOID, cred_free, struct cred *cred)
LSM_HOOK(int, 0, cred_prepare, struct cred *new, const struct cred *old,
gfp_t gfp)
LSM_HOOK(void, LSM_RET_VOID, cred_transfer, struct cred *new,
const struct cred *old)
LSM_HOOK(void, LSM_RET_VOID, cred_getsecid, const struct cred *c, u32 *secid)
LSM_HOOK(int, 0, kernel_act_as, struct cred *new, u32 secid)
LSM_HOOK(int, 0, kernel_create_files_as, struct cred *new, struct inode *inode)
LSM_HOOK(int, 0, kernel_module_request, char *kmod_name)
LSM_HOOK(int, 0, kernel_load_data, enum kernel_load_data_id id)
LSM_HOOK(int, 0, kernel_read_file, struct file *file,
enum kernel_read_file_id id)
LSM_HOOK(int, 0, kernel_post_read_file, struct file *file, char *buf,
loff_t size, enum kernel_read_file_id id)
LSM_HOOK(int, 0, task_fix_setuid, struct cred *new, const struct cred *old,
int flags)
LSM_HOOK(int, 0, task_setpgid, struct task_struct *p, pid_t pgid)
LSM_HOOK(int, 0, task_getpgid, struct task_struct *p)
LSM_HOOK(int, 0, task_getsid, struct task_struct *p)
LSM_HOOK(void, LSM_RET_VOID, task_getsecid, struct task_struct *p, u32 *secid)
LSM_HOOK(int, 0, task_setnice, struct task_struct *p, int nice)
LSM_HOOK(int, 0, task_setioprio, struct task_struct *p, int ioprio)
LSM_HOOK(int, 0, task_getioprio, struct task_struct *p)
LSM_HOOK(int, 0, task_prlimit, const struct cred *cred,
const struct cred *tcred, unsigned int flags)
LSM_HOOK(int, 0, task_setrlimit, struct task_struct *p, unsigned int resource,
struct rlimit *new_rlim)
LSM_HOOK(int, 0, task_setscheduler, struct task_struct *p)
LSM_HOOK(int, 0, task_getscheduler, struct task_struct *p)
LSM_HOOK(int, 0, task_movememory, struct task_struct *p)
LSM_HOOK(int, 0, task_kill, struct task_struct *p, struct kernel_siginfo *info,
int sig, const struct cred *cred)
LSM_HOOK(int, -ENOSYS, task_prctl, int option, unsigned long arg2,
unsigned long arg3, unsigned long arg4, unsigned long arg5)
LSM_HOOK(void, LSM_RET_VOID, task_to_inode, struct task_struct *p,
struct inode *inode)
LSM_HOOK(int, 0, ipc_permission, struct kern_ipc_perm *ipcp, short flag)
LSM_HOOK(void, LSM_RET_VOID, ipc_getsecid, struct kern_ipc_perm *ipcp,
u32 *secid)
LSM_HOOK(int, 0, msg_msg_alloc_security, struct msg_msg *msg)
LSM_HOOK(void, LSM_RET_VOID, msg_msg_free_security, struct msg_msg *msg)
LSM_HOOK(int, 0, msg_queue_alloc_security, struct kern_ipc_perm *perm)
LSM_HOOK(void, LSM_RET_VOID, msg_queue_free_security,
struct kern_ipc_perm *perm)
LSM_HOOK(int, 0, msg_queue_associate, struct kern_ipc_perm *perm, int msqflg)
LSM_HOOK(int, 0, msg_queue_msgctl, struct kern_ipc_perm *perm, int cmd)
LSM_HOOK(int, 0, msg_queue_msgsnd, struct kern_ipc_perm *perm,
struct msg_msg *msg, int msqflg)
LSM_HOOK(int, 0, msg_queue_msgrcv, struct kern_ipc_perm *perm,
struct msg_msg *msg, struct task_struct *target, long type, int mode)
LSM_HOOK(int, 0, shm_alloc_security, struct kern_ipc_perm *perm)
LSM_HOOK(void, LSM_RET_VOID, shm_free_security, struct kern_ipc_perm *perm)
LSM_HOOK(int, 0, shm_associate, struct kern_ipc_perm *perm, int shmflg)
LSM_HOOK(int, 0, shm_shmctl, struct kern_ipc_perm *perm, int cmd)
LSM_HOOK(int, 0, shm_shmat, struct kern_ipc_perm *perm, char __user *shmaddr,
int shmflg)
LSM_HOOK(int, 0, sem_alloc_security, struct kern_ipc_perm *perm)
LSM_HOOK(void, LSM_RET_VOID, sem_free_security, struct kern_ipc_perm *perm)
LSM_HOOK(int, 0, sem_associate, struct kern_ipc_perm *perm, int semflg)
LSM_HOOK(int, 0, sem_semctl, struct kern_ipc_perm *perm, int cmd)
LSM_HOOK(int, 0, sem_semop, struct kern_ipc_perm *perm, struct sembuf *sops,
unsigned nsops, int alter)
LSM_HOOK(int, 0, netlink_send, struct sock *sk, struct sk_buff *skb)
LSM_HOOK(void, LSM_RET_VOID, d_instantiate, struct dentry *dentry,
struct inode *inode)
LSM_HOOK(int, -EINVAL, getprocattr, struct task_struct *p, char *name,
char **value)
LSM_HOOK(int, -EINVAL, setprocattr, const char *name, void *value, size_t size)
LSM_HOOK(int, 0, ismaclabel, const char *name)
LSM_HOOK(int, 0, secid_to_secctx, u32 secid, char **secdata,
u32 *seclen)
LSM_HOOK(int, 0, secctx_to_secid, const char *secdata, u32 seclen, u32 *secid)
LSM_HOOK(void, LSM_RET_VOID, release_secctx, char *secdata, u32 seclen)
LSM_HOOK(void, LSM_RET_VOID, inode_invalidate_secctx, struct inode *inode)
LSM_HOOK(int, 0, inode_notifysecctx, struct inode *inode, void *ctx, u32 ctxlen)
LSM_HOOK(int, 0, inode_setsecctx, struct dentry *dentry, void *ctx, u32 ctxlen)
LSM_HOOK(int, 0, inode_getsecctx, struct inode *inode, void **ctx,
u32 *ctxlen)
#ifdef CONFIG_SECURITY_NETWORK
LSM_HOOK(int, 0, unix_stream_connect, struct sock *sock, struct sock *other,
struct sock *newsk)
LSM_HOOK(int, 0, unix_may_send, struct socket *sock, struct socket *other)
LSM_HOOK(int, 0, socket_create, int family, int type, int protocol, int kern)
LSM_HOOK(int, 0, socket_post_create, struct socket *sock, int family, int type,
int protocol, int kern)
LSM_HOOK(int, 0, socket_socketpair, struct socket *socka, struct socket *sockb)
LSM_HOOK(int, 0, socket_bind, struct socket *sock, struct sockaddr *address,
int addrlen)
LSM_HOOK(int, 0, socket_connect, struct socket *sock, struct sockaddr *address,
int addrlen)
LSM_HOOK(int, 0, socket_listen, struct socket *sock, int backlog)
LSM_HOOK(int, 0, socket_accept, struct socket *sock, struct socket *newsock)
LSM_HOOK(int, 0, socket_sendmsg, struct socket *sock, struct msghdr *msg,
int size)
LSM_HOOK(int, 0, socket_recvmsg, struct socket *sock, struct msghdr *msg,
int size, int flags)
LSM_HOOK(int, 0, socket_getsockname, struct socket *sock)
LSM_HOOK(int, 0, socket_getpeername, struct socket *sock)
LSM_HOOK(int, 0, socket_getsockopt, struct socket *sock, int level, int optname)
LSM_HOOK(int, 0, socket_setsockopt, struct socket *sock, int level, int optname)
LSM_HOOK(int, 0, socket_shutdown, struct socket *sock, int how)
LSM_HOOK(int, 0, socket_sock_rcv_skb, struct sock *sk, struct sk_buff *skb)
LSM_HOOK(int, 0, socket_getpeersec_stream, struct socket *sock,
char __user *optval, int __user *optlen, unsigned len)
LSM_HOOK(int, 0, socket_getpeersec_dgram, struct socket *sock,
struct sk_buff *skb, u32 *secid)
LSM_HOOK(int, 0, sk_alloc_security, struct sock *sk, int family, gfp_t priority)
LSM_HOOK(void, LSM_RET_VOID, sk_free_security, struct sock *sk)
LSM_HOOK(void, LSM_RET_VOID, sk_clone_security, const struct sock *sk,
struct sock *newsk)
LSM_HOOK(void, LSM_RET_VOID, sk_getsecid, struct sock *sk, u32 *secid)
LSM_HOOK(void, LSM_RET_VOID, sock_graft, struct sock *sk, struct socket *parent)
LSM_HOOK(int, 0, inet_conn_request, struct sock *sk, struct sk_buff *skb,
struct request_sock *req)
LSM_HOOK(void, LSM_RET_VOID, inet_csk_clone, struct sock *newsk,
const struct request_sock *req)
LSM_HOOK(void, LSM_RET_VOID, inet_conn_established, struct sock *sk,
struct sk_buff *skb)
LSM_HOOK(int, 0, secmark_relabel_packet, u32 secid)
LSM_HOOK(void, LSM_RET_VOID, secmark_refcount_inc, void)
LSM_HOOK(void, LSM_RET_VOID, secmark_refcount_dec, void)
LSM_HOOK(void, LSM_RET_VOID, req_classify_flow, const struct request_sock *req,
struct flowi *fl)
LSM_HOOK(int, 0, tun_dev_alloc_security, void **security)
LSM_HOOK(void, LSM_RET_VOID, tun_dev_free_security, void *security)
LSM_HOOK(int, 0, tun_dev_create, void)
LSM_HOOK(int, 0, tun_dev_attach_queue, void *security)
LSM_HOOK(int, 0, tun_dev_attach, struct sock *sk, void *security)
LSM_HOOK(int, 0, tun_dev_open, void *security)
LSM_HOOK(int, 0, sctp_assoc_request, struct sctp_endpoint *ep,
struct sk_buff *skb)
LSM_HOOK(int, 0, sctp_bind_connect, struct sock *sk, int optname,
struct sockaddr *address, int addrlen)
LSM_HOOK(void, LSM_RET_VOID, sctp_sk_clone, struct sctp_endpoint *ep,
struct sock *sk, struct sock *newsk)
#endif /* CONFIG_SECURITY_NETWORK */
#ifdef CONFIG_SECURITY_INFINIBAND
LSM_HOOK(int, 0, ib_pkey_access, void *sec, u64 subnet_prefix, u16 pkey)
LSM_HOOK(int, 0, ib_endport_manage_subnet, void *sec, const char *dev_name,
u8 port_num)
LSM_HOOK(int, 0, ib_alloc_security, void **sec)
LSM_HOOK(void, LSM_RET_VOID, ib_free_security, void *sec)
#endif /* CONFIG_SECURITY_INFINIBAND */
#ifdef CONFIG_SECURITY_NETWORK_XFRM
LSM_HOOK(int, 0, xfrm_policy_alloc_security, struct xfrm_sec_ctx **ctxp,
struct xfrm_user_sec_ctx *sec_ctx, gfp_t gfp)
LSM_HOOK(int, 0, xfrm_policy_clone_security, struct xfrm_sec_ctx *old_ctx,
struct xfrm_sec_ctx **new_ctx)
LSM_HOOK(void, LSM_RET_VOID, xfrm_policy_free_security,
struct xfrm_sec_ctx *ctx)
LSM_HOOK(int, 0, xfrm_policy_delete_security, struct xfrm_sec_ctx *ctx)
LSM_HOOK(int, 0, xfrm_state_alloc, struct xfrm_state *x,
struct xfrm_user_sec_ctx *sec_ctx)
LSM_HOOK(int, 0, xfrm_state_alloc_acquire, struct xfrm_state *x,
struct xfrm_sec_ctx *polsec, u32 secid)
LSM_HOOK(void, LSM_RET_VOID, xfrm_state_free_security, struct xfrm_state *x)
LSM_HOOK(int, 0, xfrm_state_delete_security, struct xfrm_state *x)
LSM_HOOK(int, 0, xfrm_policy_lookup, struct xfrm_sec_ctx *ctx, u32 fl_secid,
u8 dir)
LSM_HOOK(int, 1, xfrm_state_pol_flow_match, struct xfrm_state *x,
struct xfrm_policy *xp, const struct flowi *fl)
LSM_HOOK(int, 0, xfrm_decode_session, struct sk_buff *skb, u32 *secid,
int ckall)
#endif /* CONFIG_SECURITY_NETWORK_XFRM */
/* key management security hooks */
#ifdef CONFIG_KEYS
LSM_HOOK(int, 0, key_alloc, struct key *key, const struct cred *cred,
unsigned long flags)
LSM_HOOK(void, LSM_RET_VOID, key_free, struct key *key)
LSM_HOOK(int, 0, key_permission, key_ref_t key_ref, const struct cred *cred,
unsigned perm)
LSM_HOOK(int, 0, key_getsecurity, struct key *key, char **_buffer)
#endif /* CONFIG_KEYS */
#ifdef CONFIG_AUDIT
LSM_HOOK(int, 0, audit_rule_init, u32 field, u32 op, char *rulestr,
void **lsmrule)
LSM_HOOK(int, 0, audit_rule_known, struct audit_krule *krule)
LSM_HOOK(int, 0, audit_rule_match, u32 secid, u32 field, u32 op, void *lsmrule)
LSM_HOOK(void, LSM_RET_VOID, audit_rule_free, void *lsmrule)
#endif /* CONFIG_AUDIT */
#ifdef CONFIG_BPF_SYSCALL
LSM_HOOK(int, 0, bpf, int cmd, union bpf_attr *attr, unsigned int size)
LSM_HOOK(int, 0, bpf_map, struct bpf_map *map, fmode_t fmode)
LSM_HOOK(int, 0, bpf_prog, struct bpf_prog *prog)
LSM_HOOK(int, 0, bpf_map_alloc_security, struct bpf_map *map)
LSM_HOOK(void, LSM_RET_VOID, bpf_map_free_security, struct bpf_map *map)
LSM_HOOK(int, 0, bpf_prog_alloc_security, struct bpf_prog_aux *aux)
LSM_HOOK(void, LSM_RET_VOID, bpf_prog_free_security, struct bpf_prog_aux *aux)
#endif /* CONFIG_BPF_SYSCALL */
LSM_HOOK(int, 0, locked_down, enum lockdown_reason what)
#ifdef CONFIG_PERF_EVENTS
LSM_HOOK(int, 0, perf_event_open, struct perf_event_attr *attr, int type)
LSM_HOOK(int, 0, perf_event_alloc, struct perf_event *event)
LSM_HOOK(void, LSM_RET_VOID, perf_event_free, struct perf_event *event)
LSM_HOOK(int, 0, perf_event_read, struct perf_event *event)
LSM_HOOK(int, 0, perf_event_write, struct perf_event *event)
#endif /* CONFIG_PERF_EVENTS */

View File

@ -1456,625 +1456,15 @@
* @what: kernel feature being accessed
*/
union security_list_options {
int (*binder_set_context_mgr)(struct task_struct *mgr);
int (*binder_transaction)(struct task_struct *from,
struct task_struct *to);
int (*binder_transfer_binder)(struct task_struct *from,
struct task_struct *to);
int (*binder_transfer_file)(struct task_struct *from,
struct task_struct *to,
struct file *file);
int (*ptrace_access_check)(struct task_struct *child,
unsigned int mode);
int (*ptrace_traceme)(struct task_struct *parent);
int (*capget)(struct task_struct *target, kernel_cap_t *effective,
kernel_cap_t *inheritable, kernel_cap_t *permitted);
int (*capset)(struct cred *new, const struct cred *old,
const kernel_cap_t *effective,
const kernel_cap_t *inheritable,
const kernel_cap_t *permitted);
int (*capable)(const struct cred *cred,
struct user_namespace *ns,
int cap,
unsigned int opts);
int (*quotactl)(int cmds, int type, int id, struct super_block *sb);
int (*quota_on)(struct dentry *dentry);
int (*syslog)(int type);
int (*settime)(const struct timespec64 *ts, const struct timezone *tz);
int (*vm_enough_memory)(struct mm_struct *mm, long pages);
int (*bprm_set_creds)(struct linux_binprm *bprm);
int (*bprm_check_security)(struct linux_binprm *bprm);
void (*bprm_committing_creds)(struct linux_binprm *bprm);
void (*bprm_committed_creds)(struct linux_binprm *bprm);
int (*fs_context_dup)(struct fs_context *fc, struct fs_context *src_sc);
int (*fs_context_parse_param)(struct fs_context *fc, struct fs_parameter *param);
int (*sb_alloc_security)(struct super_block *sb);
void (*sb_free_security)(struct super_block *sb);
void (*sb_free_mnt_opts)(void *mnt_opts);
int (*sb_eat_lsm_opts)(char *orig, void **mnt_opts);
int (*sb_remount)(struct super_block *sb, void *mnt_opts);
int (*sb_kern_mount)(struct super_block *sb);
int (*sb_show_options)(struct seq_file *m, struct super_block *sb);
int (*sb_statfs)(struct dentry *dentry);
int (*sb_mount)(const char *dev_name, const struct path *path,
const char *type, unsigned long flags, void *data);
int (*sb_umount)(struct vfsmount *mnt, int flags);
int (*sb_pivotroot)(const struct path *old_path, const struct path *new_path);
int (*sb_set_mnt_opts)(struct super_block *sb,
void *mnt_opts,
unsigned long kern_flags,
unsigned long *set_kern_flags);
int (*sb_clone_mnt_opts)(const struct super_block *oldsb,
struct super_block *newsb,
unsigned long kern_flags,
unsigned long *set_kern_flags);
int (*sb_add_mnt_opt)(const char *option, const char *val, int len,
void **mnt_opts);
int (*move_mount)(const struct path *from_path, const struct path *to_path);
int (*dentry_init_security)(struct dentry *dentry, int mode,
const struct qstr *name, void **ctx,
u32 *ctxlen);
int (*dentry_create_files_as)(struct dentry *dentry, int mode,
struct qstr *name,
const struct cred *old,
struct cred *new);
#ifdef CONFIG_SECURITY_PATH
int (*path_unlink)(const struct path *dir, struct dentry *dentry);
int (*path_mkdir)(const struct path *dir, struct dentry *dentry,
umode_t mode);
int (*path_rmdir)(const struct path *dir, struct dentry *dentry);
int (*path_mknod)(const struct path *dir, struct dentry *dentry,
umode_t mode, unsigned int dev);
int (*path_truncate)(const struct path *path);
int (*path_symlink)(const struct path *dir, struct dentry *dentry,
const char *old_name);
int (*path_link)(struct dentry *old_dentry, const struct path *new_dir,
struct dentry *new_dentry);
int (*path_rename)(const struct path *old_dir, struct dentry *old_dentry,
const struct path *new_dir,
struct dentry *new_dentry);
int (*path_chmod)(const struct path *path, umode_t mode);
int (*path_chown)(const struct path *path, kuid_t uid, kgid_t gid);
int (*path_chroot)(const struct path *path);
#endif
/* Needed for inode based security check */
int (*path_notify)(const struct path *path, u64 mask,
unsigned int obj_type);
int (*inode_alloc_security)(struct inode *inode);
void (*inode_free_security)(struct inode *inode);
int (*inode_init_security)(struct inode *inode, struct inode *dir,
const struct qstr *qstr,
const char **name, void **value,
size_t *len);
int (*inode_create)(struct inode *dir, struct dentry *dentry,
umode_t mode);
int (*inode_link)(struct dentry *old_dentry, struct inode *dir,
struct dentry *new_dentry);
int (*inode_unlink)(struct inode *dir, struct dentry *dentry);
int (*inode_symlink)(struct inode *dir, struct dentry *dentry,
const char *old_name);
int (*inode_mkdir)(struct inode *dir, struct dentry *dentry,
umode_t mode);
int (*inode_rmdir)(struct inode *dir, struct dentry *dentry);
int (*inode_mknod)(struct inode *dir, struct dentry *dentry,
umode_t mode, dev_t dev);
int (*inode_rename)(struct inode *old_dir, struct dentry *old_dentry,
struct inode *new_dir,
struct dentry *new_dentry);
int (*inode_readlink)(struct dentry *dentry);
int (*inode_follow_link)(struct dentry *dentry, struct inode *inode,
bool rcu);
int (*inode_permission)(struct inode *inode, int mask);
int (*inode_setattr)(struct dentry *dentry, struct iattr *attr);
int (*inode_getattr)(const struct path *path);
int (*inode_setxattr)(struct dentry *dentry, const char *name,
const void *value, size_t size, int flags);
void (*inode_post_setxattr)(struct dentry *dentry, const char *name,
const void *value, size_t size,
int flags);
int (*inode_getxattr)(struct dentry *dentry, const char *name);
int (*inode_listxattr)(struct dentry *dentry);
int (*inode_removexattr)(struct dentry *dentry, const char *name);
int (*inode_need_killpriv)(struct dentry *dentry);
int (*inode_killpriv)(struct dentry *dentry);
int (*inode_getsecurity)(struct inode *inode, const char *name,
void **buffer, bool alloc);
int (*inode_setsecurity)(struct inode *inode, const char *name,
const void *value, size_t size,
int flags);
int (*inode_listsecurity)(struct inode *inode, char *buffer,
size_t buffer_size);
void (*inode_getsecid)(struct inode *inode, u32 *secid);
int (*inode_copy_up)(struct dentry *src, struct cred **new);
int (*inode_copy_up_xattr)(const char *name);
int (*kernfs_init_security)(struct kernfs_node *kn_dir,
struct kernfs_node *kn);
int (*file_permission)(struct file *file, int mask);
int (*file_alloc_security)(struct file *file);
void (*file_free_security)(struct file *file);
int (*file_ioctl)(struct file *file, unsigned int cmd,
unsigned long arg);
int (*mmap_addr)(unsigned long addr);
int (*mmap_file)(struct file *file, unsigned long reqprot,
unsigned long prot, unsigned long flags);
int (*file_mprotect)(struct vm_area_struct *vma, unsigned long reqprot,
unsigned long prot);
int (*file_lock)(struct file *file, unsigned int cmd);
int (*file_fcntl)(struct file *file, unsigned int cmd,
unsigned long arg);
void (*file_set_fowner)(struct file *file);
int (*file_send_sigiotask)(struct task_struct *tsk,
struct fown_struct *fown, int sig);
int (*file_receive)(struct file *file);
int (*file_open)(struct file *file);
int (*task_alloc)(struct task_struct *task, unsigned long clone_flags);
void (*task_free)(struct task_struct *task);
int (*cred_alloc_blank)(struct cred *cred, gfp_t gfp);
void (*cred_free)(struct cred *cred);
int (*cred_prepare)(struct cred *new, const struct cred *old,
gfp_t gfp);
void (*cred_transfer)(struct cred *new, const struct cred *old);
void (*cred_getsecid)(const struct cred *c, u32 *secid);
int (*kernel_act_as)(struct cred *new, u32 secid);
int (*kernel_create_files_as)(struct cred *new, struct inode *inode);
int (*kernel_module_request)(char *kmod_name);
int (*kernel_load_data)(enum kernel_load_data_id id);
int (*kernel_read_file)(struct file *file, enum kernel_read_file_id id);
int (*kernel_post_read_file)(struct file *file, char *buf, loff_t size,
enum kernel_read_file_id id);
int (*task_fix_setuid)(struct cred *new, const struct cred *old,
int flags);
int (*task_setpgid)(struct task_struct *p, pid_t pgid);
int (*task_getpgid)(struct task_struct *p);
int (*task_getsid)(struct task_struct *p);
void (*task_getsecid)(struct task_struct *p, u32 *secid);
int (*task_setnice)(struct task_struct *p, int nice);
int (*task_setioprio)(struct task_struct *p, int ioprio);
int (*task_getioprio)(struct task_struct *p);
int (*task_prlimit)(const struct cred *cred, const struct cred *tcred,
unsigned int flags);
int (*task_setrlimit)(struct task_struct *p, unsigned int resource,
struct rlimit *new_rlim);
int (*task_setscheduler)(struct task_struct *p);
int (*task_getscheduler)(struct task_struct *p);
int (*task_movememory)(struct task_struct *p);
int (*task_kill)(struct task_struct *p, struct kernel_siginfo *info,
int sig, const struct cred *cred);
int (*task_prctl)(int option, unsigned long arg2, unsigned long arg3,
unsigned long arg4, unsigned long arg5);
void (*task_to_inode)(struct task_struct *p, struct inode *inode);
int (*ipc_permission)(struct kern_ipc_perm *ipcp, short flag);
void (*ipc_getsecid)(struct kern_ipc_perm *ipcp, u32 *secid);
int (*msg_msg_alloc_security)(struct msg_msg *msg);
void (*msg_msg_free_security)(struct msg_msg *msg);
int (*msg_queue_alloc_security)(struct kern_ipc_perm *perm);
void (*msg_queue_free_security)(struct kern_ipc_perm *perm);
int (*msg_queue_associate)(struct kern_ipc_perm *perm, int msqflg);
int (*msg_queue_msgctl)(struct kern_ipc_perm *perm, int cmd);
int (*msg_queue_msgsnd)(struct kern_ipc_perm *perm, struct msg_msg *msg,
int msqflg);
int (*msg_queue_msgrcv)(struct kern_ipc_perm *perm, struct msg_msg *msg,
struct task_struct *target, long type,
int mode);
int (*shm_alloc_security)(struct kern_ipc_perm *perm);
void (*shm_free_security)(struct kern_ipc_perm *perm);
int (*shm_associate)(struct kern_ipc_perm *perm, int shmflg);
int (*shm_shmctl)(struct kern_ipc_perm *perm, int cmd);
int (*shm_shmat)(struct kern_ipc_perm *perm, char __user *shmaddr,
int shmflg);
int (*sem_alloc_security)(struct kern_ipc_perm *perm);
void (*sem_free_security)(struct kern_ipc_perm *perm);
int (*sem_associate)(struct kern_ipc_perm *perm, int semflg);
int (*sem_semctl)(struct kern_ipc_perm *perm, int cmd);
int (*sem_semop)(struct kern_ipc_perm *perm, struct sembuf *sops,
unsigned nsops, int alter);
int (*netlink_send)(struct sock *sk, struct sk_buff *skb);
void (*d_instantiate)(struct dentry *dentry, struct inode *inode);
int (*getprocattr)(struct task_struct *p, char *name, char **value);
int (*setprocattr)(const char *name, void *value, size_t size);
int (*ismaclabel)(const char *name);
int (*secid_to_secctx)(u32 secid, char **secdata, u32 *seclen);
int (*secctx_to_secid)(const char *secdata, u32 seclen, u32 *secid);
void (*release_secctx)(char *secdata, u32 seclen);
void (*inode_invalidate_secctx)(struct inode *inode);
int (*inode_notifysecctx)(struct inode *inode, void *ctx, u32 ctxlen);
int (*inode_setsecctx)(struct dentry *dentry, void *ctx, u32 ctxlen);
int (*inode_getsecctx)(struct inode *inode, void **ctx, u32 *ctxlen);
#ifdef CONFIG_SECURITY_NETWORK
int (*unix_stream_connect)(struct sock *sock, struct sock *other,
struct sock *newsk);
int (*unix_may_send)(struct socket *sock, struct socket *other);
int (*socket_create)(int family, int type, int protocol, int kern);
int (*socket_post_create)(struct socket *sock, int family, int type,
int protocol, int kern);
int (*socket_socketpair)(struct socket *socka, struct socket *sockb);
int (*socket_bind)(struct socket *sock, struct sockaddr *address,
int addrlen);
int (*socket_connect)(struct socket *sock, struct sockaddr *address,
int addrlen);
int (*socket_listen)(struct socket *sock, int backlog);
int (*socket_accept)(struct socket *sock, struct socket *newsock);
int (*socket_sendmsg)(struct socket *sock, struct msghdr *msg,
int size);
int (*socket_recvmsg)(struct socket *sock, struct msghdr *msg,
int size, int flags);
int (*socket_getsockname)(struct socket *sock);
int (*socket_getpeername)(struct socket *sock);
int (*socket_getsockopt)(struct socket *sock, int level, int optname);
int (*socket_setsockopt)(struct socket *sock, int level, int optname);
int (*socket_shutdown)(struct socket *sock, int how);
int (*socket_sock_rcv_skb)(struct sock *sk, struct sk_buff *skb);
int (*socket_getpeersec_stream)(struct socket *sock,
char __user *optval,
int __user *optlen, unsigned len);
int (*socket_getpeersec_dgram)(struct socket *sock,
struct sk_buff *skb, u32 *secid);
int (*sk_alloc_security)(struct sock *sk, int family, gfp_t priority);
void (*sk_free_security)(struct sock *sk);
void (*sk_clone_security)(const struct sock *sk, struct sock *newsk);
void (*sk_getsecid)(struct sock *sk, u32 *secid);
void (*sock_graft)(struct sock *sk, struct socket *parent);
int (*inet_conn_request)(struct sock *sk, struct sk_buff *skb,
struct request_sock *req);
void (*inet_csk_clone)(struct sock *newsk,
const struct request_sock *req);
void (*inet_conn_established)(struct sock *sk, struct sk_buff *skb);
int (*secmark_relabel_packet)(u32 secid);
void (*secmark_refcount_inc)(void);
void (*secmark_refcount_dec)(void);
void (*req_classify_flow)(const struct request_sock *req,
struct flowi *fl);
int (*tun_dev_alloc_security)(void **security);
void (*tun_dev_free_security)(void *security);
int (*tun_dev_create)(void);
int (*tun_dev_attach_queue)(void *security);
int (*tun_dev_attach)(struct sock *sk, void *security);
int (*tun_dev_open)(void *security);
int (*sctp_assoc_request)(struct sctp_endpoint *ep,
struct sk_buff *skb);
int (*sctp_bind_connect)(struct sock *sk, int optname,
struct sockaddr *address, int addrlen);
void (*sctp_sk_clone)(struct sctp_endpoint *ep, struct sock *sk,
struct sock *newsk);
#endif /* CONFIG_SECURITY_NETWORK */
#ifdef CONFIG_SECURITY_INFINIBAND
int (*ib_pkey_access)(void *sec, u64 subnet_prefix, u16 pkey);
int (*ib_endport_manage_subnet)(void *sec, const char *dev_name,
u8 port_num);
int (*ib_alloc_security)(void **sec);
void (*ib_free_security)(void *sec);
#endif /* CONFIG_SECURITY_INFINIBAND */
#ifdef CONFIG_SECURITY_NETWORK_XFRM
int (*xfrm_policy_alloc_security)(struct xfrm_sec_ctx **ctxp,
struct xfrm_user_sec_ctx *sec_ctx,
gfp_t gfp);
int (*xfrm_policy_clone_security)(struct xfrm_sec_ctx *old_ctx,
struct xfrm_sec_ctx **new_ctx);
void (*xfrm_policy_free_security)(struct xfrm_sec_ctx *ctx);
int (*xfrm_policy_delete_security)(struct xfrm_sec_ctx *ctx);
int (*xfrm_state_alloc)(struct xfrm_state *x,
struct xfrm_user_sec_ctx *sec_ctx);
int (*xfrm_state_alloc_acquire)(struct xfrm_state *x,
struct xfrm_sec_ctx *polsec,
u32 secid);
void (*xfrm_state_free_security)(struct xfrm_state *x);
int (*xfrm_state_delete_security)(struct xfrm_state *x);
int (*xfrm_policy_lookup)(struct xfrm_sec_ctx *ctx, u32 fl_secid,
u8 dir);
int (*xfrm_state_pol_flow_match)(struct xfrm_state *x,
struct xfrm_policy *xp,
const struct flowi *fl);
int (*xfrm_decode_session)(struct sk_buff *skb, u32 *secid, int ckall);
#endif /* CONFIG_SECURITY_NETWORK_XFRM */
/* key management security hooks */
#ifdef CONFIG_KEYS
int (*key_alloc)(struct key *key, const struct cred *cred,
unsigned long flags);
void (*key_free)(struct key *key);
int (*key_permission)(key_ref_t key_ref, const struct cred *cred,
unsigned perm);
int (*key_getsecurity)(struct key *key, char **_buffer);
#endif /* CONFIG_KEYS */
#ifdef CONFIG_AUDIT
int (*audit_rule_init)(u32 field, u32 op, char *rulestr,
void **lsmrule);
int (*audit_rule_known)(struct audit_krule *krule);
int (*audit_rule_match)(u32 secid, u32 field, u32 op, void *lsmrule);
void (*audit_rule_free)(void *lsmrule);
#endif /* CONFIG_AUDIT */
#ifdef CONFIG_BPF_SYSCALL
int (*bpf)(int cmd, union bpf_attr *attr,
unsigned int size);
int (*bpf_map)(struct bpf_map *map, fmode_t fmode);
int (*bpf_prog)(struct bpf_prog *prog);
int (*bpf_map_alloc_security)(struct bpf_map *map);
void (*bpf_map_free_security)(struct bpf_map *map);
int (*bpf_prog_alloc_security)(struct bpf_prog_aux *aux);
void (*bpf_prog_free_security)(struct bpf_prog_aux *aux);
#endif /* CONFIG_BPF_SYSCALL */
int (*locked_down)(enum lockdown_reason what);
#ifdef CONFIG_PERF_EVENTS
int (*perf_event_open)(struct perf_event_attr *attr, int type);
int (*perf_event_alloc)(struct perf_event *event);
void (*perf_event_free)(struct perf_event *event);
int (*perf_event_read)(struct perf_event *event);
int (*perf_event_write)(struct perf_event *event);
#endif
#define LSM_HOOK(RET, DEFAULT, NAME, ...) RET (*NAME)(__VA_ARGS__);
#include "lsm_hook_defs.h"
#undef LSM_HOOK
};
struct security_hook_heads {
struct hlist_head binder_set_context_mgr;
struct hlist_head binder_transaction;
struct hlist_head binder_transfer_binder;
struct hlist_head binder_transfer_file;
struct hlist_head ptrace_access_check;
struct hlist_head ptrace_traceme;
struct hlist_head capget;
struct hlist_head capset;
struct hlist_head capable;
struct hlist_head quotactl;
struct hlist_head quota_on;
struct hlist_head syslog;
struct hlist_head settime;
struct hlist_head vm_enough_memory;
struct hlist_head bprm_set_creds;
struct hlist_head bprm_check_security;
struct hlist_head bprm_committing_creds;
struct hlist_head bprm_committed_creds;
struct hlist_head fs_context_dup;
struct hlist_head fs_context_parse_param;
struct hlist_head sb_alloc_security;
struct hlist_head sb_free_security;
struct hlist_head sb_free_mnt_opts;
struct hlist_head sb_eat_lsm_opts;
struct hlist_head sb_remount;
struct hlist_head sb_kern_mount;
struct hlist_head sb_show_options;
struct hlist_head sb_statfs;
struct hlist_head sb_mount;
struct hlist_head sb_umount;
struct hlist_head sb_pivotroot;
struct hlist_head sb_set_mnt_opts;
struct hlist_head sb_clone_mnt_opts;
struct hlist_head sb_add_mnt_opt;
struct hlist_head move_mount;
struct hlist_head dentry_init_security;
struct hlist_head dentry_create_files_as;
#ifdef CONFIG_SECURITY_PATH
struct hlist_head path_unlink;
struct hlist_head path_mkdir;
struct hlist_head path_rmdir;
struct hlist_head path_mknod;
struct hlist_head path_truncate;
struct hlist_head path_symlink;
struct hlist_head path_link;
struct hlist_head path_rename;
struct hlist_head path_chmod;
struct hlist_head path_chown;
struct hlist_head path_chroot;
#endif
/* Needed for inode based modules as well */
struct hlist_head path_notify;
struct hlist_head inode_alloc_security;
struct hlist_head inode_free_security;
struct hlist_head inode_init_security;
struct hlist_head inode_create;
struct hlist_head inode_link;
struct hlist_head inode_unlink;
struct hlist_head inode_symlink;
struct hlist_head inode_mkdir;
struct hlist_head inode_rmdir;
struct hlist_head inode_mknod;
struct hlist_head inode_rename;
struct hlist_head inode_readlink;
struct hlist_head inode_follow_link;
struct hlist_head inode_permission;
struct hlist_head inode_setattr;
struct hlist_head inode_getattr;
struct hlist_head inode_setxattr;
struct hlist_head inode_post_setxattr;
struct hlist_head inode_getxattr;
struct hlist_head inode_listxattr;
struct hlist_head inode_removexattr;
struct hlist_head inode_need_killpriv;
struct hlist_head inode_killpriv;
struct hlist_head inode_getsecurity;
struct hlist_head inode_setsecurity;
struct hlist_head inode_listsecurity;
struct hlist_head inode_getsecid;
struct hlist_head inode_copy_up;
struct hlist_head inode_copy_up_xattr;
struct hlist_head kernfs_init_security;
struct hlist_head file_permission;
struct hlist_head file_alloc_security;
struct hlist_head file_free_security;
struct hlist_head file_ioctl;
struct hlist_head mmap_addr;
struct hlist_head mmap_file;
struct hlist_head file_mprotect;
struct hlist_head file_lock;
struct hlist_head file_fcntl;
struct hlist_head file_set_fowner;
struct hlist_head file_send_sigiotask;
struct hlist_head file_receive;
struct hlist_head file_open;
struct hlist_head task_alloc;
struct hlist_head task_free;
struct hlist_head cred_alloc_blank;
struct hlist_head cred_free;
struct hlist_head cred_prepare;
struct hlist_head cred_transfer;
struct hlist_head cred_getsecid;
struct hlist_head kernel_act_as;
struct hlist_head kernel_create_files_as;
struct hlist_head kernel_load_data;
struct hlist_head kernel_read_file;
struct hlist_head kernel_post_read_file;
struct hlist_head kernel_module_request;
struct hlist_head task_fix_setuid;
struct hlist_head task_setpgid;
struct hlist_head task_getpgid;
struct hlist_head task_getsid;
struct hlist_head task_getsecid;
struct hlist_head task_setnice;
struct hlist_head task_setioprio;
struct hlist_head task_getioprio;
struct hlist_head task_prlimit;
struct hlist_head task_setrlimit;
struct hlist_head task_setscheduler;
struct hlist_head task_getscheduler;
struct hlist_head task_movememory;
struct hlist_head task_kill;
struct hlist_head task_prctl;
struct hlist_head task_to_inode;
struct hlist_head ipc_permission;
struct hlist_head ipc_getsecid;
struct hlist_head msg_msg_alloc_security;
struct hlist_head msg_msg_free_security;
struct hlist_head msg_queue_alloc_security;
struct hlist_head msg_queue_free_security;
struct hlist_head msg_queue_associate;
struct hlist_head msg_queue_msgctl;
struct hlist_head msg_queue_msgsnd;
struct hlist_head msg_queue_msgrcv;
struct hlist_head shm_alloc_security;
struct hlist_head shm_free_security;
struct hlist_head shm_associate;
struct hlist_head shm_shmctl;
struct hlist_head shm_shmat;
struct hlist_head sem_alloc_security;
struct hlist_head sem_free_security;
struct hlist_head sem_associate;
struct hlist_head sem_semctl;
struct hlist_head sem_semop;
struct hlist_head netlink_send;
struct hlist_head d_instantiate;
struct hlist_head getprocattr;
struct hlist_head setprocattr;
struct hlist_head ismaclabel;
struct hlist_head secid_to_secctx;
struct hlist_head secctx_to_secid;
struct hlist_head release_secctx;
struct hlist_head inode_invalidate_secctx;
struct hlist_head inode_notifysecctx;
struct hlist_head inode_setsecctx;
struct hlist_head inode_getsecctx;
#ifdef CONFIG_SECURITY_NETWORK
struct hlist_head unix_stream_connect;
struct hlist_head unix_may_send;
struct hlist_head socket_create;
struct hlist_head socket_post_create;
struct hlist_head socket_socketpair;
struct hlist_head socket_bind;
struct hlist_head socket_connect;
struct hlist_head socket_listen;
struct hlist_head socket_accept;
struct hlist_head socket_sendmsg;
struct hlist_head socket_recvmsg;
struct hlist_head socket_getsockname;
struct hlist_head socket_getpeername;
struct hlist_head socket_getsockopt;
struct hlist_head socket_setsockopt;
struct hlist_head socket_shutdown;
struct hlist_head socket_sock_rcv_skb;
struct hlist_head socket_getpeersec_stream;
struct hlist_head socket_getpeersec_dgram;
struct hlist_head sk_alloc_security;
struct hlist_head sk_free_security;
struct hlist_head sk_clone_security;
struct hlist_head sk_getsecid;
struct hlist_head sock_graft;
struct hlist_head inet_conn_request;
struct hlist_head inet_csk_clone;
struct hlist_head inet_conn_established;
struct hlist_head secmark_relabel_packet;
struct hlist_head secmark_refcount_inc;
struct hlist_head secmark_refcount_dec;
struct hlist_head req_classify_flow;
struct hlist_head tun_dev_alloc_security;
struct hlist_head tun_dev_free_security;
struct hlist_head tun_dev_create;
struct hlist_head tun_dev_attach_queue;
struct hlist_head tun_dev_attach;
struct hlist_head tun_dev_open;
struct hlist_head sctp_assoc_request;
struct hlist_head sctp_bind_connect;
struct hlist_head sctp_sk_clone;
#endif /* CONFIG_SECURITY_NETWORK */
#ifdef CONFIG_SECURITY_INFINIBAND
struct hlist_head ib_pkey_access;
struct hlist_head ib_endport_manage_subnet;
struct hlist_head ib_alloc_security;
struct hlist_head ib_free_security;
#endif /* CONFIG_SECURITY_INFINIBAND */
#ifdef CONFIG_SECURITY_NETWORK_XFRM
struct hlist_head xfrm_policy_alloc_security;
struct hlist_head xfrm_policy_clone_security;
struct hlist_head xfrm_policy_free_security;
struct hlist_head xfrm_policy_delete_security;
struct hlist_head xfrm_state_alloc;
struct hlist_head xfrm_state_alloc_acquire;
struct hlist_head xfrm_state_free_security;
struct hlist_head xfrm_state_delete_security;
struct hlist_head xfrm_policy_lookup;
struct hlist_head xfrm_state_pol_flow_match;
struct hlist_head xfrm_decode_session;
#endif /* CONFIG_SECURITY_NETWORK_XFRM */
#ifdef CONFIG_KEYS
struct hlist_head key_alloc;
struct hlist_head key_free;
struct hlist_head key_permission;
struct hlist_head key_getsecurity;
#endif /* CONFIG_KEYS */
#ifdef CONFIG_AUDIT
struct hlist_head audit_rule_init;
struct hlist_head audit_rule_known;
struct hlist_head audit_rule_match;
struct hlist_head audit_rule_free;
#endif /* CONFIG_AUDIT */
#ifdef CONFIG_BPF_SYSCALL
struct hlist_head bpf;
struct hlist_head bpf_map;
struct hlist_head bpf_prog;
struct hlist_head bpf_map_alloc_security;
struct hlist_head bpf_map_free_security;
struct hlist_head bpf_prog_alloc_security;
struct hlist_head bpf_prog_free_security;
#endif /* CONFIG_BPF_SYSCALL */
struct hlist_head locked_down;
#ifdef CONFIG_PERF_EVENTS
struct hlist_head perf_event_open;
struct hlist_head perf_event_alloc;
struct hlist_head perf_event_free;
struct hlist_head perf_event_read;
struct hlist_head perf_event_write;
#endif
#define LSM_HOOK(RET, DEFAULT, NAME, ...) struct hlist_head NAME;
#include "lsm_hook_defs.h"
#undef LSM_HOOK
} __randomize_layout;
/*
@ -2100,6 +1490,12 @@ struct lsm_blob_sizes {
int lbs_task;
};
/*
* LSM_RET_VOID is used as the default value in LSM_HOOK definitions for void
* LSM hooks (in include/linux/lsm_hook_defs.h).
*/
#define LSM_RET_VOID ((void) 0)
/*
* Initializing a security_hook_list structure takes
* up a lot of space in a source file. This macro takes

View File

@ -3777,7 +3777,7 @@ struct sk_buff *dev_hard_start_xmit(struct sk_buff *skb, struct net_device *dev,
typedef int (*bpf_op_t)(struct net_device *dev, struct netdev_bpf *bpf);
int dev_change_xdp_fd(struct net_device *dev, struct netlink_ext_ack *extack,
int fd, u32 flags);
int fd, int expected_fd, u32 flags);
u32 __dev_xdp_query(struct net_device *dev, bpf_op_t xdp_op,
enum bpf_netdev_command cmd);
int xdp_umem_query(struct net_device *dev, u16 queue_id);

View File

@ -86,4 +86,16 @@ int tnum_strn(char *str, size_t size, struct tnum a);
/* Format a tnum as tristate binary expansion */
int tnum_sbin(char *str, size_t size, struct tnum a);
/* Returns the 32-bit subreg */
struct tnum tnum_subreg(struct tnum a);
/* Returns the tnum with the lower 32-bit subreg cleared */
struct tnum tnum_clear_subreg(struct tnum a);
/* Returns the tnum with the lower 32-bit subreg set to value */
struct tnum tnum_const_subreg(struct tnum a, u32 value);
/* Returns true if 32-bit subreg @a is a known constant*/
static inline bool tnum_subreg_is_const(struct tnum a)
{
return !(tnum_subreg(a)).mask;
}
#endif /* _LINUX_TNUM_H */

View File

@ -45,9 +45,14 @@ static inline void sock_update_classid(struct sock_cgroup_data *skcd)
sock_cgroup_set_classid(skcd, classid);
}
static inline u32 __task_get_classid(struct task_struct *task)
{
return task_cls_state(task)->classid;
}
static inline u32 task_get_classid(const struct sk_buff *skb)
{
u32 classid = task_cls_state(current)->classid;
u32 classid = __task_get_classid(current);
/* Due to the nature of the classifier it is required to ignore all
* packets originating from softirq context as accessing `current'

View File

@ -85,9 +85,8 @@ static inline struct sock *__inet6_lookup_skb(struct inet_hashinfo *hashinfo,
int iif, int sdif,
bool *refcounted)
{
struct sock *sk = skb_steal_sock(skb);
struct sock *sk = skb_steal_sock(skb, refcounted);
*refcounted = true;
if (sk)
return sk;

View File

@ -379,10 +379,9 @@ static inline struct sock *__inet_lookup_skb(struct inet_hashinfo *hashinfo,
const int sdif,
bool *refcounted)
{
struct sock *sk = skb_steal_sock(skb);
struct sock *sk = skb_steal_sock(skb, refcounted);
const struct iphdr *iph = ip_hdr(skb);
*refcounted = true;
if (sk)
return sk;

View File

@ -168,6 +168,9 @@ struct net {
#ifdef CONFIG_XFRM
struct netns_xfrm xfrm;
#endif
atomic64_t net_cookie; /* written once */
#if IS_ENABLED(CONFIG_IP_VS)
struct netns_ipvs *ipvs;
#endif
@ -225,6 +228,8 @@ extern struct list_head net_namespace_list;
struct net *get_net_ns_by_pid(pid_t pid);
struct net *get_net_ns_by_fd(int fd);
u64 net_gen_cookie(struct net *net);
#ifdef CONFIG_SYSCTL
void ipx_register_sysctl(void);
void ipx_unregister_sysctl(void);

View File

@ -1659,6 +1659,7 @@ void sock_rfree(struct sk_buff *skb);
void sock_efree(struct sk_buff *skb);
#ifdef CONFIG_INET
void sock_edemux(struct sk_buff *skb);
void sock_pfree(struct sk_buff *skb);
#else
#define sock_edemux sock_efree
#endif
@ -2526,16 +2527,14 @@ void sock_net_set(struct sock *sk, struct net *net)
write_pnet(&sk->sk_net, net);
}
static inline struct sock *skb_steal_sock(struct sk_buff *skb)
static inline bool
skb_sk_is_prefetched(struct sk_buff *skb)
{
if (skb->sk) {
struct sock *sk = skb->sk;
skb->destructor = NULL;
skb->sk = NULL;
return sk;
}
return NULL;
#ifdef CONFIG_INET
return skb->destructor == sock_pfree;
#else
return false;
#endif /* CONFIG_INET */
}
/* This helper checks if a socket is a full socket,
@ -2546,6 +2545,35 @@ static inline bool sk_fullsock(const struct sock *sk)
return (1 << sk->sk_state) & ~(TCPF_TIME_WAIT | TCPF_NEW_SYN_RECV);
}
static inline bool
sk_is_refcounted(struct sock *sk)
{
/* Only full sockets have sk->sk_flags. */
return !sk_fullsock(sk) || !sock_flag(sk, SOCK_RCU_FREE);
}
/**
* skb_steal_sock
* @skb to steal the socket from
* @refcounted is set to true if the socket is reference-counted
*/
static inline struct sock *
skb_steal_sock(struct sk_buff *skb, bool *refcounted)
{
if (skb->sk) {
struct sock *sk = skb->sk;
*refcounted = true;
if (skb_sk_is_prefetched(skb))
*refcounted = sk_is_refcounted(sk);
skb->destructor = NULL;
skb->sk = NULL;
return sk;
}
*refcounted = false;
return NULL;
}
/* Checks if this SKB belongs to an HW offloaded socket
* and whether any SW fallbacks are required based on dev.
* Check decrypted mark in case skb_orphan() cleared socket.

View File

@ -2207,8 +2207,6 @@ static inline void tcp_bpf_clone(const struct sock *sk, struct sock *newsk)
#ifdef CONFIG_NET_SOCK_MSG
int tcp_bpf_sendmsg_redir(struct sock *sk, struct sk_msg *msg, u32 bytes,
int flags);
int tcp_bpf_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
int nonblock, int flags, int *addr_len);
int __tcp_bpf_recvmsg(struct sock *sk, struct sk_psock *psock,
struct msghdr *msg, int len, int flags);
#endif /* CONFIG_NET_SOCK_MSG */

View File

@ -111,6 +111,8 @@ enum bpf_cmd {
BPF_MAP_LOOKUP_AND_DELETE_BATCH,
BPF_MAP_UPDATE_BATCH,
BPF_MAP_DELETE_BATCH,
BPF_LINK_CREATE,
BPF_LINK_UPDATE,
};
enum bpf_map_type {
@ -181,6 +183,7 @@ enum bpf_prog_type {
BPF_PROG_TYPE_TRACING,
BPF_PROG_TYPE_STRUCT_OPS,
BPF_PROG_TYPE_EXT,
BPF_PROG_TYPE_LSM,
};
enum bpf_attach_type {
@ -211,6 +214,7 @@ enum bpf_attach_type {
BPF_TRACE_FENTRY,
BPF_TRACE_FEXIT,
BPF_MODIFY_RETURN,
BPF_LSM_MAC,
__MAX_BPF_ATTACH_TYPE
};
@ -539,7 +543,7 @@ union bpf_attr {
__u32 prog_cnt;
} query;
struct {
struct { /* anonymous struct used by BPF_RAW_TRACEPOINT_OPEN command */
__u64 name;
__u32 prog_fd;
} raw_tracepoint;
@ -567,6 +571,24 @@ union bpf_attr {
__u64 probe_offset; /* output: probe_offset */
__u64 probe_addr; /* output: probe_addr */
} task_fd_query;
struct { /* struct used by BPF_LINK_CREATE command */
__u32 prog_fd; /* eBPF program to attach */
__u32 target_fd; /* object to attach to */
__u32 attach_type; /* attach type */
__u32 flags; /* extra flags */
} link_create;
struct { /* struct used by BPF_LINK_UPDATE command */
__u32 link_fd; /* link fd */
/* new program fd to update link with */
__u32 new_prog_fd;
__u32 flags; /* extra flags */
/* expected link's program fd; is specified only if
* BPF_F_REPLACE flag is set in flags */
__u32 old_prog_fd;
} link_update;
} __attribute__((aligned(8)));
/* The description below is an attempt at providing documentation to eBPF
@ -2950,6 +2972,59 @@ union bpf_attr {
* restricted to raw_tracepoint bpf programs.
* Return
* 0 on success, or a negative error in case of failure.
*
* u64 bpf_get_netns_cookie(void *ctx)
* Description
* Retrieve the cookie (generated by the kernel) of the network
* namespace the input *ctx* is associated with. The network
* namespace cookie remains stable for its lifetime and provides
* a global identifier that can be assumed unique. If *ctx* is
* NULL, then the helper returns the cookie for the initial
* network namespace. The cookie itself is very similar to that
* of bpf_get_socket_cookie() helper, but for network namespaces
* instead of sockets.
* Return
* A 8-byte long opaque number.
*
* u64 bpf_get_current_ancestor_cgroup_id(int ancestor_level)
* Description
* Return id of cgroup v2 that is ancestor of the cgroup associated
* with the current task at the *ancestor_level*. The root cgroup
* is at *ancestor_level* zero and each step down the hierarchy
* increments the level. If *ancestor_level* == level of cgroup
* associated with the current task, then return value will be the
* same as that of **bpf_get_current_cgroup_id**\ ().
*
* The helper is useful to implement policies based on cgroups
* that are upper in hierarchy than immediate cgroup associated
* with the current task.
*
* The format of returned id and helper limitations are same as in
* **bpf_get_current_cgroup_id**\ ().
* Return
* The id is returned or 0 in case the id could not be retrieved.
*
* int bpf_sk_assign(struct sk_buff *skb, struct bpf_sock *sk, u64 flags)
* Description
* Assign the *sk* to the *skb*. When combined with appropriate
* routing configuration to receive the packet towards the socket,
* will cause *skb* to be delivered to the specified socket.
* Subsequent redirection of *skb* via **bpf_redirect**\ (),
* **bpf_clone_redirect**\ () or other methods outside of BPF may
* interfere with successful delivery to the socket.
*
* This operation is only valid from TC ingress path.
*
* The *flags* argument must be zero.
* Return
* 0 on success, or a negative errno in case of failure.
*
* * **-EINVAL** Unsupported flags specified.
* * **-ENOENT** Socket is unavailable for assignment.
* * **-ENETUNREACH** Socket is unreachable (wrong netns).
* * **-EOPNOTSUPP** Unsupported operation, for example a
* call from outside of TC ingress.
* * **-ESOCKTNOSUPPORT** Socket type not supported (reuseport).
*/
#define __BPF_FUNC_MAPPER(FN) \
FN(unspec), \
@ -3073,7 +3148,10 @@ union bpf_attr {
FN(jiffies64), \
FN(read_branch_records), \
FN(get_ns_current_pid_tgid), \
FN(xdp_output),
FN(xdp_output), \
FN(get_netns_cookie), \
FN(get_current_ancestor_cgroup_id), \
FN(sk_assign),
/* integer value in 'imm' field of BPF_CALL instruction selects which helper
* function eBPF program intends to call

View File

@ -974,11 +974,12 @@ enum {
#define XDP_FLAGS_SKB_MODE (1U << 1)
#define XDP_FLAGS_DRV_MODE (1U << 2)
#define XDP_FLAGS_HW_MODE (1U << 3)
#define XDP_FLAGS_REPLACE (1U << 4)
#define XDP_FLAGS_MODES (XDP_FLAGS_SKB_MODE | \
XDP_FLAGS_DRV_MODE | \
XDP_FLAGS_HW_MODE)
#define XDP_FLAGS_MASK (XDP_FLAGS_UPDATE_IF_NOEXIST | \
XDP_FLAGS_MODES)
XDP_FLAGS_MODES | XDP_FLAGS_REPLACE)
/* These are stored into IFLA_XDP_ATTACHED on dump. */
enum {
@ -998,6 +999,7 @@ enum {
IFLA_XDP_DRV_PROG_ID,
IFLA_XDP_SKB_PROG_ID,
IFLA_XDP_HW_PROG_ID,
IFLA_XDP_EXPECTED_FD,
__IFLA_XDP_MAX,
};

View File

@ -1615,6 +1615,19 @@ config KALLSYMS_BASE_RELATIVE
# end of the "standard kernel features (expert users)" menu
# syscall, maps, verifier
config BPF_LSM
bool "LSM Instrumentation with BPF"
depends on BPF_EVENTS
depends on BPF_SYSCALL
depends on SECURITY
depends on BPF_JIT
help
Enables instrumentation of the security hooks with eBPF programs for
implementing dynamic MAC and Audit Policies.
If you are unsure how to answer this question, answer N.
config BPF_SYSCALL
bool "Enable bpf() system call"
select BPF

View File

@ -29,4 +29,5 @@ obj-$(CONFIG_DEBUG_INFO_BTF) += sysfs_btf.o
endif
ifeq ($(CONFIG_BPF_JIT),y)
obj-$(CONFIG_BPF_SYSCALL) += bpf_struct_ops.o
obj-${CONFIG_BPF_LSM} += bpf_lsm.o
endif

54
kernel/bpf/bpf_lsm.c Normal file
View File

@ -0,0 +1,54 @@
// SPDX-License-Identifier: GPL-2.0
/*
* Copyright (C) 2020 Google LLC.
*/
#include <linux/filter.h>
#include <linux/bpf.h>
#include <linux/btf.h>
#include <linux/lsm_hooks.h>
#include <linux/bpf_lsm.h>
#include <linux/kallsyms.h>
#include <linux/bpf_verifier.h>
/* For every LSM hook that allows attachment of BPF programs, declare a nop
* function where a BPF program can be attached.
*/
#define LSM_HOOK(RET, DEFAULT, NAME, ...) \
noinline RET bpf_lsm_##NAME(__VA_ARGS__) \
{ \
return DEFAULT; \
}
#include <linux/lsm_hook_defs.h>
#undef LSM_HOOK
#define BPF_LSM_SYM_PREFX "bpf_lsm_"
int bpf_lsm_verify_prog(struct bpf_verifier_log *vlog,
const struct bpf_prog *prog)
{
if (!prog->gpl_compatible) {
bpf_log(vlog,
"LSM programs must have a GPL compatible license\n");
return -EINVAL;
}
if (strncmp(BPF_LSM_SYM_PREFX, prog->aux->attach_func_name,
sizeof(BPF_LSM_SYM_PREFX) - 1)) {
bpf_log(vlog, "attach_btf_id %u points to wrong type name %s\n",
prog->aux->attach_btf_id, prog->aux->attach_func_name);
return -EINVAL;
}
return 0;
}
const struct bpf_prog_ops lsm_prog_ops = {
};
const struct bpf_verifier_ops lsm_verifier_ops = {
.get_func_proto = bpf_tracing_func_proto,
.is_valid_access = btf_ctx_access,
};

View File

@ -3477,8 +3477,8 @@ errout:
return ERR_PTR(err);
}
extern char __weak _binary__btf_vmlinux_bin_start[];
extern char __weak _binary__btf_vmlinux_bin_end[];
extern char __weak __start_BTF[];
extern char __weak __stop_BTF[];
extern struct btf *btf_vmlinux;
#define BPF_MAP_TYPE(_id, _ops)
@ -3605,9 +3605,8 @@ struct btf *btf_parse_vmlinux(void)
}
env->btf = btf;
btf->data = _binary__btf_vmlinux_bin_start;
btf->data_size = _binary__btf_vmlinux_bin_end -
_binary__btf_vmlinux_bin_start;
btf->data = __start_BTF;
btf->data_size = __stop_BTF - __start_BTF;
err = btf_parse_hdr(env);
if (err)
@ -3710,12 +3709,34 @@ bool btf_ctx_access(int off, int size, enum bpf_access_type type,
nr_args--;
}
if (arg > nr_args) {
bpf_log(log, "func '%s' doesn't have %d-th argument\n",
tname, arg + 1);
return false;
}
if (arg == nr_args) {
if (prog->expected_attach_type == BPF_TRACE_FEXIT) {
switch (prog->expected_attach_type) {
case BPF_LSM_MAC:
case BPF_TRACE_FEXIT:
/* When LSM programs are attached to void LSM hooks
* they use FEXIT trampolines and when attached to
* int LSM hooks, they use MODIFY_RETURN trampolines.
*
* While the LSM programs are BPF_MODIFY_RETURN-like
* the check:
*
* if (ret_type != 'int')
* return -EINVAL;
*
* is _not_ done here. This is still safe as LSM hooks
* have only void and int return types.
*/
if (!t)
return true;
t = btf_type_by_id(btf, t->type);
} else if (prog->expected_attach_type == BPF_MODIFY_RETURN) {
break;
case BPF_MODIFY_RETURN:
/* For now the BPF_MODIFY_RETURN can only be attached to
* functions that return an int.
*/
@ -3729,17 +3750,19 @@ bool btf_ctx_access(int off, int size, enum bpf_access_type type,
btf_kind_str[BTF_INFO_KIND(t->info)]);
return false;
}
break;
default:
bpf_log(log, "func '%s' doesn't have %d-th argument\n",
tname, arg + 1);
return false;
}
} else if (arg >= nr_args) {
bpf_log(log, "func '%s' doesn't have %d-th argument\n",
tname, arg + 1);
return false;
} else {
if (!t)
/* Default prog with 5 args */
return true;
t = btf_type_by_id(btf, args[arg].type);
}
/* skip modifiers */
while (btf_type_is_modifier(t))
t = btf_type_by_id(btf, t->type);

View File

@ -28,6 +28,69 @@ void cgroup_bpf_offline(struct cgroup *cgrp)
percpu_ref_kill(&cgrp->bpf.refcnt);
}
static void bpf_cgroup_storages_free(struct bpf_cgroup_storage *storages[])
{
enum bpf_cgroup_storage_type stype;
for_each_cgroup_storage_type(stype)
bpf_cgroup_storage_free(storages[stype]);
}
static int bpf_cgroup_storages_alloc(struct bpf_cgroup_storage *storages[],
struct bpf_prog *prog)
{
enum bpf_cgroup_storage_type stype;
for_each_cgroup_storage_type(stype) {
storages[stype] = bpf_cgroup_storage_alloc(prog, stype);
if (IS_ERR(storages[stype])) {
storages[stype] = NULL;
bpf_cgroup_storages_free(storages);
return -ENOMEM;
}
}
return 0;
}
static void bpf_cgroup_storages_assign(struct bpf_cgroup_storage *dst[],
struct bpf_cgroup_storage *src[])
{
enum bpf_cgroup_storage_type stype;
for_each_cgroup_storage_type(stype)
dst[stype] = src[stype];
}
static void bpf_cgroup_storages_link(struct bpf_cgroup_storage *storages[],
struct cgroup* cgrp,
enum bpf_attach_type attach_type)
{
enum bpf_cgroup_storage_type stype;
for_each_cgroup_storage_type(stype)
bpf_cgroup_storage_link(storages[stype], cgrp, attach_type);
}
static void bpf_cgroup_storages_unlink(struct bpf_cgroup_storage *storages[])
{
enum bpf_cgroup_storage_type stype;
for_each_cgroup_storage_type(stype)
bpf_cgroup_storage_unlink(storages[stype]);
}
/* Called when bpf_cgroup_link is auto-detached from dying cgroup.
* It drops cgroup and bpf_prog refcounts, and marks bpf_link as defunct. It
* doesn't free link memory, which will eventually be done by bpf_link's
* release() callback, when its last FD is closed.
*/
static void bpf_cgroup_link_auto_detach(struct bpf_cgroup_link *link)
{
cgroup_put(link->cgroup);
link->cgroup = NULL;
}
/**
* cgroup_bpf_release() - put references of all bpf programs and
* release all cgroup bpf data
@ -37,7 +100,6 @@ static void cgroup_bpf_release(struct work_struct *work)
{
struct cgroup *p, *cgrp = container_of(work, struct cgroup,
bpf.release_work);
enum bpf_cgroup_storage_type stype;
struct bpf_prog_array *old_array;
unsigned int type;
@ -49,11 +111,12 @@ static void cgroup_bpf_release(struct work_struct *work)
list_for_each_entry_safe(pl, tmp, progs, node) {
list_del(&pl->node);
bpf_prog_put(pl->prog);
for_each_cgroup_storage_type(stype) {
bpf_cgroup_storage_unlink(pl->storage[stype]);
bpf_cgroup_storage_free(pl->storage[stype]);
}
if (pl->prog)
bpf_prog_put(pl->prog);
if (pl->link)
bpf_cgroup_link_auto_detach(pl->link);
bpf_cgroup_storages_unlink(pl->storage);
bpf_cgroup_storages_free(pl->storage);
kfree(pl);
static_branch_dec(&cgroup_bpf_enabled_key);
}
@ -85,6 +148,18 @@ static void cgroup_bpf_release_fn(struct percpu_ref *ref)
queue_work(system_wq, &cgrp->bpf.release_work);
}
/* Get underlying bpf_prog of bpf_prog_list entry, regardless if it's through
* link or direct prog.
*/
static struct bpf_prog *prog_list_prog(struct bpf_prog_list *pl)
{
if (pl->prog)
return pl->prog;
if (pl->link)
return pl->link->link.prog;
return NULL;
}
/* count number of elements in the list.
* it's slow but the list cannot be long
*/
@ -94,7 +169,7 @@ static u32 prog_list_length(struct list_head *head)
u32 cnt = 0;
list_for_each_entry(pl, head, node) {
if (!pl->prog)
if (!prog_list_prog(pl))
continue;
cnt++;
}
@ -138,7 +213,7 @@ static int compute_effective_progs(struct cgroup *cgrp,
enum bpf_attach_type type,
struct bpf_prog_array **array)
{
enum bpf_cgroup_storage_type stype;
struct bpf_prog_array_item *item;
struct bpf_prog_array *progs;
struct bpf_prog_list *pl;
struct cgroup *p = cgrp;
@ -163,13 +238,13 @@ static int compute_effective_progs(struct cgroup *cgrp,
continue;
list_for_each_entry(pl, &p->bpf.progs[type], node) {
if (!pl->prog)
if (!prog_list_prog(pl))
continue;
progs->items[cnt].prog = pl->prog;
for_each_cgroup_storage_type(stype)
progs->items[cnt].cgroup_storage[stype] =
pl->storage[stype];
item = &progs->items[cnt];
item->prog = prog_list_prog(pl);
bpf_cgroup_storages_assign(item->cgroup_storage,
pl->storage);
cnt++;
}
} while ((p = cgroup_parent(p)));
@ -287,19 +362,60 @@ cleanup:
#define BPF_CGROUP_MAX_PROGS 64
static struct bpf_prog_list *find_attach_entry(struct list_head *progs,
struct bpf_prog *prog,
struct bpf_cgroup_link *link,
struct bpf_prog *replace_prog,
bool allow_multi)
{
struct bpf_prog_list *pl;
/* single-attach case */
if (!allow_multi) {
if (list_empty(progs))
return NULL;
return list_first_entry(progs, typeof(*pl), node);
}
list_for_each_entry(pl, progs, node) {
if (prog && pl->prog == prog)
/* disallow attaching the same prog twice */
return ERR_PTR(-EINVAL);
if (link && pl->link == link)
/* disallow attaching the same link twice */
return ERR_PTR(-EINVAL);
}
/* direct prog multi-attach w/ replacement case */
if (replace_prog) {
list_for_each_entry(pl, progs, node) {
if (pl->prog == replace_prog)
/* a match found */
return pl;
}
/* prog to replace not found for cgroup */
return ERR_PTR(-ENOENT);
}
return NULL;
}
/**
* __cgroup_bpf_attach() - Attach the program to a cgroup, and
* __cgroup_bpf_attach() - Attach the program or the link to a cgroup, and
* propagate the change to descendants
* @cgrp: The cgroup which descendants to traverse
* @prog: A program to attach
* @link: A link to attach
* @replace_prog: Previously attached program to replace if BPF_F_REPLACE is set
* @type: Type of attach operation
* @flags: Option flags
*
* Exactly one of @prog or @link can be non-null.
* Must be called with cgroup_mutex held.
*/
int __cgroup_bpf_attach(struct cgroup *cgrp, struct bpf_prog *prog,
struct bpf_prog *replace_prog,
int __cgroup_bpf_attach(struct cgroup *cgrp,
struct bpf_prog *prog, struct bpf_prog *replace_prog,
struct bpf_cgroup_link *link,
enum bpf_attach_type type, u32 flags)
{
u32 saved_flags = (flags & (BPF_F_ALLOW_OVERRIDE | BPF_F_ALLOW_MULTI));
@ -307,14 +423,19 @@ int __cgroup_bpf_attach(struct cgroup *cgrp, struct bpf_prog *prog,
struct bpf_prog *old_prog = NULL;
struct bpf_cgroup_storage *storage[MAX_BPF_CGROUP_STORAGE_TYPE] = {};
struct bpf_cgroup_storage *old_storage[MAX_BPF_CGROUP_STORAGE_TYPE] = {};
struct bpf_prog_list *pl, *replace_pl = NULL;
enum bpf_cgroup_storage_type stype;
struct bpf_prog_list *pl;
int err;
if (((flags & BPF_F_ALLOW_OVERRIDE) && (flags & BPF_F_ALLOW_MULTI)) ||
((flags & BPF_F_REPLACE) && !(flags & BPF_F_ALLOW_MULTI)))
/* invalid combination */
return -EINVAL;
if (link && (prog || replace_prog))
/* only either link or prog/replace_prog can be specified */
return -EINVAL;
if (!!replace_prog != !!(flags & BPF_F_REPLACE))
/* replace_prog implies BPF_F_REPLACE, and vice versa */
return -EINVAL;
if (!hierarchy_allows_attach(cgrp, type))
return -EPERM;
@ -329,140 +450,203 @@ int __cgroup_bpf_attach(struct cgroup *cgrp, struct bpf_prog *prog,
if (prog_list_length(progs) >= BPF_CGROUP_MAX_PROGS)
return -E2BIG;
if (flags & BPF_F_ALLOW_MULTI) {
list_for_each_entry(pl, progs, node) {
if (pl->prog == prog)
/* disallow attaching the same prog twice */
return -EINVAL;
if (pl->prog == replace_prog)
replace_pl = pl;
}
if ((flags & BPF_F_REPLACE) && !replace_pl)
/* prog to replace not found for cgroup */
return -ENOENT;
} else if (!list_empty(progs)) {
replace_pl = list_first_entry(progs, typeof(*pl), node);
}
pl = find_attach_entry(progs, prog, link, replace_prog,
flags & BPF_F_ALLOW_MULTI);
if (IS_ERR(pl))
return PTR_ERR(pl);
for_each_cgroup_storage_type(stype) {
storage[stype] = bpf_cgroup_storage_alloc(prog, stype);
if (IS_ERR(storage[stype])) {
storage[stype] = NULL;
for_each_cgroup_storage_type(stype)
bpf_cgroup_storage_free(storage[stype]);
return -ENOMEM;
}
}
if (bpf_cgroup_storages_alloc(storage, prog ? : link->link.prog))
return -ENOMEM;
if (replace_pl) {
pl = replace_pl;
if (pl) {
old_prog = pl->prog;
for_each_cgroup_storage_type(stype) {
old_storage[stype] = pl->storage[stype];
bpf_cgroup_storage_unlink(old_storage[stype]);
}
bpf_cgroup_storages_unlink(pl->storage);
bpf_cgroup_storages_assign(old_storage, pl->storage);
} else {
pl = kmalloc(sizeof(*pl), GFP_KERNEL);
if (!pl) {
for_each_cgroup_storage_type(stype)
bpf_cgroup_storage_free(storage[stype]);
bpf_cgroup_storages_free(storage);
return -ENOMEM;
}
list_add_tail(&pl->node, progs);
}
pl->prog = prog;
for_each_cgroup_storage_type(stype)
pl->storage[stype] = storage[stype];
pl->link = link;
bpf_cgroup_storages_assign(pl->storage, storage);
cgrp->bpf.flags[type] = saved_flags;
err = update_effective_progs(cgrp, type);
if (err)
goto cleanup;
static_branch_inc(&cgroup_bpf_enabled_key);
for_each_cgroup_storage_type(stype) {
if (!old_storage[stype])
continue;
bpf_cgroup_storage_free(old_storage[stype]);
}
if (old_prog) {
bpf_cgroup_storages_free(old_storage);
if (old_prog)
bpf_prog_put(old_prog);
static_branch_dec(&cgroup_bpf_enabled_key);
}
for_each_cgroup_storage_type(stype)
bpf_cgroup_storage_link(storage[stype], cgrp, type);
else
static_branch_inc(&cgroup_bpf_enabled_key);
bpf_cgroup_storages_link(pl->storage, cgrp, type);
return 0;
cleanup:
/* and cleanup the prog list */
pl->prog = old_prog;
for_each_cgroup_storage_type(stype) {
bpf_cgroup_storage_free(pl->storage[stype]);
pl->storage[stype] = old_storage[stype];
bpf_cgroup_storage_link(old_storage[stype], cgrp, type);
if (old_prog) {
pl->prog = old_prog;
pl->link = NULL;
}
if (!replace_pl) {
bpf_cgroup_storages_free(pl->storage);
bpf_cgroup_storages_assign(pl->storage, old_storage);
bpf_cgroup_storages_link(pl->storage, cgrp, type);
if (!old_prog) {
list_del(&pl->node);
kfree(pl);
}
return err;
}
/* Swap updated BPF program for given link in effective program arrays across
* all descendant cgroups. This function is guaranteed to succeed.
*/
static void replace_effective_prog(struct cgroup *cgrp,
enum bpf_attach_type type,
struct bpf_cgroup_link *link)
{
struct bpf_prog_array_item *item;
struct cgroup_subsys_state *css;
struct bpf_prog_array *progs;
struct bpf_prog_list *pl;
struct list_head *head;
struct cgroup *cg;
int pos;
css_for_each_descendant_pre(css, &cgrp->self) {
struct cgroup *desc = container_of(css, struct cgroup, self);
if (percpu_ref_is_zero(&desc->bpf.refcnt))
continue;
/* find position of link in effective progs array */
for (pos = 0, cg = desc; cg; cg = cgroup_parent(cg)) {
if (pos && !(cg->bpf.flags[type] & BPF_F_ALLOW_MULTI))
continue;
head = &cg->bpf.progs[type];
list_for_each_entry(pl, head, node) {
if (!prog_list_prog(pl))
continue;
if (pl->link == link)
goto found;
pos++;
}
}
found:
BUG_ON(!cg);
progs = rcu_dereference_protected(
desc->bpf.effective[type],
lockdep_is_held(&cgroup_mutex));
item = &progs->items[pos];
WRITE_ONCE(item->prog, link->link.prog);
}
}
/**
* __cgroup_bpf_detach() - Detach the program from a cgroup, and
* propagate the change to descendants
* __cgroup_bpf_replace() - Replace link's program and propagate the change
* to descendants
* @cgrp: The cgroup which descendants to traverse
* @prog: A program to detach or NULL
* @type: Type of detach operation
* @link: A link for which to replace BPF program
* @type: Type of attach operation
*
* Must be called with cgroup_mutex held.
*/
int __cgroup_bpf_detach(struct cgroup *cgrp, struct bpf_prog *prog,
enum bpf_attach_type type)
int __cgroup_bpf_replace(struct cgroup *cgrp, struct bpf_cgroup_link *link,
struct bpf_prog *new_prog)
{
struct list_head *progs = &cgrp->bpf.progs[type];
enum bpf_cgroup_storage_type stype;
u32 flags = cgrp->bpf.flags[type];
struct bpf_prog *old_prog = NULL;
struct list_head *progs = &cgrp->bpf.progs[link->type];
struct bpf_prog *old_prog;
struct bpf_prog_list *pl;
int err;
bool found = false;
if (flags & BPF_F_ALLOW_MULTI) {
if (!prog)
/* to detach MULTI prog the user has to specify valid FD
* of the program to be detached
*/
return -EINVAL;
} else {
if (list_empty(progs))
/* report error when trying to detach and nothing is attached */
return -ENOENT;
}
if (link->link.prog->type != new_prog->type)
return -EINVAL;
if (flags & BPF_F_ALLOW_MULTI) {
/* find the prog and detach it */
list_for_each_entry(pl, progs, node) {
if (pl->prog != prog)
continue;
old_prog = prog;
/* mark it deleted, so it's ignored while
* recomputing effective
*/
pl->prog = NULL;
list_for_each_entry(pl, progs, node) {
if (pl->link == link) {
found = true;
break;
}
if (!old_prog)
return -ENOENT;
} else {
/* to maintain backward compatibility NONE and OVERRIDE cgroups
* allow detaching with invalid FD (prog==NULL)
*/
pl = list_first_entry(progs, typeof(*pl), node);
old_prog = pl->prog;
pl->prog = NULL;
}
if (!found)
return -ENOENT;
old_prog = xchg(&link->link.prog, new_prog);
replace_effective_prog(cgrp, link->type, link);
bpf_prog_put(old_prog);
return 0;
}
static struct bpf_prog_list *find_detach_entry(struct list_head *progs,
struct bpf_prog *prog,
struct bpf_cgroup_link *link,
bool allow_multi)
{
struct bpf_prog_list *pl;
if (!allow_multi) {
if (list_empty(progs))
/* report error when trying to detach and nothing is attached */
return ERR_PTR(-ENOENT);
/* to maintain backward compatibility NONE and OVERRIDE cgroups
* allow detaching with invalid FD (prog==NULL) in legacy mode
*/
return list_first_entry(progs, typeof(*pl), node);
}
if (!prog && !link)
/* to detach MULTI prog the user has to specify valid FD
* of the program or link to be detached
*/
return ERR_PTR(-EINVAL);
/* find the prog or link and detach it */
list_for_each_entry(pl, progs, node) {
if (pl->prog == prog && pl->link == link)
return pl;
}
return ERR_PTR(-ENOENT);
}
/**
* __cgroup_bpf_detach() - Detach the program or link from a cgroup, and
* propagate the change to descendants
* @cgrp: The cgroup which descendants to traverse
* @prog: A program to detach or NULL
* @prog: A link to detach or NULL
* @type: Type of detach operation
*
* At most one of @prog or @link can be non-NULL.
* Must be called with cgroup_mutex held.
*/
int __cgroup_bpf_detach(struct cgroup *cgrp, struct bpf_prog *prog,
struct bpf_cgroup_link *link, enum bpf_attach_type type)
{
struct list_head *progs = &cgrp->bpf.progs[type];
u32 flags = cgrp->bpf.flags[type];
struct bpf_prog_list *pl;
struct bpf_prog *old_prog;
int err;
if (prog && link)
/* only one of prog or link can be specified */
return -EINVAL;
pl = find_detach_entry(progs, prog, link, flags & BPF_F_ALLOW_MULTI);
if (IS_ERR(pl))
return PTR_ERR(pl);
/* mark it deleted, so it's ignored while recomputing effective */
old_prog = pl->prog;
pl->prog = NULL;
pl->link = NULL;
err = update_effective_progs(cgrp, type);
if (err)
@ -470,22 +654,21 @@ int __cgroup_bpf_detach(struct cgroup *cgrp, struct bpf_prog *prog,
/* now can actually delete it from this cgroup list */
list_del(&pl->node);
for_each_cgroup_storage_type(stype) {
bpf_cgroup_storage_unlink(pl->storage[stype]);
bpf_cgroup_storage_free(pl->storage[stype]);
}
bpf_cgroup_storages_unlink(pl->storage);
bpf_cgroup_storages_free(pl->storage);
kfree(pl);
if (list_empty(progs))
/* last program was detached, reset flags to zero */
cgrp->bpf.flags[type] = 0;
bpf_prog_put(old_prog);
if (old_prog)
bpf_prog_put(old_prog);
static_branch_dec(&cgroup_bpf_enabled_key);
return 0;
cleanup:
/* and restore back old_prog */
/* restore back prog or link */
pl->prog = old_prog;
pl->link = link;
return err;
}
@ -498,6 +681,7 @@ int __cgroup_bpf_query(struct cgroup *cgrp, const union bpf_attr *attr,
struct list_head *progs = &cgrp->bpf.progs[type];
u32 flags = cgrp->bpf.flags[type];
struct bpf_prog_array *effective;
struct bpf_prog *prog;
int cnt, ret = 0, i;
effective = rcu_dereference_protected(cgrp->bpf.effective[type],
@ -528,7 +712,8 @@ int __cgroup_bpf_query(struct cgroup *cgrp, const union bpf_attr *attr,
i = 0;
list_for_each_entry(pl, progs, node) {
id = pl->prog->aux->id;
prog = prog_list_prog(pl);
id = prog->aux->id;
if (copy_to_user(prog_ids + i, &id, sizeof(id)))
return -EFAULT;
if (++i == cnt)
@ -558,8 +743,8 @@ int cgroup_bpf_prog_attach(const union bpf_attr *attr,
}
}
ret = cgroup_bpf_attach(cgrp, prog, replace_prog, attr->attach_type,
attr->attach_flags);
ret = cgroup_bpf_attach(cgrp, prog, replace_prog, NULL,
attr->attach_type, attr->attach_flags);
if (replace_prog)
bpf_prog_put(replace_prog);
@ -581,7 +766,7 @@ int cgroup_bpf_prog_detach(const union bpf_attr *attr, enum bpf_prog_type ptype)
if (IS_ERR(prog))
prog = NULL;
ret = cgroup_bpf_detach(cgrp, prog, attr->attach_type, 0);
ret = cgroup_bpf_detach(cgrp, prog, attr->attach_type);
if (prog)
bpf_prog_put(prog);
@ -589,6 +774,90 @@ int cgroup_bpf_prog_detach(const union bpf_attr *attr, enum bpf_prog_type ptype)
return ret;
}
static void bpf_cgroup_link_release(struct bpf_link *link)
{
struct bpf_cgroup_link *cg_link =
container_of(link, struct bpf_cgroup_link, link);
/* link might have been auto-detached by dying cgroup already,
* in that case our work is done here
*/
if (!cg_link->cgroup)
return;
mutex_lock(&cgroup_mutex);
/* re-check cgroup under lock again */
if (!cg_link->cgroup) {
mutex_unlock(&cgroup_mutex);
return;
}
WARN_ON(__cgroup_bpf_detach(cg_link->cgroup, NULL, cg_link,
cg_link->type));
mutex_unlock(&cgroup_mutex);
cgroup_put(cg_link->cgroup);
}
static void bpf_cgroup_link_dealloc(struct bpf_link *link)
{
struct bpf_cgroup_link *cg_link =
container_of(link, struct bpf_cgroup_link, link);
kfree(cg_link);
}
const struct bpf_link_ops bpf_cgroup_link_lops = {
.release = bpf_cgroup_link_release,
.dealloc = bpf_cgroup_link_dealloc,
};
int cgroup_bpf_link_attach(const union bpf_attr *attr, struct bpf_prog *prog)
{
struct bpf_cgroup_link *link;
struct file *link_file;
struct cgroup *cgrp;
int err, link_fd;
if (attr->link_create.flags)
return -EINVAL;
cgrp = cgroup_get_from_fd(attr->link_create.target_fd);
if (IS_ERR(cgrp))
return PTR_ERR(cgrp);
link = kzalloc(sizeof(*link), GFP_USER);
if (!link) {
err = -ENOMEM;
goto out_put_cgroup;
}
bpf_link_init(&link->link, &bpf_cgroup_link_lops, prog);
link->cgroup = cgrp;
link->type = attr->link_create.attach_type;
link_file = bpf_link_new_file(&link->link, &link_fd);
if (IS_ERR(link_file)) {
kfree(link);
err = PTR_ERR(link_file);
goto out_put_cgroup;
}
err = cgroup_bpf_attach(cgrp, NULL, NULL, link, link->type,
BPF_F_ALLOW_MULTI);
if (err) {
bpf_link_cleanup(&link->link, link_file, link_fd);
goto out_put_cgroup;
}
fd_install(link_fd, link_file);
return link_fd;
out_put_cgroup:
cgroup_put(cgrp);
return err;
}
int cgroup_bpf_prog_query(const union bpf_attr *attr,
union bpf_attr __user *uattr)
{

View File

@ -2156,6 +2156,7 @@ const struct bpf_func_proto bpf_get_current_pid_tgid_proto __weak;
const struct bpf_func_proto bpf_get_current_uid_gid_proto __weak;
const struct bpf_func_proto bpf_get_current_comm_proto __weak;
const struct bpf_func_proto bpf_get_current_cgroup_id_proto __weak;
const struct bpf_func_proto bpf_get_current_ancestor_cgroup_id_proto __weak;
const struct bpf_func_proto bpf_get_local_storage_proto __weak;
const struct bpf_func_proto bpf_get_ns_current_pid_tgid_proto __weak;

View File

@ -340,6 +340,24 @@ const struct bpf_func_proto bpf_get_current_cgroup_id_proto = {
.ret_type = RET_INTEGER,
};
BPF_CALL_1(bpf_get_current_ancestor_cgroup_id, int, ancestor_level)
{
struct cgroup *cgrp = task_dfl_cgroup(current);
struct cgroup *ancestor;
ancestor = cgroup_ancestor(cgrp, ancestor_level);
if (!ancestor)
return 0;
return cgroup_id(ancestor);
}
const struct bpf_func_proto bpf_get_current_ancestor_cgroup_id_proto = {
.func = bpf_get_current_ancestor_cgroup_id,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_ANYTHING,
};
#ifdef CONFIG_CGROUP_BPF
DECLARE_PER_CPU(struct bpf_cgroup_storage*,
bpf_cgroup_storage[MAX_BPF_CGROUP_STORAGE_TYPE]);

View File

@ -25,6 +25,7 @@
#include <linux/nospec.h>
#include <linux/audit.h>
#include <uapi/linux/btf.h>
#include <linux/bpf_lsm.h>
#define IS_FD_ARRAY(map) ((map)->map_type == BPF_MAP_TYPE_PERF_EVENT_ARRAY || \
(map)->map_type == BPF_MAP_TYPE_CGROUP_ARRAY || \
@ -1942,6 +1943,7 @@ bpf_prog_load_check_attach(enum bpf_prog_type prog_type,
switch (prog_type) {
case BPF_PROG_TYPE_TRACING:
case BPF_PROG_TYPE_LSM:
case BPF_PROG_TYPE_STRUCT_OPS:
case BPF_PROG_TYPE_EXT:
break;
@ -2181,13 +2183,6 @@ static int bpf_obj_get(const union bpf_attr *attr)
attr->file_flags);
}
struct bpf_link {
atomic64_t refcnt;
const struct bpf_link_ops *ops;
struct bpf_prog *prog;
struct work_struct work;
};
void bpf_link_init(struct bpf_link *link, const struct bpf_link_ops *ops,
struct bpf_prog *prog)
{
@ -2201,8 +2196,8 @@ void bpf_link_init(struct bpf_link *link, const struct bpf_link_ops *ops,
* anon_inode's release() call. This helper manages marking bpf_link as
* defunct, releases anon_inode file and puts reserved FD.
*/
static void bpf_link_cleanup(struct bpf_link *link, struct file *link_file,
int link_fd)
void bpf_link_cleanup(struct bpf_link *link, struct file *link_file,
int link_fd)
{
link->prog = NULL;
fput(link_file);
@ -2260,7 +2255,6 @@ static int bpf_link_release(struct inode *inode, struct file *filp)
#ifdef CONFIG_PROC_FS
static const struct bpf_link_ops bpf_raw_tp_lops;
static const struct bpf_link_ops bpf_tracing_link_lops;
static const struct bpf_link_ops bpf_xdp_link_lops;
static void bpf_link_show_fdinfo(struct seq_file *m, struct file *filp)
{
@ -2273,6 +2267,10 @@ static void bpf_link_show_fdinfo(struct seq_file *m, struct file *filp)
link_type = "raw_tracepoint";
else if (link->ops == &bpf_tracing_link_lops)
link_type = "tracing";
#ifdef CONFIG_CGROUP_BPF
else if (link->ops == &bpf_cgroup_link_lops)
link_type = "cgroup";
#endif
else
link_type = "unknown";
@ -2375,10 +2373,28 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog)
struct file *link_file;
int link_fd, err;
if (prog->expected_attach_type != BPF_TRACE_FENTRY &&
prog->expected_attach_type != BPF_TRACE_FEXIT &&
prog->expected_attach_type != BPF_MODIFY_RETURN &&
prog->type != BPF_PROG_TYPE_EXT) {
switch (prog->type) {
case BPF_PROG_TYPE_TRACING:
if (prog->expected_attach_type != BPF_TRACE_FENTRY &&
prog->expected_attach_type != BPF_TRACE_FEXIT &&
prog->expected_attach_type != BPF_MODIFY_RETURN) {
err = -EINVAL;
goto out_put_prog;
}
break;
case BPF_PROG_TYPE_EXT:
if (prog->expected_attach_type != 0) {
err = -EINVAL;
goto out_put_prog;
}
break;
case BPF_PROG_TYPE_LSM:
if (prog->expected_attach_type != BPF_LSM_MAC) {
err = -EINVAL;
goto out_put_prog;
}
break;
default:
err = -EINVAL;
goto out_put_prog;
}
@ -2457,16 +2473,10 @@ static int bpf_raw_tracepoint_open(const union bpf_attr *attr)
if (IS_ERR(prog))
return PTR_ERR(prog);
if (prog->type != BPF_PROG_TYPE_RAW_TRACEPOINT &&
prog->type != BPF_PROG_TYPE_TRACING &&
prog->type != BPF_PROG_TYPE_EXT &&
prog->type != BPF_PROG_TYPE_RAW_TRACEPOINT_WRITABLE) {
err = -EINVAL;
goto out_put_prog;
}
if (prog->type == BPF_PROG_TYPE_TRACING ||
prog->type == BPF_PROG_TYPE_EXT) {
switch (prog->type) {
case BPF_PROG_TYPE_TRACING:
case BPF_PROG_TYPE_EXT:
case BPF_PROG_TYPE_LSM:
if (attr->raw_tracepoint.name) {
/* The attach point for this category of programs
* should be specified via btf_id during program load.
@ -2474,11 +2484,14 @@ static int bpf_raw_tracepoint_open(const union bpf_attr *attr)
err = -EINVAL;
goto out_put_prog;
}
if (prog->expected_attach_type == BPF_TRACE_RAW_TP)
if (prog->type == BPF_PROG_TYPE_TRACING &&
prog->expected_attach_type == BPF_TRACE_RAW_TP) {
tp_name = prog->aux->attach_func_name;
else
return bpf_tracing_prog_attach(prog);
} else {
break;
}
return bpf_tracing_prog_attach(prog);
case BPF_PROG_TYPE_RAW_TRACEPOINT:
case BPF_PROG_TYPE_RAW_TRACEPOINT_WRITABLE:
if (strncpy_from_user(buf,
u64_to_user_ptr(attr->raw_tracepoint.name),
sizeof(buf) - 1) < 0) {
@ -2487,6 +2500,10 @@ static int bpf_raw_tracepoint_open(const union bpf_attr *attr)
}
buf[sizeof(buf) - 1] = 0;
tp_name = buf;
break;
default:
err = -EINVAL;
goto out_put_prog;
}
btp = bpf_get_raw_tracepoint(tp_name);
@ -2543,6 +2560,50 @@ static int bpf_prog_attach_check_attach_type(const struct bpf_prog *prog,
}
}
static enum bpf_prog_type
attach_type_to_prog_type(enum bpf_attach_type attach_type)
{
switch (attach_type) {
case BPF_CGROUP_INET_INGRESS:
case BPF_CGROUP_INET_EGRESS:
return BPF_PROG_TYPE_CGROUP_SKB;
break;
case BPF_CGROUP_INET_SOCK_CREATE:
case BPF_CGROUP_INET4_POST_BIND:
case BPF_CGROUP_INET6_POST_BIND:
return BPF_PROG_TYPE_CGROUP_SOCK;
case BPF_CGROUP_INET4_BIND:
case BPF_CGROUP_INET6_BIND:
case BPF_CGROUP_INET4_CONNECT:
case BPF_CGROUP_INET6_CONNECT:
case BPF_CGROUP_UDP4_SENDMSG:
case BPF_CGROUP_UDP6_SENDMSG:
case BPF_CGROUP_UDP4_RECVMSG:
case BPF_CGROUP_UDP6_RECVMSG:
return BPF_PROG_TYPE_CGROUP_SOCK_ADDR;
case BPF_CGROUP_SOCK_OPS:
return BPF_PROG_TYPE_SOCK_OPS;
case BPF_CGROUP_DEVICE:
return BPF_PROG_TYPE_CGROUP_DEVICE;
case BPF_SK_MSG_VERDICT:
return BPF_PROG_TYPE_SK_MSG;
case BPF_SK_SKB_STREAM_PARSER:
case BPF_SK_SKB_STREAM_VERDICT:
return BPF_PROG_TYPE_SK_SKB;
case BPF_LIRC_MODE2:
return BPF_PROG_TYPE_LIRC_MODE2;
case BPF_FLOW_DISSECTOR:
return BPF_PROG_TYPE_FLOW_DISSECTOR;
case BPF_CGROUP_SYSCTL:
return BPF_PROG_TYPE_CGROUP_SYSCTL;
case BPF_CGROUP_GETSOCKOPT:
case BPF_CGROUP_SETSOCKOPT:
return BPF_PROG_TYPE_CGROUP_SOCKOPT;
default:
return BPF_PROG_TYPE_UNSPEC;
}
}
#define BPF_PROG_ATTACH_LAST_FIELD replace_bpf_fd
#define BPF_F_ATTACH_MASK \
@ -2563,55 +2624,9 @@ static int bpf_prog_attach(const union bpf_attr *attr)
if (attr->attach_flags & ~BPF_F_ATTACH_MASK)
return -EINVAL;
switch (attr->attach_type) {
case BPF_CGROUP_INET_INGRESS:
case BPF_CGROUP_INET_EGRESS:
ptype = BPF_PROG_TYPE_CGROUP_SKB;
break;
case BPF_CGROUP_INET_SOCK_CREATE:
case BPF_CGROUP_INET4_POST_BIND:
case BPF_CGROUP_INET6_POST_BIND:
ptype = BPF_PROG_TYPE_CGROUP_SOCK;
break;
case BPF_CGROUP_INET4_BIND:
case BPF_CGROUP_INET6_BIND:
case BPF_CGROUP_INET4_CONNECT:
case BPF_CGROUP_INET6_CONNECT:
case BPF_CGROUP_UDP4_SENDMSG:
case BPF_CGROUP_UDP6_SENDMSG:
case BPF_CGROUP_UDP4_RECVMSG:
case BPF_CGROUP_UDP6_RECVMSG:
ptype = BPF_PROG_TYPE_CGROUP_SOCK_ADDR;
break;
case BPF_CGROUP_SOCK_OPS:
ptype = BPF_PROG_TYPE_SOCK_OPS;
break;
case BPF_CGROUP_DEVICE:
ptype = BPF_PROG_TYPE_CGROUP_DEVICE;
break;
case BPF_SK_MSG_VERDICT:
ptype = BPF_PROG_TYPE_SK_MSG;
break;
case BPF_SK_SKB_STREAM_PARSER:
case BPF_SK_SKB_STREAM_VERDICT:
ptype = BPF_PROG_TYPE_SK_SKB;
break;
case BPF_LIRC_MODE2:
ptype = BPF_PROG_TYPE_LIRC_MODE2;
break;
case BPF_FLOW_DISSECTOR:
ptype = BPF_PROG_TYPE_FLOW_DISSECTOR;
break;
case BPF_CGROUP_SYSCTL:
ptype = BPF_PROG_TYPE_CGROUP_SYSCTL;
break;
case BPF_CGROUP_GETSOCKOPT:
case BPF_CGROUP_SETSOCKOPT:
ptype = BPF_PROG_TYPE_CGROUP_SOCKOPT;
break;
default:
ptype = attach_type_to_prog_type(attr->attach_type);
if (ptype == BPF_PROG_TYPE_UNSPEC)
return -EINVAL;
}
prog = bpf_prog_get_type(attr->attach_bpf_fd, ptype);
if (IS_ERR(prog))
@ -2633,8 +2648,17 @@ static int bpf_prog_attach(const union bpf_attr *attr)
case BPF_PROG_TYPE_FLOW_DISSECTOR:
ret = skb_flow_dissector_bpf_prog_attach(attr, prog);
break;
default:
case BPF_PROG_TYPE_CGROUP_DEVICE:
case BPF_PROG_TYPE_CGROUP_SKB:
case BPF_PROG_TYPE_CGROUP_SOCK:
case BPF_PROG_TYPE_CGROUP_SOCK_ADDR:
case BPF_PROG_TYPE_CGROUP_SOCKOPT:
case BPF_PROG_TYPE_CGROUP_SYSCTL:
case BPF_PROG_TYPE_SOCK_OPS:
ret = cgroup_bpf_prog_attach(attr, ptype, prog);
break;
default:
ret = -EINVAL;
}
if (ret)
@ -2654,53 +2678,27 @@ static int bpf_prog_detach(const union bpf_attr *attr)
if (CHECK_ATTR(BPF_PROG_DETACH))
return -EINVAL;
switch (attr->attach_type) {
case BPF_CGROUP_INET_INGRESS:
case BPF_CGROUP_INET_EGRESS:
ptype = BPF_PROG_TYPE_CGROUP_SKB;
break;
case BPF_CGROUP_INET_SOCK_CREATE:
case BPF_CGROUP_INET4_POST_BIND:
case BPF_CGROUP_INET6_POST_BIND:
ptype = BPF_PROG_TYPE_CGROUP_SOCK;
break;
case BPF_CGROUP_INET4_BIND:
case BPF_CGROUP_INET6_BIND:
case BPF_CGROUP_INET4_CONNECT:
case BPF_CGROUP_INET6_CONNECT:
case BPF_CGROUP_UDP4_SENDMSG:
case BPF_CGROUP_UDP6_SENDMSG:
case BPF_CGROUP_UDP4_RECVMSG:
case BPF_CGROUP_UDP6_RECVMSG:
ptype = BPF_PROG_TYPE_CGROUP_SOCK_ADDR;
break;
case BPF_CGROUP_SOCK_OPS:
ptype = BPF_PROG_TYPE_SOCK_OPS;
break;
case BPF_CGROUP_DEVICE:
ptype = BPF_PROG_TYPE_CGROUP_DEVICE;
break;
case BPF_SK_MSG_VERDICT:
ptype = attach_type_to_prog_type(attr->attach_type);
switch (ptype) {
case BPF_PROG_TYPE_SK_MSG:
case BPF_PROG_TYPE_SK_SKB:
return sock_map_get_from_fd(attr, NULL);
case BPF_SK_SKB_STREAM_PARSER:
case BPF_SK_SKB_STREAM_VERDICT:
return sock_map_get_from_fd(attr, NULL);
case BPF_LIRC_MODE2:
case BPF_PROG_TYPE_LIRC_MODE2:
return lirc_prog_detach(attr);
case BPF_FLOW_DISSECTOR:
case BPF_PROG_TYPE_FLOW_DISSECTOR:
return skb_flow_dissector_bpf_prog_detach(attr);
case BPF_CGROUP_SYSCTL:
ptype = BPF_PROG_TYPE_CGROUP_SYSCTL;
break;
case BPF_CGROUP_GETSOCKOPT:
case BPF_CGROUP_SETSOCKOPT:
ptype = BPF_PROG_TYPE_CGROUP_SOCKOPT;
break;
case BPF_PROG_TYPE_CGROUP_DEVICE:
case BPF_PROG_TYPE_CGROUP_SKB:
case BPF_PROG_TYPE_CGROUP_SOCK:
case BPF_PROG_TYPE_CGROUP_SOCK_ADDR:
case BPF_PROG_TYPE_CGROUP_SOCKOPT:
case BPF_PROG_TYPE_CGROUP_SYSCTL:
case BPF_PROG_TYPE_SOCK_OPS:
return cgroup_bpf_prog_detach(attr, ptype);
default:
return -EINVAL;
}
return cgroup_bpf_prog_detach(attr, ptype);
}
#define BPF_PROG_QUERY_LAST_FIELD query.prog_cnt
@ -2734,7 +2732,7 @@ static int bpf_prog_query(const union bpf_attr *attr,
case BPF_CGROUP_SYSCTL:
case BPF_CGROUP_GETSOCKOPT:
case BPF_CGROUP_SETSOCKOPT:
break;
return cgroup_bpf_prog_query(attr, uattr);
case BPF_LIRC_MODE2:
return lirc_prog_query(attr, uattr);
case BPF_FLOW_DISSECTOR:
@ -2742,8 +2740,6 @@ static int bpf_prog_query(const union bpf_attr *attr,
default:
return -EINVAL;
}
return cgroup_bpf_prog_query(attr, uattr);
}
#define BPF_PROG_TEST_RUN_LAST_FIELD test.ctx_out
@ -3564,6 +3560,104 @@ err_put:
return err;
}
#define BPF_LINK_CREATE_LAST_FIELD link_create.flags
static int link_create(union bpf_attr *attr)
{
enum bpf_prog_type ptype;
struct bpf_prog *prog;
int ret;
if (!capable(CAP_NET_ADMIN))
return -EPERM;
if (CHECK_ATTR(BPF_LINK_CREATE))
return -EINVAL;
ptype = attach_type_to_prog_type(attr->link_create.attach_type);
if (ptype == BPF_PROG_TYPE_UNSPEC)
return -EINVAL;
prog = bpf_prog_get_type(attr->link_create.prog_fd, ptype);
if (IS_ERR(prog))
return PTR_ERR(prog);
ret = bpf_prog_attach_check_attach_type(prog,
attr->link_create.attach_type);
if (ret)
goto err_out;
switch (ptype) {
case BPF_PROG_TYPE_CGROUP_SKB:
case BPF_PROG_TYPE_CGROUP_SOCK:
case BPF_PROG_TYPE_CGROUP_SOCK_ADDR:
case BPF_PROG_TYPE_SOCK_OPS:
case BPF_PROG_TYPE_CGROUP_DEVICE:
case BPF_PROG_TYPE_CGROUP_SYSCTL:
case BPF_PROG_TYPE_CGROUP_SOCKOPT:
ret = cgroup_bpf_link_attach(attr, prog);
break;
default:
ret = -EINVAL;
}
err_out:
if (ret < 0)
bpf_prog_put(prog);
return ret;
}
#define BPF_LINK_UPDATE_LAST_FIELD link_update.old_prog_fd
static int link_update(union bpf_attr *attr)
{
struct bpf_prog *old_prog = NULL, *new_prog;
struct bpf_link *link;
u32 flags;
int ret;
if (!capable(CAP_NET_ADMIN))
return -EPERM;
if (CHECK_ATTR(BPF_LINK_UPDATE))
return -EINVAL;
flags = attr->link_update.flags;
if (flags & ~BPF_F_REPLACE)
return -EINVAL;
link = bpf_link_get_from_fd(attr->link_update.link_fd);
if (IS_ERR(link))
return PTR_ERR(link);
new_prog = bpf_prog_get(attr->link_update.new_prog_fd);
if (IS_ERR(new_prog))
return PTR_ERR(new_prog);
if (flags & BPF_F_REPLACE) {
old_prog = bpf_prog_get(attr->link_update.old_prog_fd);
if (IS_ERR(old_prog)) {
ret = PTR_ERR(old_prog);
old_prog = NULL;
goto out_put_progs;
}
}
#ifdef CONFIG_CGROUP_BPF
if (link->ops == &bpf_cgroup_link_lops) {
ret = cgroup_bpf_replace(link, old_prog, new_prog);
goto out_put_progs;
}
#endif
ret = -EINVAL;
out_put_progs:
if (old_prog)
bpf_prog_put(old_prog);
if (ret)
bpf_prog_put(new_prog);
return ret;
}
SYSCALL_DEFINE3(bpf, int, cmd, union bpf_attr __user *, uattr, unsigned int, size)
{
union bpf_attr attr;
@ -3675,6 +3769,12 @@ SYSCALL_DEFINE3(bpf, int, cmd, union bpf_attr __user *, uattr, unsigned int, siz
case BPF_MAP_DELETE_BATCH:
err = bpf_map_do_batch(&attr, uattr, BPF_MAP_DELETE_BATCH);
break;
case BPF_LINK_CREATE:
err = link_create(&attr);
break;
case BPF_LINK_UPDATE:
err = link_update(&attr);
break;
default:
err = -EINVAL;
break;

View File

@ -9,15 +9,15 @@
#include <linux/sysfs.h>
/* See scripts/link-vmlinux.sh, gen_btf() func for details */
extern char __weak _binary__btf_vmlinux_bin_start[];
extern char __weak _binary__btf_vmlinux_bin_end[];
extern char __weak __start_BTF[];
extern char __weak __stop_BTF[];
static ssize_t
btf_vmlinux_read(struct file *file, struct kobject *kobj,
struct bin_attribute *bin_attr,
char *buf, loff_t off, size_t len)
{
memcpy(buf, _binary__btf_vmlinux_bin_start + off, len);
memcpy(buf, __start_BTF + off, len);
return len;
}
@ -30,15 +30,14 @@ static struct kobject *btf_kobj;
static int __init btf_vmlinux_init(void)
{
if (!_binary__btf_vmlinux_bin_start)
if (!__start_BTF)
return 0;
btf_kobj = kobject_create_and_add("btf", kernel_kobj);
if (!btf_kobj)
return -ENOMEM;
bin_attr_btf_vmlinux.size = _binary__btf_vmlinux_bin_end -
_binary__btf_vmlinux_bin_start;
bin_attr_btf_vmlinux.size = __stop_BTF - __start_BTF;
return sysfs_create_bin_file(btf_kobj, &bin_attr_btf_vmlinux);
}

View File

@ -194,3 +194,18 @@ int tnum_sbin(char *str, size_t size, struct tnum a)
str[min(size - 1, (size_t)64)] = 0;
return 64;
}
struct tnum tnum_subreg(struct tnum a)
{
return tnum_cast(a, 4);
}
struct tnum tnum_clear_subreg(struct tnum a)
{
return tnum_lshift(tnum_rshift(a, 32), 32);
}
struct tnum tnum_const_subreg(struct tnum a, u32 value)
{
return tnum_or(tnum_clear_subreg(a), tnum_const(value));
}

View File

@ -6,6 +6,7 @@
#include <linux/ftrace.h>
#include <linux/rbtree_latch.h>
#include <linux/perf_event.h>
#include <linux/btf.h>
/* dummy _ops. The verifier will operate on target program's ops. */
const struct bpf_verifier_ops bpf_extension_verifier_ops = {
@ -233,15 +234,23 @@ out:
return err;
}
static enum bpf_tramp_prog_type bpf_attach_type_to_tramp(enum bpf_attach_type t)
static enum bpf_tramp_prog_type bpf_attach_type_to_tramp(struct bpf_prog *prog)
{
switch (t) {
switch (prog->expected_attach_type) {
case BPF_TRACE_FENTRY:
return BPF_TRAMP_FENTRY;
case BPF_MODIFY_RETURN:
return BPF_TRAMP_MODIFY_RETURN;
case BPF_TRACE_FEXIT:
return BPF_TRAMP_FEXIT;
case BPF_LSM_MAC:
if (!prog->aux->attach_func_proto->type)
/* The function returns void, we cannot modify its
* return value.
*/
return BPF_TRAMP_FEXIT;
else
return BPF_TRAMP_MODIFY_RETURN;
default:
return BPF_TRAMP_REPLACE;
}
@ -255,7 +264,7 @@ int bpf_trampoline_link_prog(struct bpf_prog *prog)
int cnt;
tr = prog->aux->trampoline;
kind = bpf_attach_type_to_tramp(prog->expected_attach_type);
kind = bpf_attach_type_to_tramp(prog);
mutex_lock(&tr->mutex);
if (tr->extension_prog) {
/* cannot attach fentry/fexit if extension prog is attached.
@ -305,7 +314,7 @@ int bpf_trampoline_unlink_prog(struct bpf_prog *prog)
int err;
tr = prog->aux->trampoline;
kind = bpf_attach_type_to_tramp(prog->expected_attach_type);
kind = bpf_attach_type_to_tramp(prog);
mutex_lock(&tr->mutex);
if (kind == BPF_TRAMP_REPLACE) {
WARN_ON_ONCE(!tr->extension_prog);

File diff suppressed because it is too large Load Diff

View File

@ -6303,27 +6303,58 @@ void cgroup_sk_free(struct sock_cgroup_data *skcd)
#endif /* CONFIG_SOCK_CGROUP_DATA */
#ifdef CONFIG_CGROUP_BPF
int cgroup_bpf_attach(struct cgroup *cgrp, struct bpf_prog *prog,
struct bpf_prog *replace_prog, enum bpf_attach_type type,
int cgroup_bpf_attach(struct cgroup *cgrp,
struct bpf_prog *prog, struct bpf_prog *replace_prog,
struct bpf_cgroup_link *link,
enum bpf_attach_type type,
u32 flags)
{
int ret;
mutex_lock(&cgroup_mutex);
ret = __cgroup_bpf_attach(cgrp, prog, replace_prog, type, flags);
ret = __cgroup_bpf_attach(cgrp, prog, replace_prog, link, type, flags);
mutex_unlock(&cgroup_mutex);
return ret;
}
int cgroup_bpf_replace(struct bpf_link *link, struct bpf_prog *old_prog,
struct bpf_prog *new_prog)
{
struct bpf_cgroup_link *cg_link;
int ret;
if (link->ops != &bpf_cgroup_link_lops)
return -EINVAL;
cg_link = container_of(link, struct bpf_cgroup_link, link);
mutex_lock(&cgroup_mutex);
/* link might have been auto-released by dying cgroup, so fail */
if (!cg_link->cgroup) {
ret = -EINVAL;
goto out_unlock;
}
if (old_prog && link->prog != old_prog) {
ret = -EPERM;
goto out_unlock;
}
ret = __cgroup_bpf_replace(cg_link->cgroup, cg_link, new_prog);
out_unlock:
mutex_unlock(&cgroup_mutex);
return ret;
}
int cgroup_bpf_detach(struct cgroup *cgrp, struct bpf_prog *prog,
enum bpf_attach_type type, u32 flags)
enum bpf_attach_type type)
{
int ret;
mutex_lock(&cgroup_mutex);
ret = __cgroup_bpf_detach(cgrp, prog, type);
ret = __cgroup_bpf_detach(cgrp, prog, NULL, type);
mutex_unlock(&cgroup_mutex);
return ret;
}
int cgroup_bpf_query(struct cgroup *cgrp, const union bpf_attr *attr,
union bpf_attr __user *uattr)
{

View File

@ -779,8 +779,8 @@ static const struct bpf_func_proto bpf_send_signal_thread_proto = {
.arg1_type = ARG_ANYTHING,
};
static const struct bpf_func_proto *
tracing_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
const struct bpf_func_proto *
bpf_tracing_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
{
switch (func_id) {
case BPF_FUNC_map_lookup_elem:
@ -865,7 +865,7 @@ kprobe_prog_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
return &bpf_override_return_proto;
#endif
default:
return tracing_func_proto(func_id, prog);
return bpf_tracing_func_proto(func_id, prog);
}
}
@ -975,7 +975,7 @@ tp_prog_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
case BPF_FUNC_get_stack:
return &bpf_get_stack_proto_tp;
default:
return tracing_func_proto(func_id, prog);
return bpf_tracing_func_proto(func_id, prog);
}
}
@ -1082,7 +1082,7 @@ pe_prog_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
case BPF_FUNC_read_branch_records:
return &bpf_read_branch_records_proto;
default:
return tracing_func_proto(func_id, prog);
return bpf_tracing_func_proto(func_id, prog);
}
}
@ -1210,7 +1210,7 @@ raw_tp_prog_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
case BPF_FUNC_get_stack:
return &bpf_get_stack_proto_raw_tp;
default:
return tracing_func_proto(func_id, prog);
return bpf_tracing_func_proto(func_id, prog);
}
}

View File

@ -114,6 +114,9 @@ out:
* architecture dependent calling conventions. 7+ can be supported in the
* future.
*/
__diag_push();
__diag_ignore(GCC, 8, "-Wmissing-prototypes",
"Global functions as their definitions will be in vmlinux BTF");
int noinline bpf_fentry_test1(int a)
{
return a + 1;
@ -149,6 +152,7 @@ int noinline bpf_modify_return_test(int a, int *b)
*b += 1;
return a + *b;
}
__diag_pop();
ALLOW_ERROR_INJECTION(bpf_modify_return_test, ERRNO);

View File

@ -8655,15 +8655,17 @@ static void dev_xdp_uninstall(struct net_device *dev)
* @dev: device
* @extack: netlink extended ack
* @fd: new program fd or negative value to clear
* @expected_fd: old program fd that userspace expects to replace or clear
* @flags: xdp-related flags
*
* Set or clear a bpf program for a device
*/
int dev_change_xdp_fd(struct net_device *dev, struct netlink_ext_ack *extack,
int fd, u32 flags)
int fd, int expected_fd, u32 flags)
{
const struct net_device_ops *ops = dev->netdev_ops;
enum bpf_netdev_command query;
u32 prog_id, expected_id = 0;
struct bpf_prog *prog = NULL;
bpf_op_t bpf_op, bpf_chk;
bool offload;
@ -8684,15 +8686,29 @@ int dev_change_xdp_fd(struct net_device *dev, struct netlink_ext_ack *extack,
if (bpf_op == bpf_chk)
bpf_chk = generic_xdp_install;
if (fd >= 0) {
u32 prog_id;
prog_id = __dev_xdp_query(dev, bpf_op, query);
if (flags & XDP_FLAGS_REPLACE) {
if (expected_fd >= 0) {
prog = bpf_prog_get_type_dev(expected_fd,
BPF_PROG_TYPE_XDP,
bpf_op == ops->ndo_bpf);
if (IS_ERR(prog))
return PTR_ERR(prog);
expected_id = prog->aux->id;
bpf_prog_put(prog);
}
if (prog_id != expected_id) {
NL_SET_ERR_MSG(extack, "Active program does not match expected");
return -EEXIST;
}
}
if (fd >= 0) {
if (!offload && __dev_xdp_query(dev, bpf_chk, XDP_QUERY_PROG)) {
NL_SET_ERR_MSG(extack, "native and generic XDP can't be active at the same time");
return -EEXIST;
}
prog_id = __dev_xdp_query(dev, bpf_op, query);
if ((flags & XDP_FLAGS_UPDATE_IF_NOEXIST) && prog_id) {
NL_SET_ERR_MSG(extack, "XDP program already attached");
return -EBUSY;
@ -8715,7 +8731,7 @@ int dev_change_xdp_fd(struct net_device *dev, struct netlink_ext_ack *extack,
return 0;
}
} else {
if (!__dev_xdp_query(dev, bpf_op, query))
if (!prog_id)
return 0;
}

View File

@ -2642,6 +2642,19 @@ static const struct bpf_func_proto bpf_msg_pop_data_proto = {
.arg4_type = ARG_ANYTHING,
};
#ifdef CONFIG_CGROUP_NET_CLASSID
BPF_CALL_0(bpf_get_cgroup_classid_curr)
{
return __task_get_classid(current);
}
static const struct bpf_func_proto bpf_get_cgroup_classid_curr_proto = {
.func = bpf_get_cgroup_classid_curr,
.gpl_only = false,
.ret_type = RET_INTEGER,
};
#endif
BPF_CALL_1(bpf_get_cgroup_classid, const struct sk_buff *, skb)
{
return task_get_classid(skb);
@ -4117,6 +4130,18 @@ static const struct bpf_func_proto bpf_get_socket_cookie_sock_addr_proto = {
.arg1_type = ARG_PTR_TO_CTX,
};
BPF_CALL_1(bpf_get_socket_cookie_sock, struct sock *, ctx)
{
return sock_gen_cookie(ctx);
}
static const struct bpf_func_proto bpf_get_socket_cookie_sock_proto = {
.func = bpf_get_socket_cookie_sock,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_CTX,
};
BPF_CALL_1(bpf_get_socket_cookie_sock_ops, struct bpf_sock_ops_kern *, ctx)
{
return sock_gen_cookie(ctx->sk);
@ -4129,6 +4154,39 @@ static const struct bpf_func_proto bpf_get_socket_cookie_sock_ops_proto = {
.arg1_type = ARG_PTR_TO_CTX,
};
static u64 __bpf_get_netns_cookie(struct sock *sk)
{
#ifdef CONFIG_NET_NS
return net_gen_cookie(sk ? sk->sk_net.net : &init_net);
#else
return 0;
#endif
}
BPF_CALL_1(bpf_get_netns_cookie_sock, struct sock *, ctx)
{
return __bpf_get_netns_cookie(ctx);
}
static const struct bpf_func_proto bpf_get_netns_cookie_sock_proto = {
.func = bpf_get_netns_cookie_sock,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_CTX_OR_NULL,
};
BPF_CALL_1(bpf_get_netns_cookie_sock_addr, struct bpf_sock_addr_kern *, ctx)
{
return __bpf_get_netns_cookie(ctx ? ctx->sk : NULL);
}
static const struct bpf_func_proto bpf_get_netns_cookie_sock_addr_proto = {
.func = bpf_get_netns_cookie_sock_addr,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_CTX_OR_NULL,
};
BPF_CALL_1(bpf_get_socket_uid, struct sk_buff *, skb)
{
struct sock *sk = sk_to_full_sk(skb->sk);
@ -4147,8 +4205,8 @@ static const struct bpf_func_proto bpf_get_socket_uid_proto = {
.arg1_type = ARG_PTR_TO_CTX,
};
BPF_CALL_5(bpf_sockopt_event_output, struct bpf_sock_ops_kern *, bpf_sock,
struct bpf_map *, map, u64, flags, void *, data, u64, size)
BPF_CALL_5(bpf_event_output_data, void *, ctx, struct bpf_map *, map, u64, flags,
void *, data, u64, size)
{
if (unlikely(flags & ~(BPF_F_INDEX_MASK)))
return -EINVAL;
@ -4156,8 +4214,8 @@ BPF_CALL_5(bpf_sockopt_event_output, struct bpf_sock_ops_kern *, bpf_sock,
return bpf_event_output(map, flags, data, size, NULL, 0, NULL);
}
static const struct bpf_func_proto bpf_sockopt_event_output_proto = {
.func = bpf_sockopt_event_output,
static const struct bpf_func_proto bpf_event_output_data_proto = {
.func = bpf_event_output_data,
.gpl_only = true,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_CTX,
@ -5343,8 +5401,7 @@ static const struct bpf_func_proto bpf_sk_lookup_udp_proto = {
BPF_CALL_1(bpf_sk_release, struct sock *, sk)
{
/* Only full sockets have sk->sk_flags. */
if (!sk_fullsock(sk) || !sock_flag(sk, SOCK_RCU_FREE))
if (sk_is_refcounted(sk))
sock_gen_put(sk);
return 0;
}
@ -5860,6 +5917,36 @@ static const struct bpf_func_proto bpf_tcp_gen_syncookie_proto = {
.arg5_type = ARG_CONST_SIZE,
};
BPF_CALL_3(bpf_sk_assign, struct sk_buff *, skb, struct sock *, sk, u64, flags)
{
if (flags != 0)
return -EINVAL;
if (!skb_at_tc_ingress(skb))
return -EOPNOTSUPP;
if (unlikely(dev_net(skb->dev) != sock_net(sk)))
return -ENETUNREACH;
if (unlikely(sk->sk_reuseport))
return -ESOCKTNOSUPPORT;
if (sk_is_refcounted(sk) &&
unlikely(!refcount_inc_not_zero(&sk->sk_refcnt)))
return -ENOENT;
skb_orphan(skb);
skb->sk = sk;
skb->destructor = sock_pfree;
return 0;
}
static const struct bpf_func_proto bpf_sk_assign_proto = {
.func = bpf_sk_assign,
.gpl_only = false,
.ret_type = RET_INTEGER,
.arg1_type = ARG_PTR_TO_CTX,
.arg2_type = ARG_PTR_TO_SOCK_COMMON,
.arg3_type = ARG_ANYTHING,
};
#endif /* CONFIG_INET */
bool bpf_helper_changes_pkt_data(void *func)
@ -5954,6 +6041,26 @@ sock_filter_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
return &bpf_get_current_uid_gid_proto;
case BPF_FUNC_get_local_storage:
return &bpf_get_local_storage_proto;
case BPF_FUNC_get_socket_cookie:
return &bpf_get_socket_cookie_sock_proto;
case BPF_FUNC_get_netns_cookie:
return &bpf_get_netns_cookie_sock_proto;
case BPF_FUNC_perf_event_output:
return &bpf_event_output_data_proto;
case BPF_FUNC_get_current_pid_tgid:
return &bpf_get_current_pid_tgid_proto;
case BPF_FUNC_get_current_comm:
return &bpf_get_current_comm_proto;
#ifdef CONFIG_CGROUPS
case BPF_FUNC_get_current_cgroup_id:
return &bpf_get_current_cgroup_id_proto;
case BPF_FUNC_get_current_ancestor_cgroup_id:
return &bpf_get_current_ancestor_cgroup_id_proto;
#endif
#ifdef CONFIG_CGROUP_NET_CLASSID
case BPF_FUNC_get_cgroup_classid:
return &bpf_get_cgroup_classid_curr_proto;
#endif
default:
return bpf_base_func_proto(func_id);
}
@ -5978,8 +6085,26 @@ sock_addr_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
}
case BPF_FUNC_get_socket_cookie:
return &bpf_get_socket_cookie_sock_addr_proto;
case BPF_FUNC_get_netns_cookie:
return &bpf_get_netns_cookie_sock_addr_proto;
case BPF_FUNC_get_local_storage:
return &bpf_get_local_storage_proto;
case BPF_FUNC_perf_event_output:
return &bpf_event_output_data_proto;
case BPF_FUNC_get_current_pid_tgid:
return &bpf_get_current_pid_tgid_proto;
case BPF_FUNC_get_current_comm:
return &bpf_get_current_comm_proto;
#ifdef CONFIG_CGROUPS
case BPF_FUNC_get_current_cgroup_id:
return &bpf_get_current_cgroup_id_proto;
case BPF_FUNC_get_current_ancestor_cgroup_id:
return &bpf_get_current_ancestor_cgroup_id_proto;
#endif
#ifdef CONFIG_CGROUP_NET_CLASSID
case BPF_FUNC_get_cgroup_classid:
return &bpf_get_cgroup_classid_curr_proto;
#endif
#ifdef CONFIG_INET
case BPF_FUNC_sk_lookup_tcp:
return &bpf_sock_addr_sk_lookup_tcp_proto;
@ -6153,6 +6278,8 @@ tc_cls_act_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
return &bpf_skb_ecn_set_ce_proto;
case BPF_FUNC_tcp_gen_syncookie:
return &bpf_tcp_gen_syncookie_proto;
case BPF_FUNC_sk_assign:
return &bpf_sk_assign_proto;
#endif
default:
return bpf_base_func_proto(func_id);
@ -6222,7 +6349,7 @@ sock_ops_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
case BPF_FUNC_get_local_storage:
return &bpf_get_local_storage_proto;
case BPF_FUNC_perf_event_output:
return &bpf_sockopt_event_output_proto;
return &bpf_event_output_data_proto;
case BPF_FUNC_sk_storage_get:
return &bpf_sk_storage_get_proto;
case BPF_FUNC_sk_storage_delete:

View File

@ -69,6 +69,20 @@ EXPORT_SYMBOL_GPL(pernet_ops_rwsem);
static unsigned int max_gen_ptrs = INITIAL_NET_GEN_PTRS;
static atomic64_t cookie_gen;
u64 net_gen_cookie(struct net *net)
{
while (1) {
u64 res = atomic64_read(&net->net_cookie);
if (res)
return res;
res = atomic64_inc_return(&cookie_gen);
atomic64_cmpxchg(&net->net_cookie, 0, res);
}
}
static struct net_generic *net_alloc_generic(void)
{
struct net_generic *ng;
@ -1087,6 +1101,7 @@ static int __init net_ns_init(void)
panic("Could not allocate generic netns");
rcu_assign_pointer(init_net.gen, ng);
net_gen_cookie(&init_net);
down_write(&pernet_ops_rwsem);
if (setup_net(&init_net, &init_user_ns))

View File

@ -1872,7 +1872,9 @@ static const struct nla_policy ifla_port_policy[IFLA_PORT_MAX+1] = {
};
static const struct nla_policy ifla_xdp_policy[IFLA_XDP_MAX + 1] = {
[IFLA_XDP_UNSPEC] = { .strict_start_type = IFLA_XDP_EXPECTED_FD },
[IFLA_XDP_FD] = { .type = NLA_S32 },
[IFLA_XDP_EXPECTED_FD] = { .type = NLA_S32 },
[IFLA_XDP_ATTACHED] = { .type = NLA_U8 },
[IFLA_XDP_FLAGS] = { .type = NLA_U32 },
[IFLA_XDP_PROG_ID] = { .type = NLA_U32 },
@ -2799,8 +2801,20 @@ static int do_setlink(const struct sk_buff *skb,
}
if (xdp[IFLA_XDP_FD]) {
int expected_fd = -1;
if (xdp_flags & XDP_FLAGS_REPLACE) {
if (!xdp[IFLA_XDP_EXPECTED_FD]) {
err = -EINVAL;
goto errout;
}
expected_fd =
nla_get_s32(xdp[IFLA_XDP_EXPECTED_FD]);
}
err = dev_change_xdp_fd(dev, extack,
nla_get_s32(xdp[IFLA_XDP_FD]),
expected_fd,
xdp_flags);
if (err)
goto errout;

View File

@ -2071,6 +2071,18 @@ void sock_efree(struct sk_buff *skb)
}
EXPORT_SYMBOL(sock_efree);
/* Buffer destructor for prefetch/receive path where reference count may
* not be held, e.g. for listen sockets.
*/
#ifdef CONFIG_INET
void sock_pfree(struct sk_buff *skb)
{
if (sk_is_refcounted(skb->sk))
sock_gen_put(skb->sk);
}
EXPORT_SYMBOL(sock_pfree);
#endif /* CONFIG_INET */
kuid_t sock_i_uid(struct sock *sk)
{
kuid_t uid;

View File

@ -7,6 +7,7 @@
#include <linux/btf.h>
#include <linux/filter.h>
#include <net/tcp.h>
#include <net/bpf_sk_storage.h>
static u32 optional_ops[] = {
offsetof(struct tcp_congestion_ops, init),
@ -27,6 +28,27 @@ static u32 unsupported_ops[] = {
static const struct btf_type *tcp_sock_type;
static u32 tcp_sock_id, sock_id;
static int btf_sk_storage_get_ids[5];
static struct bpf_func_proto btf_sk_storage_get_proto __read_mostly;
static int btf_sk_storage_delete_ids[5];
static struct bpf_func_proto btf_sk_storage_delete_proto __read_mostly;
static void convert_sk_func_proto(struct bpf_func_proto *to, int *to_btf_ids,
const struct bpf_func_proto *from)
{
int i;
*to = *from;
to->btf_id = to_btf_ids;
for (i = 0; i < ARRAY_SIZE(to->arg_type); i++) {
if (to->arg_type[i] == ARG_PTR_TO_SOCKET) {
to->arg_type[i] = ARG_PTR_TO_BTF_ID;
to->btf_id[i] = tcp_sock_id;
}
}
}
static int bpf_tcp_ca_init(struct btf *btf)
{
s32 type_id;
@ -42,6 +64,13 @@ static int bpf_tcp_ca_init(struct btf *btf)
tcp_sock_id = type_id;
tcp_sock_type = btf_type_by_id(btf, tcp_sock_id);
convert_sk_func_proto(&btf_sk_storage_get_proto,
btf_sk_storage_get_ids,
&bpf_sk_storage_get_proto);
convert_sk_func_proto(&btf_sk_storage_delete_proto,
btf_sk_storage_delete_ids,
&bpf_sk_storage_delete_proto);
return 0;
}
@ -167,6 +196,10 @@ bpf_tcp_ca_get_func_proto(enum bpf_func_id func_id,
switch (func_id) {
case BPF_FUNC_tcp_send_ack:
return &bpf_tcp_send_ack_proto;
case BPF_FUNC_sk_storage_get:
return &btf_sk_storage_get_proto;
case BPF_FUNC_sk_storage_delete:
return &btf_sk_storage_delete_proto;
default:
return bpf_base_func_proto(func_id);
}

View File

@ -509,7 +509,8 @@ static struct sk_buff *ip_rcv_core(struct sk_buff *skb, struct net *net)
IPCB(skb)->iif = skb->skb_iif;
/* Must drop socket now because of tproxy. */
skb_orphan(skb);
if (!skb_sk_is_prefetched(skb))
skb_orphan(skb);
return skb;

View File

@ -10,38 +10,6 @@
#include <net/inet_common.h>
#include <net/tls.h>
static bool tcp_bpf_stream_read(const struct sock *sk)
{
struct sk_psock *psock;
bool empty = true;
rcu_read_lock();
psock = sk_psock(sk);
if (likely(psock))
empty = list_empty(&psock->ingress_msg);
rcu_read_unlock();
return !empty;
}
static int tcp_bpf_wait_data(struct sock *sk, struct sk_psock *psock,
int flags, long timeo, int *err)
{
DEFINE_WAIT_FUNC(wait, woken_wake_function);
int ret = 0;
if (!timeo)
return ret;
add_wait_queue(sk_sleep(sk), &wait);
sk_set_bit(SOCKWQ_ASYNC_WAITDATA, sk);
ret = sk_wait_event(sk, &timeo,
!list_empty(&psock->ingress_msg) ||
!skb_queue_empty(&sk->sk_receive_queue), &wait);
sk_clear_bit(SOCKWQ_ASYNC_WAITDATA, sk);
remove_wait_queue(sk_sleep(sk), &wait);
return ret;
}
int __tcp_bpf_recvmsg(struct sock *sk, struct sk_psock *psock,
struct msghdr *msg, int len, int flags)
{
@ -115,49 +83,6 @@ int __tcp_bpf_recvmsg(struct sock *sk, struct sk_psock *psock,
}
EXPORT_SYMBOL_GPL(__tcp_bpf_recvmsg);
int tcp_bpf_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
int nonblock, int flags, int *addr_len)
{
struct sk_psock *psock;
int copied, ret;
psock = sk_psock_get(sk);
if (unlikely(!psock))
return tcp_recvmsg(sk, msg, len, nonblock, flags, addr_len);
if (unlikely(flags & MSG_ERRQUEUE))
return inet_recv_error(sk, msg, len, addr_len);
if (!skb_queue_empty(&sk->sk_receive_queue) &&
sk_psock_queue_empty(psock))
return tcp_recvmsg(sk, msg, len, nonblock, flags, addr_len);
lock_sock(sk);
msg_bytes_ready:
copied = __tcp_bpf_recvmsg(sk, psock, msg, len, flags);
if (!copied) {
int data, err = 0;
long timeo;
timeo = sock_rcvtimeo(sk, nonblock);
data = tcp_bpf_wait_data(sk, psock, flags, timeo, &err);
if (data) {
if (!sk_psock_queue_empty(psock))
goto msg_bytes_ready;
release_sock(sk);
sk_psock_put(sk, psock);
return tcp_recvmsg(sk, msg, len, nonblock, flags, addr_len);
}
if (err) {
ret = err;
goto out;
}
copied = -EAGAIN;
}
ret = copied;
out:
release_sock(sk);
sk_psock_put(sk, psock);
return ret;
}
static int bpf_tcp_ingress(struct sock *sk, struct sk_psock *psock,
struct sk_msg *msg, u32 apply_bytes, int flags)
{
@ -298,6 +223,82 @@ int tcp_bpf_sendmsg_redir(struct sock *sk, struct sk_msg *msg,
}
EXPORT_SYMBOL_GPL(tcp_bpf_sendmsg_redir);
#ifdef CONFIG_BPF_STREAM_PARSER
static bool tcp_bpf_stream_read(const struct sock *sk)
{
struct sk_psock *psock;
bool empty = true;
rcu_read_lock();
psock = sk_psock(sk);
if (likely(psock))
empty = list_empty(&psock->ingress_msg);
rcu_read_unlock();
return !empty;
}
static int tcp_bpf_wait_data(struct sock *sk, struct sk_psock *psock,
int flags, long timeo, int *err)
{
DEFINE_WAIT_FUNC(wait, woken_wake_function);
int ret = 0;
if (!timeo)
return ret;
add_wait_queue(sk_sleep(sk), &wait);
sk_set_bit(SOCKWQ_ASYNC_WAITDATA, sk);
ret = sk_wait_event(sk, &timeo,
!list_empty(&psock->ingress_msg) ||
!skb_queue_empty(&sk->sk_receive_queue), &wait);
sk_clear_bit(SOCKWQ_ASYNC_WAITDATA, sk);
remove_wait_queue(sk_sleep(sk), &wait);
return ret;
}
static int tcp_bpf_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
int nonblock, int flags, int *addr_len)
{
struct sk_psock *psock;
int copied, ret;
psock = sk_psock_get(sk);
if (unlikely(!psock))
return tcp_recvmsg(sk, msg, len, nonblock, flags, addr_len);
if (unlikely(flags & MSG_ERRQUEUE))
return inet_recv_error(sk, msg, len, addr_len);
if (!skb_queue_empty(&sk->sk_receive_queue) &&
sk_psock_queue_empty(psock))
return tcp_recvmsg(sk, msg, len, nonblock, flags, addr_len);
lock_sock(sk);
msg_bytes_ready:
copied = __tcp_bpf_recvmsg(sk, psock, msg, len, flags);
if (!copied) {
int data, err = 0;
long timeo;
timeo = sock_rcvtimeo(sk, nonblock);
data = tcp_bpf_wait_data(sk, psock, flags, timeo, &err);
if (data) {
if (!sk_psock_queue_empty(psock))
goto msg_bytes_ready;
release_sock(sk);
sk_psock_put(sk, psock);
return tcp_recvmsg(sk, msg, len, nonblock, flags, addr_len);
}
if (err) {
ret = err;
goto out;
}
copied = -EAGAIN;
}
ret = copied;
out:
release_sock(sk);
sk_psock_put(sk, psock);
return ret;
}
static int tcp_bpf_send_verdict(struct sock *sk, struct sk_psock *psock,
struct sk_msg *msg, int *copied, int flags)
{
@ -528,7 +529,6 @@ out_err:
return copied ? copied : err;
}
#ifdef CONFIG_BPF_STREAM_PARSER
enum {
TCP_BPF_IPV4,
TCP_BPF_IPV6,

View File

@ -2288,6 +2288,7 @@ int __udp4_lib_rcv(struct sk_buff *skb, struct udp_table *udptable,
struct rtable *rt = skb_rtable(skb);
__be32 saddr, daddr;
struct net *net = dev_net(skb->dev);
bool refcounted;
/*
* Validate the packet.
@ -2313,7 +2314,7 @@ int __udp4_lib_rcv(struct sk_buff *skb, struct udp_table *udptable,
if (udp4_csum_init(skb, uh, proto))
goto csum_error;
sk = skb_steal_sock(skb);
sk = skb_steal_sock(skb, &refcounted);
if (sk) {
struct dst_entry *dst = skb_dst(skb);
int ret;
@ -2322,7 +2323,8 @@ int __udp4_lib_rcv(struct sk_buff *skb, struct udp_table *udptable,
udp_sk_rx_dst_set(sk, dst);
ret = udp_unicast_rcv_skb(sk, skb, uh);
sock_put(sk);
if (refcounted)
sock_put(sk);
return ret;
}

View File

@ -285,7 +285,8 @@ static struct sk_buff *ip6_rcv_core(struct sk_buff *skb, struct net_device *dev,
rcu_read_unlock();
/* Must drop socket now because of tproxy. */
skb_orphan(skb);
if (!skb_sk_is_prefetched(skb))
skb_orphan(skb);
return skb;
err:

View File

@ -843,6 +843,7 @@ int __udp6_lib_rcv(struct sk_buff *skb, struct udp_table *udptable,
struct net *net = dev_net(skb->dev);
struct udphdr *uh;
struct sock *sk;
bool refcounted;
u32 ulen = 0;
if (!pskb_may_pull(skb, sizeof(struct udphdr)))
@ -879,7 +880,7 @@ int __udp6_lib_rcv(struct sk_buff *skb, struct udp_table *udptable,
goto csum_error;
/* Check if the socket is already available, e.g. due to early demux */
sk = skb_steal_sock(skb);
sk = skb_steal_sock(skb, &refcounted);
if (sk) {
struct dst_entry *dst = skb_dst(skb);
int ret;
@ -888,12 +889,14 @@ int __udp6_lib_rcv(struct sk_buff *skb, struct udp_table *udptable,
udp6_sk_rx_dst_set(sk, dst);
if (!uh->check && !udp_sk(sk)->no_check6_rx) {
sock_put(sk);
if (refcounted)
sock_put(sk);
goto report_csum_error;
}
ret = udp6_unicast_rcv_skb(sk, skb, uh);
sock_put(sk);
if (refcounted)
sock_put(sk);
return ret;
}

View File

@ -12,6 +12,7 @@
#include <linux/bpf.h>
#include <net/netlink.h>
#include <net/sock.h>
#include <net/pkt_sched.h>
#include <net/pkt_cls.h>
@ -53,6 +54,8 @@ static int tcf_bpf_act(struct sk_buff *skb, const struct tc_action *act,
bpf_compute_data_pointers(skb);
filter_res = BPF_PROG_RUN(filter, skb);
}
if (skb_sk_is_prefetched(skb) && filter_res != TC_ACT_OK)
skb_orphan(skb);
rcu_read_unlock();
/* A BPF program may overwrite the default action opcode.

View File

@ -64,11 +64,11 @@ fds_example-objs := fds_example.o
sockex1-objs := sockex1_user.o
sockex2-objs := sockex2_user.o
sockex3-objs := bpf_load.o sockex3_user.o
tracex1-objs := bpf_load.o tracex1_user.o
tracex1-objs := bpf_load.o tracex1_user.o $(TRACE_HELPERS)
tracex2-objs := bpf_load.o tracex2_user.o
tracex3-objs := bpf_load.o tracex3_user.o
tracex4-objs := bpf_load.o tracex4_user.o
tracex5-objs := bpf_load.o tracex5_user.o
tracex5-objs := bpf_load.o tracex5_user.o $(TRACE_HELPERS)
tracex6-objs := bpf_load.o tracex6_user.o
tracex7-objs := bpf_load.o tracex7_user.o
test_probe_write_user-objs := bpf_load.o test_probe_write_user_user.o
@ -88,8 +88,8 @@ xdp2-objs := xdp1_user.o
xdp_router_ipv4-objs := xdp_router_ipv4_user.o
test_current_task_under_cgroup-objs := bpf_load.o $(CGROUP_HELPERS) \
test_current_task_under_cgroup_user.o
trace_event-objs := bpf_load.o trace_event_user.o $(TRACE_HELPERS)
sampleip-objs := bpf_load.o sampleip_user.o $(TRACE_HELPERS)
trace_event-objs := trace_event_user.o $(TRACE_HELPERS)
sampleip-objs := sampleip_user.o $(TRACE_HELPERS)
tc_l2_redirect-objs := bpf_load.o tc_l2_redirect_user.o
lwt_len_hist-objs := bpf_load.o lwt_len_hist_user.o
xdp_tx_iptunnel-objs := xdp_tx_iptunnel_user.o

View File

@ -665,23 +665,3 @@ int load_bpf_file_fixup_map(const char *path, fixup_map_cb fixup_map)
{
return do_load_bpf_file(path, fixup_map);
}
void read_trace_pipe(void)
{
int trace_fd;
trace_fd = open(DEBUGFS "trace_pipe", O_RDONLY, 0);
if (trace_fd < 0)
return;
while (1) {
static char buf[4096];
ssize_t sz;
sz = read(trace_fd, buf, sizeof(buf) - 1);
if (sz > 0) {
buf[sz] = 0;
puts(buf);
}
}
}

View File

@ -53,6 +53,5 @@ extern int map_data_count;
int load_bpf_file(char *path);
int load_bpf_file_fixup_map(const char *path, fixup_map_cb fixup_map);
void read_trace_pipe(void);
int bpf_set_link_xdp_fd(int ifindex, int fd, __u32 flags);
#endif

View File

@ -10,21 +10,23 @@
#include <errno.h>
#include <signal.h>
#include <string.h>
#include <assert.h>
#include <linux/perf_event.h>
#include <linux/ptrace.h>
#include <linux/bpf.h>
#include <sys/ioctl.h>
#include <bpf/bpf.h>
#include <bpf/libbpf.h>
#include "bpf_load.h"
#include "perf-sys.h"
#include "trace_helpers.h"
#define __must_check
#include <linux/err.h>
#define DEFAULT_FREQ 99
#define DEFAULT_SECS 5
#define MAX_IPS 8192
#define PAGE_OFFSET 0xffff880000000000
static int map_fd;
static int nr_cpus;
static void usage(void)
@ -34,9 +36,10 @@ static void usage(void)
printf(" duration # sampling duration (seconds), default 5\n");
}
static int sampling_start(int *pmu_fd, int freq)
static int sampling_start(int freq, struct bpf_program *prog,
struct bpf_link *links[])
{
int i;
int i, pmu_fd;
struct perf_event_attr pe_sample_attr = {
.type = PERF_TYPE_SOFTWARE,
@ -47,26 +50,30 @@ static int sampling_start(int *pmu_fd, int freq)
};
for (i = 0; i < nr_cpus; i++) {
pmu_fd[i] = sys_perf_event_open(&pe_sample_attr, -1 /* pid */, i,
pmu_fd = sys_perf_event_open(&pe_sample_attr, -1 /* pid */, i,
-1 /* group_fd */, 0 /* flags */);
if (pmu_fd[i] < 0) {
if (pmu_fd < 0) {
fprintf(stderr, "ERROR: Initializing perf sampling\n");
return 1;
}
assert(ioctl(pmu_fd[i], PERF_EVENT_IOC_SET_BPF,
prog_fd[0]) == 0);
assert(ioctl(pmu_fd[i], PERF_EVENT_IOC_ENABLE, 0) == 0);
links[i] = bpf_program__attach_perf_event(prog, pmu_fd);
if (IS_ERR(links[i])) {
fprintf(stderr, "ERROR: Attach perf event\n");
links[i] = NULL;
close(pmu_fd);
return 1;
}
}
return 0;
}
static void sampling_end(int *pmu_fd)
static void sampling_end(struct bpf_link *links[])
{
int i;
for (i = 0; i < nr_cpus; i++)
close(pmu_fd[i]);
bpf_link__destroy(links[i]);
}
struct ipcount {
@ -128,14 +135,17 @@ static void print_ip_map(int fd)
static void int_exit(int sig)
{
printf("\n");
print_ip_map(map_fd[0]);
print_ip_map(map_fd);
exit(0);
}
int main(int argc, char **argv)
{
int opt, freq = DEFAULT_FREQ, secs = DEFAULT_SECS, error = 1;
struct bpf_object *obj = NULL;
struct bpf_program *prog;
struct bpf_link **links;
char filename[256];
int *pmu_fd, opt, freq = DEFAULT_FREQ, secs = DEFAULT_SECS;
/* process arguments */
while ((opt = getopt(argc, argv, "F:h")) != -1) {
@ -163,38 +173,58 @@ int main(int argc, char **argv)
}
/* create perf FDs for each CPU */
nr_cpus = sysconf(_SC_NPROCESSORS_CONF);
pmu_fd = malloc(nr_cpus * sizeof(int));
if (pmu_fd == NULL) {
fprintf(stderr, "ERROR: malloc of pmu_fd\n");
return 1;
nr_cpus = sysconf(_SC_NPROCESSORS_ONLN);
links = calloc(nr_cpus, sizeof(struct bpf_link *));
if (!links) {
fprintf(stderr, "ERROR: malloc of links\n");
goto cleanup;
}
snprintf(filename, sizeof(filename), "%s_kern.o", argv[0]);
obj = bpf_object__open_file(filename, NULL);
if (IS_ERR(obj)) {
fprintf(stderr, "ERROR: opening BPF object file failed\n");
obj = NULL;
goto cleanup;
}
prog = bpf_object__find_program_by_name(obj, "do_sample");
if (!prog) {
fprintf(stderr, "ERROR: finding a prog in obj file failed\n");
goto cleanup;
}
/* load BPF program */
snprintf(filename, sizeof(filename), "%s_kern.o", argv[0]);
if (load_bpf_file(filename)) {
fprintf(stderr, "ERROR: loading BPF program (errno %d):\n",
errno);
if (strcmp(bpf_log_buf, "") == 0)
fprintf(stderr, "Try: ulimit -l unlimited\n");
else
fprintf(stderr, "%s", bpf_log_buf);
return 1;
if (bpf_object__load(obj)) {
fprintf(stderr, "ERROR: loading BPF object file failed\n");
goto cleanup;
}
map_fd = bpf_object__find_map_fd_by_name(obj, "ip_map");
if (map_fd < 0) {
fprintf(stderr, "ERROR: finding a map in obj file failed\n");
goto cleanup;
}
signal(SIGINT, int_exit);
signal(SIGTERM, int_exit);
/* do sampling */
printf("Sampling at %d Hertz for %d seconds. Ctrl-C also ends.\n",
freq, secs);
if (sampling_start(pmu_fd, freq) != 0)
return 1;
if (sampling_start(freq, prog, links) != 0)
goto cleanup;
sleep(secs);
sampling_end(pmu_fd);
free(pmu_fd);
error = 0;
cleanup:
sampling_end(links);
/* output sample counts */
print_ip_map(map_fd[0]);
if (!error)
print_ip_map(map_fd);
return 0;
free(links);
bpf_object__close(obj);
return error;
}

View File

@ -6,22 +6,25 @@
#include <stdlib.h>
#include <stdbool.h>
#include <string.h>
#include <fcntl.h>
#include <poll.h>
#include <sys/ioctl.h>
#include <linux/perf_event.h>
#include <linux/bpf.h>
#include <signal.h>
#include <assert.h>
#include <errno.h>
#include <sys/resource.h>
#include <bpf/bpf.h>
#include <bpf/libbpf.h>
#include "bpf_load.h"
#include "perf-sys.h"
#include "trace_helpers.h"
#define __must_check
#include <linux/err.h>
#define SAMPLE_FREQ 50
static int pid;
/* counts, stackmap */
static int map_fd[2];
struct bpf_program *prog;
static bool sys_read_seen, sys_write_seen;
static void print_ksym(__u64 addr)
@ -91,10 +94,10 @@ static void print_stack(struct key_t *key, __u64 count)
}
}
static void int_exit(int sig)
static void err_exit(int err)
{
kill(0, SIGKILL);
exit(0);
kill(pid, SIGKILL);
exit(err);
}
static void print_stacks(void)
@ -102,7 +105,7 @@ static void print_stacks(void)
struct key_t key = {}, next_key;
__u64 value;
__u32 stackid = 0, next_id;
int fd = map_fd[0], stack_map = map_fd[1];
int error = 1, fd = map_fd[0], stack_map = map_fd[1];
sys_read_seen = sys_write_seen = false;
while (bpf_map_get_next_key(fd, &key, &next_key) == 0) {
@ -114,7 +117,7 @@ static void print_stacks(void)
printf("\n");
if (!sys_read_seen || !sys_write_seen) {
printf("BUG kernel stack doesn't contain sys_read() and sys_write()\n");
int_exit(0);
err_exit(error);
}
/* clear stack map */
@ -136,43 +139,52 @@ static inline int generate_load(void)
static void test_perf_event_all_cpu(struct perf_event_attr *attr)
{
int nr_cpus = sysconf(_SC_NPROCESSORS_CONF);
int *pmu_fd = malloc(nr_cpus * sizeof(int));
int i, error = 0;
int nr_cpus = sysconf(_SC_NPROCESSORS_ONLN);
struct bpf_link **links = calloc(nr_cpus, sizeof(struct bpf_link *));
int i, pmu_fd, error = 1;
if (!links) {
printf("malloc of links failed\n");
goto err;
}
/* system wide perf event, no need to inherit */
attr->inherit = 0;
/* open perf_event on all cpus */
for (i = 0; i < nr_cpus; i++) {
pmu_fd[i] = sys_perf_event_open(attr, -1, i, -1, 0);
if (pmu_fd[i] < 0) {
pmu_fd = sys_perf_event_open(attr, -1, i, -1, 0);
if (pmu_fd < 0) {
printf("sys_perf_event_open failed\n");
error = 1;
goto all_cpu_err;
}
assert(ioctl(pmu_fd[i], PERF_EVENT_IOC_SET_BPF, prog_fd[0]) == 0);
assert(ioctl(pmu_fd[i], PERF_EVENT_IOC_ENABLE) == 0);
links[i] = bpf_program__attach_perf_event(prog, pmu_fd);
if (IS_ERR(links[i])) {
printf("bpf_program__attach_perf_event failed\n");
links[i] = NULL;
close(pmu_fd);
goto all_cpu_err;
}
}
if (generate_load() < 0) {
error = 1;
if (generate_load() < 0)
goto all_cpu_err;
}
print_stacks();
error = 0;
all_cpu_err:
for (i--; i >= 0; i--) {
ioctl(pmu_fd[i], PERF_EVENT_IOC_DISABLE);
close(pmu_fd[i]);
}
free(pmu_fd);
for (i--; i >= 0; i--)
bpf_link__destroy(links[i]);
err:
free(links);
if (error)
int_exit(0);
err_exit(error);
}
static void test_perf_event_task(struct perf_event_attr *attr)
{
int pmu_fd, error = 0;
struct bpf_link *link = NULL;
int pmu_fd, error = 1;
/* per task perf event, enable inherit so the "dd ..." command can be traced properly.
* Enabling inherit will cause bpf_perf_prog_read_time helper failure.
@ -183,21 +195,25 @@ static void test_perf_event_task(struct perf_event_attr *attr)
pmu_fd = sys_perf_event_open(attr, 0, -1, -1, 0);
if (pmu_fd < 0) {
printf("sys_perf_event_open failed\n");
int_exit(0);
}
assert(ioctl(pmu_fd, PERF_EVENT_IOC_SET_BPF, prog_fd[0]) == 0);
assert(ioctl(pmu_fd, PERF_EVENT_IOC_ENABLE) == 0);
if (generate_load() < 0) {
error = 1;
goto err;
}
link = bpf_program__attach_perf_event(prog, pmu_fd);
if (IS_ERR(link)) {
printf("bpf_program__attach_perf_event failed\n");
link = NULL;
close(pmu_fd);
goto err;
}
if (generate_load() < 0)
goto err;
print_stacks();
error = 0;
err:
ioctl(pmu_fd, PERF_EVENT_IOC_DISABLE);
close(pmu_fd);
bpf_link__destroy(link);
if (error)
int_exit(0);
err_exit(error);
}
static void test_bpf_perf_event(void)
@ -282,29 +298,60 @@ static void test_bpf_perf_event(void)
int main(int argc, char **argv)
{
struct rlimit r = {RLIM_INFINITY, RLIM_INFINITY};
struct bpf_object *obj = NULL;
char filename[256];
int error = 1;
snprintf(filename, sizeof(filename), "%s_kern.o", argv[0]);
setrlimit(RLIMIT_MEMLOCK, &r);
signal(SIGINT, int_exit);
signal(SIGTERM, int_exit);
signal(SIGINT, err_exit);
signal(SIGTERM, err_exit);
if (load_kallsyms()) {
printf("failed to process /proc/kallsyms\n");
return 1;
goto cleanup;
}
if (load_bpf_file(filename)) {
printf("%s", bpf_log_buf);
return 2;
obj = bpf_object__open_file(filename, NULL);
if (IS_ERR(obj)) {
printf("opening BPF object file failed\n");
obj = NULL;
goto cleanup;
}
if (fork() == 0) {
prog = bpf_object__find_program_by_name(obj, "bpf_prog1");
if (!prog) {
printf("finding a prog in obj file failed\n");
goto cleanup;
}
/* load BPF program */
if (bpf_object__load(obj)) {
printf("loading BPF object file failed\n");
goto cleanup;
}
map_fd[0] = bpf_object__find_map_fd_by_name(obj, "counts");
map_fd[1] = bpf_object__find_map_fd_by_name(obj, "stackmap");
if (map_fd[0] < 0 || map_fd[1] < 0) {
printf("finding a counts/stackmap map in obj file failed\n");
goto cleanup;
}
pid = fork();
if (pid == 0) {
read_trace_pipe();
return 0;
} else if (pid == -1) {
printf("couldn't spawn process\n");
goto cleanup;
}
test_bpf_perf_event();
int_exit(0);
return 0;
error = 0;
cleanup:
bpf_object__close(obj);
err_exit(error);
}

View File

@ -4,6 +4,7 @@
#include <unistd.h>
#include <bpf/bpf.h>
#include "bpf_load.h"
#include "trace_helpers.h"
int main(int ac, char **argv)
{

View File

@ -8,6 +8,7 @@
#include <bpf/bpf.h>
#include "bpf_load.h"
#include <sys/resource.h>
#include "trace_helpers.h"
/* install fake seccomp program to enable seccomp code path inside the kernel,
* so that our kprobe attached to seccomp_phase1() can be triggered

View File

@ -113,9 +113,6 @@ vmlinux_link()
gen_btf()
{
local pahole_ver
local bin_arch
local bin_format
local bin_file
if ! [ -x "$(command -v ${PAHOLE})" ]; then
echo >&2 "BTF: ${1}: pahole (${PAHOLE}) is not available"
@ -133,17 +130,16 @@ gen_btf()
info "BTF" ${2}
LLVM_OBJCOPY=${OBJCOPY} ${PAHOLE} -J ${1}
# dump .BTF section into raw binary file to link with final vmlinux
bin_arch=$(LANG=C ${OBJDUMP} -f ${1} | grep architecture | \
cut -d, -f1 | cut -d' ' -f2)
bin_format=$(LANG=C ${OBJDUMP} -f ${1} | grep 'file format' | \
awk '{print $4}')
bin_file=.btf.vmlinux.bin
${OBJCOPY} --change-section-address .BTF=0 \
--set-section-flags .BTF=alloc -O binary \
--only-section=.BTF ${1} $bin_file
${OBJCOPY} -I binary -O ${bin_format} -B ${bin_arch} \
--rename-section .data=.BTF $bin_file ${2}
# Create ${2} which contains just .BTF section but no symbols. Add
# SHF_ALLOC because .BTF will be part of the vmlinux image. --strip-all
# deletes all symbols including __start_BTF and __stop_BTF, which will
# be redefined in the linker script. Add 2>/dev/null to suppress GNU
# objcopy warnings: "empty loadable segment detected at ..."
${OBJCOPY} --only-section=.BTF --set-section-flags .BTF=alloc,readonly \
--strip-all ${1} ${2} 2>/dev/null
# Change e_type to ET_REL so that it can be used to link final vmlinux.
# Unlike GNU ld, lld does not allow an ET_EXEC input.
printf '\1' | dd of=${2} conv=notrunc bs=1 seek=16 status=none
}
# Create ${2} .o file with all symbols from the ${1} object file

View File

@ -277,11 +277,11 @@ endchoice
config LSM
string "Ordered list of enabled LSMs"
default "lockdown,yama,loadpin,safesetid,integrity,smack,selinux,tomoyo,apparmor" if DEFAULT_SECURITY_SMACK
default "lockdown,yama,loadpin,safesetid,integrity,apparmor,selinux,smack,tomoyo" if DEFAULT_SECURITY_APPARMOR
default "lockdown,yama,loadpin,safesetid,integrity,tomoyo" if DEFAULT_SECURITY_TOMOYO
default "lockdown,yama,loadpin,safesetid,integrity" if DEFAULT_SECURITY_DAC
default "lockdown,yama,loadpin,safesetid,integrity,selinux,smack,tomoyo,apparmor"
default "lockdown,yama,loadpin,safesetid,integrity,smack,selinux,tomoyo,apparmor,bpf" if DEFAULT_SECURITY_SMACK
default "lockdown,yama,loadpin,safesetid,integrity,apparmor,selinux,smack,tomoyo,bpf" if DEFAULT_SECURITY_APPARMOR
default "lockdown,yama,loadpin,safesetid,integrity,tomoyo,bpf" if DEFAULT_SECURITY_TOMOYO
default "lockdown,yama,loadpin,safesetid,integrity,bpf" if DEFAULT_SECURITY_DAC
default "lockdown,yama,loadpin,safesetid,integrity,selinux,smack,tomoyo,apparmor,bpf"
help
A comma-separated list of LSMs, in initialization order.
Any LSMs left off this list will be ignored. This can be

View File

@ -12,6 +12,7 @@ subdir-$(CONFIG_SECURITY_YAMA) += yama
subdir-$(CONFIG_SECURITY_LOADPIN) += loadpin
subdir-$(CONFIG_SECURITY_SAFESETID) += safesetid
subdir-$(CONFIG_SECURITY_LOCKDOWN_LSM) += lockdown
subdir-$(CONFIG_BPF_LSM) += bpf
# always enable default capabilities
obj-y += commoncap.o
@ -30,6 +31,7 @@ obj-$(CONFIG_SECURITY_LOADPIN) += loadpin/
obj-$(CONFIG_SECURITY_SAFESETID) += safesetid/
obj-$(CONFIG_SECURITY_LOCKDOWN_LSM) += lockdown/
obj-$(CONFIG_CGROUP_DEVICE) += device_cgroup.o
obj-$(CONFIG_BPF_LSM) += bpf/
# Object integrity file lists
subdir-$(CONFIG_INTEGRITY) += integrity

5
security/bpf/Makefile Normal file
View File

@ -0,0 +1,5 @@
# SPDX-License-Identifier: GPL-2.0
#
# Copyright (C) 2020 Google LLC.
obj-$(CONFIG_BPF_LSM) := hooks.o

26
security/bpf/hooks.c Normal file
View File

@ -0,0 +1,26 @@
// SPDX-License-Identifier: GPL-2.0
/*
* Copyright (C) 2020 Google LLC.
*/
#include <linux/lsm_hooks.h>
#include <linux/bpf_lsm.h>
static struct security_hook_list bpf_lsm_hooks[] __lsm_ro_after_init = {
#define LSM_HOOK(RET, DEFAULT, NAME, ...) \
LSM_HOOK_INIT(NAME, bpf_lsm_##NAME),
#include <linux/lsm_hook_defs.h>
#undef LSM_HOOK
};
static int __init bpf_lsm_init(void)
{
security_add_hooks(bpf_lsm_hooks, ARRAY_SIZE(bpf_lsm_hooks), "bpf");
pr_info("LSM support for eBPF active\n");
return 0;
}
DEFINE_LSM(bpf) = {
.name = "bpf",
.init = bpf_lsm_init,
};

View File

@ -668,6 +668,25 @@ static void __init lsm_early_task(struct task_struct *task)
panic("%s: Early task alloc failed.\n", __func__);
}
/*
* The default value of the LSM hook is defined in linux/lsm_hook_defs.h and
* can be accessed with:
*
* LSM_RET_DEFAULT(<hook_name>)
*
* The macros below define static constants for the default value of each
* LSM hook.
*/
#define LSM_RET_DEFAULT(NAME) (NAME##_default)
#define DECLARE_LSM_RET_DEFAULT_void(DEFAULT, NAME)
#define DECLARE_LSM_RET_DEFAULT_int(DEFAULT, NAME) \
static const int LSM_RET_DEFAULT(NAME) = (DEFAULT);
#define LSM_HOOK(RET, DEFAULT, NAME, ...) \
DECLARE_LSM_RET_DEFAULT_##RET(DEFAULT, NAME)
#include <linux/lsm_hook_defs.h>
#undef LSM_HOOK
/*
* Hook list operation macros.
*
@ -1338,16 +1357,16 @@ int security_inode_getsecurity(struct inode *inode, const char *name, void **buf
int rc;
if (unlikely(IS_PRIVATE(inode)))
return -EOPNOTSUPP;
return LSM_RET_DEFAULT(inode_getsecurity);
/*
* Only one module will provide an attribute with a given name.
*/
hlist_for_each_entry(hp, &security_hook_heads.inode_getsecurity, list) {
rc = hp->hook.inode_getsecurity(inode, name, buffer, alloc);
if (rc != -EOPNOTSUPP)
if (rc != LSM_RET_DEFAULT(inode_getsecurity))
return rc;
}
return -EOPNOTSUPP;
return LSM_RET_DEFAULT(inode_getsecurity);
}
int security_inode_setsecurity(struct inode *inode, const char *name, const void *value, size_t size, int flags)
@ -1356,17 +1375,17 @@ int security_inode_setsecurity(struct inode *inode, const char *name, const void
int rc;
if (unlikely(IS_PRIVATE(inode)))
return -EOPNOTSUPP;
return LSM_RET_DEFAULT(inode_setsecurity);
/*
* Only one module will provide an attribute with a given name.
*/
hlist_for_each_entry(hp, &security_hook_heads.inode_setsecurity, list) {
rc = hp->hook.inode_setsecurity(inode, name, value, size,
flags);
if (rc != -EOPNOTSUPP)
if (rc != LSM_RET_DEFAULT(inode_setsecurity))
return rc;
}
return -EOPNOTSUPP;
return LSM_RET_DEFAULT(inode_setsecurity);
}
int security_inode_listsecurity(struct inode *inode, char *buffer, size_t buffer_size)
@ -1740,12 +1759,12 @@ int security_task_prctl(int option, unsigned long arg2, unsigned long arg3,
unsigned long arg4, unsigned long arg5)
{
int thisrc;
int rc = -ENOSYS;
int rc = LSM_RET_DEFAULT(task_prctl);
struct security_hook_list *hp;
hlist_for_each_entry(hp, &security_hook_heads.task_prctl, list) {
thisrc = hp->hook.task_prctl(option, arg2, arg3, arg4, arg5);
if (thisrc != -ENOSYS) {
if (thisrc != LSM_RET_DEFAULT(task_prctl)) {
rc = thisrc;
if (thisrc != 0)
break;
@ -1917,7 +1936,7 @@ int security_getprocattr(struct task_struct *p, const char *lsm, char *name,
continue;
return hp->hook.getprocattr(p, name, value);
}
return -EINVAL;
return LSM_RET_DEFAULT(getprocattr);
}
int security_setprocattr(const char *lsm, const char *name, void *value,
@ -1930,7 +1949,7 @@ int security_setprocattr(const char *lsm, const char *name, void *value,
continue;
return hp->hook.setprocattr(name, value, size);
}
return -EINVAL;
return LSM_RET_DEFAULT(setprocattr);
}
int security_netlink_send(struct sock *sk, struct sk_buff *skb)
@ -2315,7 +2334,7 @@ int security_xfrm_state_pol_flow_match(struct xfrm_state *x,
const struct flowi *fl)
{
struct security_hook_list *hp;
int rc = 1;
int rc = LSM_RET_DEFAULT(xfrm_state_pol_flow_match);
/*
* Since this function is expected to return 0 or 1, the judgment

View File

@ -0,0 +1,116 @@
==================
bpftool-struct_ops
==================
-------------------------------------------------------------------------------
tool to register/unregister/introspect BPF struct_ops
-------------------------------------------------------------------------------
:Manual section: 8
SYNOPSIS
========
**bpftool** [*OPTIONS*] **struct_ops** *COMMAND*
*OPTIONS* := { { **-j** | **--json** } [{ **-p** | **--pretty** }] }
*COMMANDS* :=
{ **show** | **list** | **dump** | **register** | **unregister** | **help** }
STRUCT_OPS COMMANDS
===================
| **bpftool** **struct_ops { show | list }** [*STRUCT_OPS_MAP*]
| **bpftool** **struct_ops dump** [*STRUCT_OPS_MAP*]
| **bpftool** **struct_ops register** *OBJ*
| **bpftool** **struct_ops unregister** *STRUCT_OPS_MAP*
| **bpftool** **struct_ops help**
|
| *STRUCT_OPS_MAP* := { **id** *STRUCT_OPS_MAP_ID* | **name** *STRUCT_OPS_MAP_NAME* }
| *OBJ* := /a/file/of/bpf_struct_ops.o
DESCRIPTION
===========
**bpftool struct_ops { show | list }** [*STRUCT_OPS_MAP*]
Show brief information about the struct_ops in the system.
If *STRUCT_OPS_MAP* is specified, it shows information only
for the given struct_ops. Otherwise, it lists all struct_ops
currently existing in the system.
Output will start with struct_ops map ID, followed by its map
name and its struct_ops's kernel type.
**bpftool struct_ops dump** [*STRUCT_OPS_MAP*]
Dump details information about the struct_ops in the system.
If *STRUCT_OPS_MAP* is specified, it dumps information only
for the given struct_ops. Otherwise, it dumps all struct_ops
currently existing in the system.
**bpftool struct_ops register** *OBJ*
Register bpf struct_ops from *OBJ*. All struct_ops under
the ELF section ".struct_ops" will be registered to
its kernel subsystem.
**bpftool struct_ops unregister** *STRUCT_OPS_MAP*
Unregister the *STRUCT_OPS_MAP* from the kernel subsystem.
**bpftool struct_ops help**
Print short help message.
OPTIONS
=======
-h, --help
Print short generic help message (similar to **bpftool help**).
-V, --version
Print version number (similar to **bpftool version**).
-j, --json
Generate JSON output. For commands that cannot produce JSON, this
option has no effect.
-p, --pretty
Generate human-readable JSON output. Implies **-j**.
-d, --debug
Print all logs available, even debug-level information. This
includes logs from libbpf as well as from the verifier, when
attempting to load programs.
EXAMPLES
========
**# bpftool struct_ops show**
::
100: dctcp tcp_congestion_ops
105: cubic tcp_congestion_ops
**# bpftool struct_ops unregister id 105**
::
Unregistered tcp_congestion_ops cubic id 105
**# bpftool struct_ops register bpf_cubic.o**
::
Registered tcp_congestion_ops cubic id 110
SEE ALSO
========
**bpf**\ (2),
**bpf-helpers**\ (7),
**bpftool**\ (8),
**bpftool-prog**\ (8),
**bpftool-map**\ (8),
**bpftool-cgroup**\ (8),
**bpftool-feature**\ (8),
**bpftool-net**\ (8),
**bpftool-perf**\ (8),
**bpftool-btf**\ (8)
**bpftool-gen**\ (8)

View File

@ -576,6 +576,34 @@ _bpftool()
;;
esac
;;
struct_ops)
local STRUCT_OPS_TYPE='id name'
case $command in
show|list|dump|unregister)
case $prev in
$command)
COMPREPLY=( $( compgen -W "$STRUCT_OPS_TYPE" -- "$cur" ) )
;;
id)
_bpftool_get_map_ids_for_type struct_ops
;;
name)
_bpftool_get_map_names_for_type struct_ops
;;
esac
return 0
;;
register)
_filedir
return 0
;;
*)
[[ $prev == $object ]] && \
COMPREPLY=( $( compgen -W 'register unregister show list dump help' \
-- "$cur" ) )
;;
esac
;;
map)
local MAP_TYPE='id pinned name'
case $command in

View File

@ -4,11 +4,13 @@
#include <ctype.h>
#include <stdio.h> /* for (FILE *) used by json_writer */
#include <string.h>
#include <unistd.h>
#include <asm/byteorder.h>
#include <linux/bitops.h>
#include <linux/btf.h>
#include <linux/err.h>
#include <bpf/btf.h>
#include <bpf/bpf.h>
#include "json_writer.h"
#include "main.h"
@ -22,13 +24,102 @@
static int btf_dumper_do_type(const struct btf_dumper *d, __u32 type_id,
__u8 bit_offset, const void *data);
static void btf_dumper_ptr(const void *data, json_writer_t *jw,
bool is_plain_text)
static int btf_dump_func(const struct btf *btf, char *func_sig,
const struct btf_type *func_proto,
const struct btf_type *func, int pos, int size);
static int dump_prog_id_as_func_ptr(const struct btf_dumper *d,
const struct btf_type *func_proto,
__u32 prog_id)
{
if (is_plain_text)
jsonw_printf(jw, "%p", *(void **)data);
struct bpf_prog_info_linear *prog_info = NULL;
const struct btf_type *func_type;
const char *prog_name = NULL;
struct bpf_func_info *finfo;
struct btf *prog_btf = NULL;
struct bpf_prog_info *info;
int prog_fd, func_sig_len;
char prog_str[1024];
/* Get the ptr's func_proto */
func_sig_len = btf_dump_func(d->btf, prog_str, func_proto, NULL, 0,
sizeof(prog_str));
if (func_sig_len == -1)
return -1;
if (!prog_id)
goto print;
/* Get the bpf_prog's name. Obtain from func_info. */
prog_fd = bpf_prog_get_fd_by_id(prog_id);
if (prog_fd == -1)
goto print;
prog_info = bpf_program__get_prog_info_linear(prog_fd,
1UL << BPF_PROG_INFO_FUNC_INFO);
close(prog_fd);
if (IS_ERR(prog_info)) {
prog_info = NULL;
goto print;
}
info = &prog_info->info;
if (!info->btf_id || !info->nr_func_info ||
btf__get_from_id(info->btf_id, &prog_btf))
goto print;
finfo = (struct bpf_func_info *)info->func_info;
func_type = btf__type_by_id(prog_btf, finfo->type_id);
if (!func_type || !btf_is_func(func_type))
goto print;
prog_name = btf__name_by_offset(prog_btf, func_type->name_off);
print:
if (!prog_id)
snprintf(&prog_str[func_sig_len],
sizeof(prog_str) - func_sig_len, " 0");
else if (prog_name)
snprintf(&prog_str[func_sig_len],
sizeof(prog_str) - func_sig_len,
" %s/prog_id:%u", prog_name, prog_id);
else
jsonw_printf(jw, "%lu", *(unsigned long *)data);
snprintf(&prog_str[func_sig_len],
sizeof(prog_str) - func_sig_len,
" <unknown_prog_name>/prog_id:%u", prog_id);
prog_str[sizeof(prog_str) - 1] = '\0';
jsonw_string(d->jw, prog_str);
btf__free(prog_btf);
free(prog_info);
return 0;
}
static void btf_dumper_ptr(const struct btf_dumper *d,
const struct btf_type *t,
const void *data)
{
unsigned long value = *(unsigned long *)data;
const struct btf_type *ptr_type;
__s32 ptr_type_id;
if (!d->prog_id_as_func_ptr || value > UINT32_MAX)
goto print_ptr_value;
ptr_type_id = btf__resolve_type(d->btf, t->type);
if (ptr_type_id < 0)
goto print_ptr_value;
ptr_type = btf__type_by_id(d->btf, ptr_type_id);
if (!ptr_type || !btf_is_func_proto(ptr_type))
goto print_ptr_value;
if (!dump_prog_id_as_func_ptr(d, ptr_type, value))
return;
print_ptr_value:
if (d->is_plain_text)
jsonw_printf(d->jw, "%p", (void *)value);
else
jsonw_printf(d->jw, "%lu", value);
}
static int btf_dumper_modifier(const struct btf_dumper *d, __u32 type_id,
@ -43,9 +134,78 @@ static int btf_dumper_modifier(const struct btf_dumper *d, __u32 type_id,
return btf_dumper_do_type(d, actual_type_id, bit_offset, data);
}
static void btf_dumper_enum(const void *data, json_writer_t *jw)
static int btf_dumper_enum(const struct btf_dumper *d,
const struct btf_type *t,
const void *data)
{
jsonw_printf(jw, "%d", *(int *)data);
const struct btf_enum *enums = btf_enum(t);
__s64 value;
__u16 i;
switch (t->size) {
case 8:
value = *(__s64 *)data;
break;
case 4:
value = *(__s32 *)data;
break;
case 2:
value = *(__s16 *)data;
break;
case 1:
value = *(__s8 *)data;
break;
default:
return -EINVAL;
}
for (i = 0; i < btf_vlen(t); i++) {
if (value == enums[i].val) {
jsonw_string(d->jw,
btf__name_by_offset(d->btf,
enums[i].name_off));
return 0;
}
}
jsonw_int(d->jw, value);
return 0;
}
static bool is_str_array(const struct btf *btf, const struct btf_array *arr,
const char *s)
{
const struct btf_type *elem_type;
const char *end_s;
if (!arr->nelems)
return false;
elem_type = btf__type_by_id(btf, arr->type);
/* Not skipping typedef. typedef to char does not count as
* a string now.
*/
while (elem_type && btf_is_mod(elem_type))
elem_type = btf__type_by_id(btf, elem_type->type);
if (!elem_type || !btf_is_int(elem_type) || elem_type->size != 1)
return false;
if (btf_int_encoding(elem_type) != BTF_INT_CHAR &&
strcmp("char", btf__name_by_offset(btf, elem_type->name_off)))
return false;
end_s = s + arr->nelems;
while (s < end_s) {
if (!*s)
return true;
if (*s <= 0x1f || *s >= 0x7f)
return false;
s++;
}
/* '\0' is not found */
return false;
}
static int btf_dumper_array(const struct btf_dumper *d, __u32 type_id,
@ -57,6 +217,11 @@ static int btf_dumper_array(const struct btf_dumper *d, __u32 type_id,
int ret = 0;
__u32 i;
if (is_str_array(d->btf, arr, data)) {
jsonw_string(d->jw, data);
return 0;
}
elem_size = btf__resolve_size(d->btf, arr->type);
if (elem_size < 0)
return elem_size;
@ -366,10 +531,9 @@ static int btf_dumper_do_type(const struct btf_dumper *d, __u32 type_id,
case BTF_KIND_ARRAY:
return btf_dumper_array(d, type_id, data);
case BTF_KIND_ENUM:
btf_dumper_enum(data, d->jw);
return 0;
return btf_dumper_enum(d, t, data);
case BTF_KIND_PTR:
btf_dumper_ptr(data, d->jw, d->is_plain_text);
btf_dumper_ptr(d, t, data);
return 0;
case BTF_KIND_UNKN:
jsonw_printf(d->jw, "(unknown)");
@ -414,10 +578,6 @@ int btf_dumper_type(const struct btf_dumper *d, __u32 type_id,
return -1; \
} while (0)
static int btf_dump_func(const struct btf *btf, char *func_sig,
const struct btf_type *func_proto,
const struct btf_type *func, int pos, int size);
static int __btf_dumper_type_only(const struct btf *btf, __u32 type_id,
char *func_sig, int pos, int size)
{
@ -526,8 +686,15 @@ static int btf_dump_func(const struct btf *btf, char *func_sig,
BTF_PRINT_ARG(", ");
if (arg->type) {
BTF_PRINT_TYPE(arg->type);
BTF_PRINT_ARG("%s",
btf__name_by_offset(btf, arg->name_off));
if (arg->name_off)
BTF_PRINT_ARG("%s",
btf__name_by_offset(btf, arg->name_off));
else if (pos && func_sig[pos - 1] == ' ')
/* Remove unnecessary space for
* FUNC_PROTO that does not have
* arg->name_off
*/
func_sig[--pos] = '\0';
} else {
BTF_PRINT_ARG("...");
}

View File

@ -58,7 +58,7 @@ static int do_help(int argc, char **argv)
" %s batch file FILE\n"
" %s version\n"
"\n"
" OBJECT := { prog | map | cgroup | perf | net | feature | btf | gen }\n"
" OBJECT := { prog | map | cgroup | perf | net | feature | btf | gen | struct_ops }\n"
" " HELP_SPEC_OPTIONS "\n"
"",
bin_name, bin_name, bin_name);
@ -221,6 +221,7 @@ static const struct cmd cmds[] = {
{ "feature", do_feature },
{ "btf", do_btf },
{ "gen", do_gen },
{ "struct_ops", do_struct_ops },
{ "version", do_version },
{ 0 }
};

View File

@ -161,6 +161,7 @@ int do_tracelog(int argc, char **arg);
int do_feature(int argc, char **argv);
int do_btf(int argc, char **argv);
int do_gen(int argc, char **argv);
int do_struct_ops(int argc, char **argv);
int parse_u32_arg(int *argc, char ***argv, __u32 *val, const char *what);
int prog_parse_fd(int *argc, char ***argv);
@ -205,6 +206,7 @@ struct btf_dumper {
const struct btf *btf;
json_writer_t *jw;
bool is_plain_text;
bool prog_id_as_func_ptr;
};
/* btf_dumper_type - print data along with type information

View File

@ -0,0 +1,596 @@
// SPDX-License-Identifier: (GPL-2.0-only OR BSD-2-Clause)
/* Copyright (C) 2020 Facebook */
#include <errno.h>
#include <stdio.h>
#include <unistd.h>
#include <linux/err.h>
#include <bpf/bpf.h>
#include <bpf/btf.h>
#include <bpf/libbpf.h>
#include "json_writer.h"
#include "main.h"
#define STRUCT_OPS_VALUE_PREFIX "bpf_struct_ops_"
static const struct btf_type *map_info_type;
static __u32 map_info_alloc_len;
static struct btf *btf_vmlinux;
static __s32 map_info_type_id;
struct res {
unsigned int nr_maps;
unsigned int nr_errs;
};
static const struct btf *get_btf_vmlinux(void)
{
if (btf_vmlinux)
return btf_vmlinux;
btf_vmlinux = libbpf_find_kernel_btf();
if (IS_ERR(btf_vmlinux))
p_err("struct_ops requires kernel CONFIG_DEBUG_INFO_BTF=y");
return btf_vmlinux;
}
static const char *get_kern_struct_ops_name(const struct bpf_map_info *info)
{
const struct btf *kern_btf;
const struct btf_type *t;
const char *st_ops_name;
kern_btf = get_btf_vmlinux();
if (IS_ERR(kern_btf))
return "<btf_vmlinux_not_found>";
t = btf__type_by_id(kern_btf, info->btf_vmlinux_value_type_id);
st_ops_name = btf__name_by_offset(kern_btf, t->name_off);
st_ops_name += strlen(STRUCT_OPS_VALUE_PREFIX);
return st_ops_name;
}
static __s32 get_map_info_type_id(void)
{
const struct btf *kern_btf;
if (map_info_type_id)
return map_info_type_id;
kern_btf = get_btf_vmlinux();
if (IS_ERR(kern_btf)) {
map_info_type_id = PTR_ERR(kern_btf);
return map_info_type_id;
}
map_info_type_id = btf__find_by_name_kind(kern_btf, "bpf_map_info",
BTF_KIND_STRUCT);
if (map_info_type_id < 0) {
p_err("can't find bpf_map_info from btf_vmlinux");
return map_info_type_id;
}
map_info_type = btf__type_by_id(kern_btf, map_info_type_id);
/* Ensure map_info_alloc() has at least what the bpftool needs */
map_info_alloc_len = map_info_type->size;
if (map_info_alloc_len < sizeof(struct bpf_map_info))
map_info_alloc_len = sizeof(struct bpf_map_info);
return map_info_type_id;
}
/* If the subcmd needs to print out the bpf_map_info,
* it should always call map_info_alloc to allocate
* a bpf_map_info object instead of allocating it
* on the stack.
*
* map_info_alloc() will take the running kernel's btf
* into account. i.e. it will consider the
* sizeof(struct bpf_map_info) of the running kernel.
*
* It will enable the "struct_ops" cmd to print the latest
* "struct bpf_map_info".
*
* [ Recall that "struct_ops" requires the kernel's btf to
* be available ]
*/
static struct bpf_map_info *map_info_alloc(__u32 *alloc_len)
{
struct bpf_map_info *info;
if (get_map_info_type_id() < 0)
return NULL;
info = calloc(1, map_info_alloc_len);
if (!info)
p_err("mem alloc failed");
else
*alloc_len = map_info_alloc_len;
return info;
}
/* It iterates all struct_ops maps of the system.
* It returns the fd in "*res_fd" and map_info in "*info".
* In the very first iteration, info->id should be 0.
* An optional map "*name" filter can be specified.
* The filter can be made more flexible in the future.
* e.g. filter by kernel-struct-ops-name, regex-name, glob-name, ...etc.
*
* Return value:
* 1: A struct_ops map found. It is returned in "*res_fd" and "*info".
* The caller can continue to call get_next in the future.
* 0: No struct_ops map is returned.
* All struct_ops map has been found.
* -1: Error and the caller should abort the iteration.
*/
static int get_next_struct_ops_map(const char *name, int *res_fd,
struct bpf_map_info *info, __u32 info_len)
{
__u32 id = info->id;
int err, fd;
while (true) {
err = bpf_map_get_next_id(id, &id);
if (err) {
if (errno == ENOENT)
return 0;
p_err("can't get next map: %s", strerror(errno));
return -1;
}
fd = bpf_map_get_fd_by_id(id);
if (fd < 0) {
if (errno == ENOENT)
continue;
p_err("can't get map by id (%u): %s",
id, strerror(errno));
return -1;
}
err = bpf_obj_get_info_by_fd(fd, info, &info_len);
if (err) {
p_err("can't get map info: %s", strerror(errno));
close(fd);
return -1;
}
if (info->type == BPF_MAP_TYPE_STRUCT_OPS &&
(!name || !strcmp(name, info->name))) {
*res_fd = fd;
return 1;
}
close(fd);
}
}
static int cmd_retval(const struct res *res, bool must_have_one_map)
{
if (res->nr_errs || (!res->nr_maps && must_have_one_map))
return -1;
return 0;
}
/* "data" is the work_func private storage */
typedef int (*work_func)(int fd, const struct bpf_map_info *info, void *data,
struct json_writer *wtr);
/* Find all struct_ops map in the system.
* Filter out by "name" (if specified).
* Then call "func(fd, info, data, wtr)" on each struct_ops map found.
*/
static struct res do_search(const char *name, work_func func, void *data,
struct json_writer *wtr)
{
struct bpf_map_info *info;
struct res res = {};
__u32 info_len;
int fd, err;
info = map_info_alloc(&info_len);
if (!info) {
res.nr_errs++;
return res;
}
if (wtr)
jsonw_start_array(wtr);
while ((err = get_next_struct_ops_map(name, &fd, info, info_len)) == 1) {
res.nr_maps++;
err = func(fd, info, data, wtr);
if (err)
res.nr_errs++;
close(fd);
}
if (wtr)
jsonw_end_array(wtr);
if (err)
res.nr_errs++;
if (!wtr && name && !res.nr_errs && !res.nr_maps)
/* It is not printing empty [].
* Thus, needs to specifically say nothing found
* for "name" here.
*/
p_err("no struct_ops found for %s", name);
else if (!wtr && json_output && !res.nr_errs)
/* The "func()" above is not writing any json (i.e. !wtr
* test here).
*
* However, "-j" is enabled and there is no errs here,
* so call json_null() as the current convention of
* other cmds.
*/
jsonw_null(json_wtr);
free(info);
return res;
}
static struct res do_one_id(const char *id_str, work_func func, void *data,
struct json_writer *wtr)
{
struct bpf_map_info *info;
struct res res = {};
unsigned long id;
__u32 info_len;
char *endptr;
int fd;
id = strtoul(id_str, &endptr, 0);
if (*endptr || !id || id > UINT32_MAX) {
p_err("invalid id %s", id_str);
res.nr_errs++;
return res;
}
fd = bpf_map_get_fd_by_id(id);
if (fd == -1) {
p_err("can't get map by id (%lu): %s", id, strerror(errno));
res.nr_errs++;
return res;
}
info = map_info_alloc(&info_len);
if (!info) {
res.nr_errs++;
goto done;
}
if (bpf_obj_get_info_by_fd(fd, info, &info_len)) {
p_err("can't get map info: %s", strerror(errno));
res.nr_errs++;
goto done;
}
if (info->type != BPF_MAP_TYPE_STRUCT_OPS) {
p_err("%s id %u is not a struct_ops map", info->name, info->id);
res.nr_errs++;
goto done;
}
res.nr_maps++;
if (func(fd, info, data, wtr))
res.nr_errs++;
else if (!wtr && json_output)
/* The "func()" above is not writing any json (i.e. !wtr
* test here).
*
* However, "-j" is enabled and there is no errs here,
* so call json_null() as the current convention of
* other cmds.
*/
jsonw_null(json_wtr);
done:
free(info);
close(fd);
return res;
}
static struct res do_work_on_struct_ops(const char *search_type,
const char *search_term,
work_func func, void *data,
struct json_writer *wtr)
{
if (search_type) {
if (is_prefix(search_type, "id"))
return do_one_id(search_term, func, data, wtr);
else if (!is_prefix(search_type, "name"))
usage();
}
return do_search(search_term, func, data, wtr);
}
static int __do_show(int fd, const struct bpf_map_info *info, void *data,
struct json_writer *wtr)
{
if (wtr) {
jsonw_start_object(wtr);
jsonw_uint_field(wtr, "id", info->id);
jsonw_string_field(wtr, "name", info->name);
jsonw_string_field(wtr, "kernel_struct_ops",
get_kern_struct_ops_name(info));
jsonw_end_object(wtr);
} else {
printf("%u: %-15s %-32s\n", info->id, info->name,
get_kern_struct_ops_name(info));
}
return 0;
}
static int do_show(int argc, char **argv)
{
const char *search_type = NULL, *search_term = NULL;
struct res res;
if (argc && argc != 2)
usage();
if (argc == 2) {
search_type = GET_ARG();
search_term = GET_ARG();
}
res = do_work_on_struct_ops(search_type, search_term, __do_show,
NULL, json_wtr);
return cmd_retval(&res, !!search_term);
}
static int __do_dump(int fd, const struct bpf_map_info *info, void *data,
struct json_writer *wtr)
{
struct btf_dumper *d = (struct btf_dumper *)data;
const struct btf_type *struct_ops_type;
const struct btf *kern_btf = d->btf;
const char *struct_ops_name;
int zero = 0;
void *value;
/* note: d->jw == wtr */
kern_btf = d->btf;
/* The kernel supporting BPF_MAP_TYPE_STRUCT_OPS must have
* btf_vmlinux_value_type_id.
*/
struct_ops_type = btf__type_by_id(kern_btf,
info->btf_vmlinux_value_type_id);
struct_ops_name = btf__name_by_offset(kern_btf,
struct_ops_type->name_off);
value = calloc(1, info->value_size);
if (!value) {
p_err("mem alloc failed");
return -1;
}
if (bpf_map_lookup_elem(fd, &zero, value)) {
p_err("can't lookup struct_ops map %s id %u",
info->name, info->id);
free(value);
return -1;
}
jsonw_start_object(wtr);
jsonw_name(wtr, "bpf_map_info");
btf_dumper_type(d, map_info_type_id, (void *)info);
jsonw_end_object(wtr);
jsonw_start_object(wtr);
jsonw_name(wtr, struct_ops_name);
btf_dumper_type(d, info->btf_vmlinux_value_type_id, value);
jsonw_end_object(wtr);
free(value);
return 0;
}
static int do_dump(int argc, char **argv)
{
const char *search_type = NULL, *search_term = NULL;
json_writer_t *wtr = json_wtr;
const struct btf *kern_btf;
struct btf_dumper d = {};
struct res res;
if (argc && argc != 2)
usage();
if (argc == 2) {
search_type = GET_ARG();
search_term = GET_ARG();
}
kern_btf = get_btf_vmlinux();
if (IS_ERR(kern_btf))
return -1;
if (!json_output) {
wtr = jsonw_new(stdout);
if (!wtr) {
p_err("can't create json writer");
return -1;
}
jsonw_pretty(wtr, true);
}
d.btf = kern_btf;
d.jw = wtr;
d.is_plain_text = !json_output;
d.prog_id_as_func_ptr = true;
res = do_work_on_struct_ops(search_type, search_term, __do_dump, &d,
wtr);
if (!json_output)
jsonw_destroy(&wtr);
return cmd_retval(&res, !!search_term);
}
static int __do_unregister(int fd, const struct bpf_map_info *info, void *data,
struct json_writer *wtr)
{
int zero = 0;
if (bpf_map_delete_elem(fd, &zero)) {
p_err("can't unload %s %s id %u: %s",
get_kern_struct_ops_name(info), info->name,
info->id, strerror(errno));
return -1;
}
p_info("Unregistered %s %s id %u",
get_kern_struct_ops_name(info), info->name,
info->id);
return 0;
}
static int do_unregister(int argc, char **argv)
{
const char *search_type, *search_term;
struct res res;
if (argc != 2)
usage();
search_type = GET_ARG();
search_term = GET_ARG();
res = do_work_on_struct_ops(search_type, search_term,
__do_unregister, NULL, NULL);
return cmd_retval(&res, true);
}
static int do_register(int argc, char **argv)
{
const struct bpf_map_def *def;
struct bpf_map_info info = {};
__u32 info_len = sizeof(info);
int nr_errs = 0, nr_maps = 0;
struct bpf_object *obj;
struct bpf_link *link;
struct bpf_map *map;
const char *file;
if (argc != 1)
usage();
file = GET_ARG();
obj = bpf_object__open(file);
if (IS_ERR_OR_NULL(obj))
return -1;
set_max_rlimit();
if (bpf_object__load(obj)) {
bpf_object__close(obj);
return -1;
}
bpf_object__for_each_map(map, obj) {
def = bpf_map__def(map);
if (def->type != BPF_MAP_TYPE_STRUCT_OPS)
continue;
link = bpf_map__attach_struct_ops(map);
if (IS_ERR(link)) {
p_err("can't register struct_ops %s: %s",
bpf_map__name(map),
strerror(-PTR_ERR(link)));
nr_errs++;
continue;
}
nr_maps++;
bpf_link__disconnect(link);
bpf_link__destroy(link);
if (!bpf_obj_get_info_by_fd(bpf_map__fd(map), &info,
&info_len))
p_info("Registered %s %s id %u",
get_kern_struct_ops_name(&info),
bpf_map__name(map),
info.id);
else
/* Not p_err. The struct_ops was attached
* successfully.
*/
p_info("Registered %s but can't find id: %s",
bpf_map__name(map), strerror(errno));
}
bpf_object__close(obj);
if (nr_errs)
return -1;
if (!nr_maps) {
p_err("no struct_ops found in %s", file);
return -1;
}
if (json_output)
jsonw_null(json_wtr);
return 0;
}
static int do_help(int argc, char **argv)
{
if (json_output) {
jsonw_null(json_wtr);
return 0;
}
fprintf(stderr,
"Usage: %s %s { show | list } [STRUCT_OPS_MAP]\n"
" %s %s dump [STRUCT_OPS_MAP]\n"
" %s %s register OBJ\n"
" %s %s unregister STRUCT_OPS_MAP\n"
" %s %s help\n"
"\n"
" OPTIONS := { {-j|--json} [{-p|--pretty}] }\n"
" STRUCT_OPS_MAP := [ id STRUCT_OPS_MAP_ID | name STRUCT_OPS_MAP_NAME ]\n",
bin_name, argv[-2], bin_name, argv[-2],
bin_name, argv[-2], bin_name, argv[-2],
bin_name, argv[-2]);
return 0;
}
static const struct cmd cmds[] = {
{ "show", do_show },
{ "list", do_show },
{ "register", do_register },
{ "unregister", do_unregister },
{ "dump", do_dump },
{ "help", do_help },
{ 0 }
};
int do_struct_ops(int argc, char **argv)
{
int err;
err = cmd_select(cmds, argc, argv, do_help);
btf__free(btf_vmlinux);
return err;
}

View File

@ -111,6 +111,8 @@ enum bpf_cmd {
BPF_MAP_LOOKUP_AND_DELETE_BATCH,
BPF_MAP_UPDATE_BATCH,
BPF_MAP_DELETE_BATCH,
BPF_LINK_CREATE,
BPF_LINK_UPDATE,
};
enum bpf_map_type {
@ -181,6 +183,7 @@ enum bpf_prog_type {
BPF_PROG_TYPE_TRACING,
BPF_PROG_TYPE_STRUCT_OPS,
BPF_PROG_TYPE_EXT,
BPF_PROG_TYPE_LSM,
};
enum bpf_attach_type {
@ -211,6 +214,7 @@ enum bpf_attach_type {
BPF_TRACE_FENTRY,
BPF_TRACE_FEXIT,
BPF_MODIFY_RETURN,
BPF_LSM_MAC,
__MAX_BPF_ATTACH_TYPE
};
@ -539,7 +543,7 @@ union bpf_attr {
__u32 prog_cnt;
} query;
struct {
struct { /* anonymous struct used by BPF_RAW_TRACEPOINT_OPEN command */
__u64 name;
__u32 prog_fd;
} raw_tracepoint;
@ -567,6 +571,24 @@ union bpf_attr {
__u64 probe_offset; /* output: probe_offset */
__u64 probe_addr; /* output: probe_addr */
} task_fd_query;
struct { /* struct used by BPF_LINK_CREATE command */
__u32 prog_fd; /* eBPF program to attach */
__u32 target_fd; /* object to attach to */
__u32 attach_type; /* attach type */
__u32 flags; /* extra flags */
} link_create;
struct { /* struct used by BPF_LINK_UPDATE command */
__u32 link_fd; /* link fd */
/* new program fd to update link with */
__u32 new_prog_fd;
__u32 flags; /* extra flags */
/* expected link's program fd; is specified only if
* BPF_F_REPLACE flag is set in flags */
__u32 old_prog_fd;
} link_update;
} __attribute__((aligned(8)));
/* The description below is an attempt at providing documentation to eBPF
@ -2950,6 +2972,59 @@ union bpf_attr {
* restricted to raw_tracepoint bpf programs.
* Return
* 0 on success, or a negative error in case of failure.
*
* u64 bpf_get_netns_cookie(void *ctx)
* Description
* Retrieve the cookie (generated by the kernel) of the network
* namespace the input *ctx* is associated with. The network
* namespace cookie remains stable for its lifetime and provides
* a global identifier that can be assumed unique. If *ctx* is
* NULL, then the helper returns the cookie for the initial
* network namespace. The cookie itself is very similar to that
* of bpf_get_socket_cookie() helper, but for network namespaces
* instead of sockets.
* Return
* A 8-byte long opaque number.
*
* u64 bpf_get_current_ancestor_cgroup_id(int ancestor_level)
* Description
* Return id of cgroup v2 that is ancestor of the cgroup associated
* with the current task at the *ancestor_level*. The root cgroup
* is at *ancestor_level* zero and each step down the hierarchy
* increments the level. If *ancestor_level* == level of cgroup
* associated with the current task, then return value will be the
* same as that of **bpf_get_current_cgroup_id**\ ().
*
* The helper is useful to implement policies based on cgroups
* that are upper in hierarchy than immediate cgroup associated
* with the current task.
*
* The format of returned id and helper limitations are same as in
* **bpf_get_current_cgroup_id**\ ().
* Return
* The id is returned or 0 in case the id could not be retrieved.
*
* int bpf_sk_assign(struct sk_buff *skb, struct bpf_sock *sk, u64 flags)
* Description
* Assign the *sk* to the *skb*. When combined with appropriate
* routing configuration to receive the packet towards the socket,
* will cause *skb* to be delivered to the specified socket.
* Subsequent redirection of *skb* via **bpf_redirect**\ (),
* **bpf_clone_redirect**\ () or other methods outside of BPF may
* interfere with successful delivery to the socket.
*
* This operation is only valid from TC ingress path.
*
* The *flags* argument must be zero.
* Return
* 0 on success, or a negative errno in case of failure.
*
* * **-EINVAL** Unsupported flags specified.
* * **-ENOENT** Socket is unavailable for assignment.
* * **-ENETUNREACH** Socket is unreachable (wrong netns).
* * **-EOPNOTSUPP** Unsupported operation, for example a
* call from outside of TC ingress.
* * **-ESOCKTNOSUPPORT** Socket type not supported (reuseport).
*/
#define __BPF_FUNC_MAPPER(FN) \
FN(unspec), \
@ -3073,7 +3148,10 @@ union bpf_attr {
FN(jiffies64), \
FN(read_branch_records), \
FN(get_ns_current_pid_tgid), \
FN(xdp_output),
FN(xdp_output), \
FN(get_netns_cookie), \
FN(get_current_ancestor_cgroup_id), \
FN(sk_assign),
/* integer value in 'imm' field of BPF_CALL instruction selects which helper
* function eBPF program intends to call

View File

@ -962,11 +962,12 @@ enum {
#define XDP_FLAGS_SKB_MODE (1U << 1)
#define XDP_FLAGS_DRV_MODE (1U << 2)
#define XDP_FLAGS_HW_MODE (1U << 3)
#define XDP_FLAGS_REPLACE (1U << 4)
#define XDP_FLAGS_MODES (XDP_FLAGS_SKB_MODE | \
XDP_FLAGS_DRV_MODE | \
XDP_FLAGS_HW_MODE)
#define XDP_FLAGS_MASK (XDP_FLAGS_UPDATE_IF_NOEXIST | \
XDP_FLAGS_MODES)
XDP_FLAGS_MODES | XDP_FLAGS_REPLACE)
/* These are stored into IFLA_XDP_ATTACHED on dump. */
enum {
@ -986,6 +987,7 @@ enum {
IFLA_XDP_DRV_PROG_ID,
IFLA_XDP_SKB_PROG_ID,
IFLA_XDP_HW_PROG_ID,
IFLA_XDP_EXPECTED_FD,
__IFLA_XDP_MAX,
};

View File

@ -235,7 +235,8 @@ int bpf_load_program_xattr(const struct bpf_load_program_attr *load_attr,
memset(&attr, 0, sizeof(attr));
attr.prog_type = load_attr->prog_type;
attr.expected_attach_type = load_attr->expected_attach_type;
if (attr.prog_type == BPF_PROG_TYPE_STRUCT_OPS) {
if (attr.prog_type == BPF_PROG_TYPE_STRUCT_OPS ||
attr.prog_type == BPF_PROG_TYPE_LSM) {
attr.attach_btf_id = load_attr->attach_btf_id;
} else if (attr.prog_type == BPF_PROG_TYPE_TRACING ||
attr.prog_type == BPF_PROG_TYPE_EXT) {
@ -584,6 +585,40 @@ int bpf_prog_detach2(int prog_fd, int target_fd, enum bpf_attach_type type)
return sys_bpf(BPF_PROG_DETACH, &attr, sizeof(attr));
}
int bpf_link_create(int prog_fd, int target_fd,
enum bpf_attach_type attach_type,
const struct bpf_link_create_opts *opts)
{
union bpf_attr attr;
if (!OPTS_VALID(opts, bpf_link_create_opts))
return -EINVAL;
memset(&attr, 0, sizeof(attr));
attr.link_create.prog_fd = prog_fd;
attr.link_create.target_fd = target_fd;
attr.link_create.attach_type = attach_type;
return sys_bpf(BPF_LINK_CREATE, &attr, sizeof(attr));
}
int bpf_link_update(int link_fd, int new_prog_fd,
const struct bpf_link_update_opts *opts)
{
union bpf_attr attr;
if (!OPTS_VALID(opts, bpf_link_update_opts))
return -EINVAL;
memset(&attr, 0, sizeof(attr));
attr.link_update.link_fd = link_fd;
attr.link_update.new_prog_fd = new_prog_fd;
attr.link_update.flags = OPTS_GET(opts, flags, 0);
attr.link_update.old_prog_fd = OPTS_GET(opts, old_prog_fd, 0);
return sys_bpf(BPF_LINK_UPDATE, &attr, sizeof(attr));
}
int bpf_prog_query(int target_fd, enum bpf_attach_type type, __u32 query_flags,
__u32 *attach_flags, __u32 *prog_ids, __u32 *prog_cnt)
{

View File

@ -168,6 +168,25 @@ LIBBPF_API int bpf_prog_detach(int attachable_fd, enum bpf_attach_type type);
LIBBPF_API int bpf_prog_detach2(int prog_fd, int attachable_fd,
enum bpf_attach_type type);
struct bpf_link_create_opts {
size_t sz; /* size of this struct for forward/backward compatibility */
};
#define bpf_link_create_opts__last_field sz
LIBBPF_API int bpf_link_create(int prog_fd, int target_fd,
enum bpf_attach_type attach_type,
const struct bpf_link_create_opts *opts);
struct bpf_link_update_opts {
size_t sz; /* size of this struct for forward/backward compatibility */
__u32 flags; /* extra flags */
__u32 old_prog_fd; /* expected old program FD */
};
#define bpf_link_update_opts__last_field old_prog_fd
LIBBPF_API int bpf_link_update(int link_fd, int new_prog_fd,
const struct bpf_link_update_opts *opts);
struct bpf_prog_test_run_attr {
int prog_fd;
int repeat;

View File

@ -390,7 +390,7 @@ ____##name(struct pt_regs *ctx, ##args)
#define ___bpf_kretprobe_args0() ctx
#define ___bpf_kretprobe_args1(x) \
___bpf_kretprobe_args0(), (void *)PT_REGS_RET(ctx)
___bpf_kretprobe_args0(), (void *)PT_REGS_RC(ctx)
#define ___bpf_kretprobe_args(args...) \
___bpf_apply(___bpf_kretprobe_args, ___bpf_narg(args))(args)

View File

@ -657,22 +657,32 @@ int btf__finalize_data(struct bpf_object *obj, struct btf *btf)
int btf__load(struct btf *btf)
{
__u32 log_buf_size = BPF_LOG_BUF_SIZE;
__u32 log_buf_size = 0;
char *log_buf = NULL;
int err = 0;
if (btf->fd >= 0)
return -EEXIST;
log_buf = malloc(log_buf_size);
if (!log_buf)
return -ENOMEM;
retry_load:
if (log_buf_size) {
log_buf = malloc(log_buf_size);
if (!log_buf)
return -ENOMEM;
*log_buf = 0;
*log_buf = 0;
}
btf->fd = bpf_load_btf(btf->data, btf->data_size,
log_buf, log_buf_size, false);
if (btf->fd < 0) {
if (!log_buf || errno == ENOSPC) {
log_buf_size = max((__u32)BPF_LOG_BUF_SIZE,
log_buf_size << 1);
free(log_buf);
goto retry_load;
}
err = -errno;
pr_warn("Error loading BTF: %s(%d)\n", strerror(errno), errno);
if (*log_buf)

View File

@ -1845,7 +1845,6 @@ resolve_func_ptr(const struct btf *btf, __u32 id, __u32 *res_id)
* type definition, while using only sizeof(void *) space in ELF data section.
*/
static bool get_map_field_int(const char *map_name, const struct btf *btf,
const struct btf_type *def,
const struct btf_member *m, __u32 *res)
{
const struct btf_type *t = skip_mods_and_typedefs(btf, m->type, NULL);
@ -1972,19 +1971,19 @@ static int bpf_object__init_user_btf_map(struct bpf_object *obj,
return -EINVAL;
}
if (strcmp(name, "type") == 0) {
if (!get_map_field_int(map_name, obj->btf, def, m,
if (!get_map_field_int(map_name, obj->btf, m,
&map->def.type))
return -EINVAL;
pr_debug("map '%s': found type = %u.\n",
map_name, map->def.type);
} else if (strcmp(name, "max_entries") == 0) {
if (!get_map_field_int(map_name, obj->btf, def, m,
if (!get_map_field_int(map_name, obj->btf, m,
&map->def.max_entries))
return -EINVAL;
pr_debug("map '%s': found max_entries = %u.\n",
map_name, map->def.max_entries);
} else if (strcmp(name, "map_flags") == 0) {
if (!get_map_field_int(map_name, obj->btf, def, m,
if (!get_map_field_int(map_name, obj->btf, m,
&map->def.map_flags))
return -EINVAL;
pr_debug("map '%s': found map_flags = %u.\n",
@ -1992,8 +1991,7 @@ static int bpf_object__init_user_btf_map(struct bpf_object *obj,
} else if (strcmp(name, "key_size") == 0) {
__u32 sz;
if (!get_map_field_int(map_name, obj->btf, def, m,
&sz))
if (!get_map_field_int(map_name, obj->btf, m, &sz))
return -EINVAL;
pr_debug("map '%s': found key_size = %u.\n",
map_name, sz);
@ -2035,8 +2033,7 @@ static int bpf_object__init_user_btf_map(struct bpf_object *obj,
} else if (strcmp(name, "value_size") == 0) {
__u32 sz;
if (!get_map_field_int(map_name, obj->btf, def, m,
&sz))
if (!get_map_field_int(map_name, obj->btf, m, &sz))
return -EINVAL;
pr_debug("map '%s': found value_size = %u.\n",
map_name, sz);
@ -2079,8 +2076,7 @@ static int bpf_object__init_user_btf_map(struct bpf_object *obj,
__u32 val;
int err;
if (!get_map_field_int(map_name, obj->btf, def, m,
&val))
if (!get_map_field_int(map_name, obj->btf, m, &val))
return -EINVAL;
pr_debug("map '%s': found pinning = %u.\n",
map_name, val);
@ -2362,7 +2358,8 @@ static int bpf_object__finalize_btf(struct bpf_object *obj)
static inline bool libbpf_prog_needs_vmlinux_btf(struct bpf_program *prog)
{
if (prog->type == BPF_PROG_TYPE_STRUCT_OPS)
if (prog->type == BPF_PROG_TYPE_STRUCT_OPS ||
prog->type == BPF_PROG_TYPE_LSM)
return true;
/* BPF_PROG_TYPE_TRACING programs which do not attach to other programs
@ -4855,8 +4852,8 @@ load_program(struct bpf_program *prog, struct bpf_insn *insns, int insns_cnt,
{
struct bpf_load_program_attr load_attr;
char *cp, errmsg[STRERR_BUFSIZE];
int log_buf_size = BPF_LOG_BUF_SIZE;
char *log_buf;
size_t log_buf_size = 0;
char *log_buf = NULL;
int btf_fd, ret;
if (!insns || !insns_cnt)
@ -4870,7 +4867,8 @@ load_program(struct bpf_program *prog, struct bpf_insn *insns, int insns_cnt,
load_attr.insns = insns;
load_attr.insns_cnt = insns_cnt;
load_attr.license = license;
if (prog->type == BPF_PROG_TYPE_STRUCT_OPS) {
if (prog->type == BPF_PROG_TYPE_STRUCT_OPS ||
prog->type == BPF_PROG_TYPE_LSM) {
load_attr.attach_btf_id = prog->attach_btf_id;
} else if (prog->type == BPF_PROG_TYPE_TRACING ||
prog->type == BPF_PROG_TYPE_EXT) {
@ -4896,22 +4894,28 @@ load_program(struct bpf_program *prog, struct bpf_insn *insns, int insns_cnt,
load_attr.prog_flags = prog->prog_flags;
retry_load:
log_buf = malloc(log_buf_size);
if (!log_buf)
pr_warn("Alloc log buffer for bpf loader error, continue without log\n");
if (log_buf_size) {
log_buf = malloc(log_buf_size);
if (!log_buf)
return -ENOMEM;
*log_buf = 0;
}
ret = bpf_load_program_xattr(&load_attr, log_buf, log_buf_size);
if (ret >= 0) {
if (load_attr.log_level)
if (log_buf && load_attr.log_level)
pr_debug("verifier log:\n%s", log_buf);
*pfd = ret;
ret = 0;
goto out;
}
if (errno == ENOSPC) {
log_buf_size <<= 1;
if (!log_buf || errno == ENOSPC) {
log_buf_size = max((size_t)BPF_LOG_BUF_SIZE,
log_buf_size << 1);
free(log_buf);
goto retry_load;
}
@ -4955,6 +4959,7 @@ int bpf_program__load(struct bpf_program *prog, char *license, __u32 kern_ver)
int err = 0, fd, i, btf_id;
if ((prog->type == BPF_PROG_TYPE_TRACING ||
prog->type == BPF_PROG_TYPE_LSM ||
prog->type == BPF_PROG_TYPE_EXT) && !prog->attach_btf_id) {
btf_id = libbpf_find_attach_btf_id(prog);
if (btf_id <= 0)
@ -6194,6 +6199,7 @@ bool bpf_program__is_##NAME(const struct bpf_program *prog) \
} \
BPF_PROG_TYPE_FNS(socket_filter, BPF_PROG_TYPE_SOCKET_FILTER);
BPF_PROG_TYPE_FNS(lsm, BPF_PROG_TYPE_LSM);
BPF_PROG_TYPE_FNS(kprobe, BPF_PROG_TYPE_KPROBE);
BPF_PROG_TYPE_FNS(sched_cls, BPF_PROG_TYPE_SCHED_CLS);
BPF_PROG_TYPE_FNS(sched_act, BPF_PROG_TYPE_SCHED_ACT);
@ -6260,6 +6266,8 @@ static struct bpf_link *attach_raw_tp(const struct bpf_sec_def *sec,
struct bpf_program *prog);
static struct bpf_link *attach_trace(const struct bpf_sec_def *sec,
struct bpf_program *prog);
static struct bpf_link *attach_lsm(const struct bpf_sec_def *sec,
struct bpf_program *prog);
struct bpf_sec_def {
const char *sec;
@ -6310,6 +6318,10 @@ static const struct bpf_sec_def section_defs[] = {
SEC_DEF("freplace/", EXT,
.is_attach_btf = true,
.attach_fn = attach_trace),
SEC_DEF("lsm/", LSM,
.is_attach_btf = true,
.expected_attach_type = BPF_LSM_MAC,
.attach_fn = attach_lsm),
BPF_PROG_SEC("xdp", BPF_PROG_TYPE_XDP),
BPF_PROG_SEC("perf_event", BPF_PROG_TYPE_PERF_EVENT),
BPF_PROG_SEC("lwt_in", BPF_PROG_TYPE_LWT_IN),
@ -6572,6 +6584,7 @@ invalid_prog:
}
#define BTF_TRACE_PREFIX "btf_trace_"
#define BTF_LSM_PREFIX "bpf_lsm_"
#define BTF_MAX_NAME_SIZE 128
static int find_btf_by_prefix_kind(const struct btf *btf, const char *prefix,
@ -6599,6 +6612,9 @@ static inline int __find_vmlinux_btf_id(struct btf *btf, const char *name,
if (attach_type == BPF_TRACE_RAW_TP)
err = find_btf_by_prefix_kind(btf, BTF_TRACE_PREFIX, name,
BTF_KIND_TYPEDEF);
else if (attach_type == BPF_LSM_MAC)
err = find_btf_by_prefix_kind(btf, BTF_LSM_PREFIX, name,
BTF_KIND_FUNC);
else
err = btf__find_by_name_kind(btf, name, BTF_KIND_FUNC);
@ -6756,6 +6772,17 @@ void *bpf_map__priv(const struct bpf_map *map)
return map ? map->priv : ERR_PTR(-EINVAL);
}
int bpf_map__set_initial_value(struct bpf_map *map,
const void *data, size_t size)
{
if (!map->mmaped || map->libbpf_type == LIBBPF_MAP_KCONFIG ||
size != map->def.value_size || map->fd >= 0)
return -EINVAL;
memcpy(map->mmaped, data, size);
return 0;
}
bool bpf_map__is_offload_neutral(const struct bpf_map *map)
{
return map->def.type == BPF_MAP_TYPE_PERF_EVENT_ARRAY;
@ -6951,6 +6978,12 @@ struct bpf_link {
bool disconnected;
};
/* Replace link's underlying BPF program with the new one */
int bpf_link__update_program(struct bpf_link *link, struct bpf_program *prog)
{
return bpf_link_update(bpf_link__fd(link), bpf_program__fd(prog), NULL);
}
/* Release "ownership" of underlying BPF resource (typically, BPF program
* attached to some BPF hook, e.g., tracepoint, kprobe, etc). Disconnected
* link, when destructed through bpf_link__destroy() call won't attempt to
@ -7452,7 +7485,8 @@ static struct bpf_link *attach_raw_tp(const struct bpf_sec_def *sec,
return bpf_program__attach_raw_tracepoint(prog, tp_name);
}
struct bpf_link *bpf_program__attach_trace(struct bpf_program *prog)
/* Common logic for all BPF program types that attach to a btf_id */
static struct bpf_link *bpf_program__attach_btf_id(struct bpf_program *prog)
{
char errmsg[STRERR_BUFSIZE];
struct bpf_link *link;
@ -7474,7 +7508,7 @@ struct bpf_link *bpf_program__attach_trace(struct bpf_program *prog)
if (pfd < 0) {
pfd = -errno;
free(link);
pr_warn("program '%s': failed to attach to trace: %s\n",
pr_warn("program '%s': failed to attach: %s\n",
bpf_program__title(prog, false),
libbpf_strerror_r(pfd, errmsg, sizeof(errmsg)));
return ERR_PTR(pfd);
@ -7483,12 +7517,68 @@ struct bpf_link *bpf_program__attach_trace(struct bpf_program *prog)
return (struct bpf_link *)link;
}
struct bpf_link *bpf_program__attach_trace(struct bpf_program *prog)
{
return bpf_program__attach_btf_id(prog);
}
struct bpf_link *bpf_program__attach_lsm(struct bpf_program *prog)
{
return bpf_program__attach_btf_id(prog);
}
static struct bpf_link *attach_trace(const struct bpf_sec_def *sec,
struct bpf_program *prog)
{
return bpf_program__attach_trace(prog);
}
static struct bpf_link *attach_lsm(const struct bpf_sec_def *sec,
struct bpf_program *prog)
{
return bpf_program__attach_lsm(prog);
}
struct bpf_link *
bpf_program__attach_cgroup(struct bpf_program *prog, int cgroup_fd)
{
const struct bpf_sec_def *sec_def;
enum bpf_attach_type attach_type;
char errmsg[STRERR_BUFSIZE];
struct bpf_link *link;
int prog_fd, link_fd;
prog_fd = bpf_program__fd(prog);
if (prog_fd < 0) {
pr_warn("program '%s': can't attach before loaded\n",
bpf_program__title(prog, false));
return ERR_PTR(-EINVAL);
}
link = calloc(1, sizeof(*link));
if (!link)
return ERR_PTR(-ENOMEM);
link->detach = &bpf_link__detach_fd;
attach_type = bpf_program__get_expected_attach_type(prog);
if (!attach_type) {
sec_def = find_sec_def(bpf_program__title(prog, false));
if (sec_def)
attach_type = sec_def->attach_type;
}
link_fd = bpf_link_create(prog_fd, cgroup_fd, attach_type, NULL);
if (link_fd < 0) {
link_fd = -errno;
free(link);
pr_warn("program '%s': failed to attach to cgroup: %s\n",
bpf_program__title(prog, false),
libbpf_strerror_r(link_fd, errmsg, sizeof(errmsg)));
return ERR_PTR(link_fd);
}
link->fd = link_fd;
return link;
}
struct bpf_link *bpf_program__attach(struct bpf_program *prog)
{
const struct bpf_sec_def *sec_def;

View File

@ -224,6 +224,8 @@ LIBBPF_API int bpf_link__fd(const struct bpf_link *link);
LIBBPF_API const char *bpf_link__pin_path(const struct bpf_link *link);
LIBBPF_API int bpf_link__pin(struct bpf_link *link, const char *path);
LIBBPF_API int bpf_link__unpin(struct bpf_link *link);
LIBBPF_API int bpf_link__update_program(struct bpf_link *link,
struct bpf_program *prog);
LIBBPF_API void bpf_link__disconnect(struct bpf_link *link);
LIBBPF_API int bpf_link__destroy(struct bpf_link *link);
@ -245,11 +247,17 @@ bpf_program__attach_tracepoint(struct bpf_program *prog,
LIBBPF_API struct bpf_link *
bpf_program__attach_raw_tracepoint(struct bpf_program *prog,
const char *tp_name);
LIBBPF_API struct bpf_link *
bpf_program__attach_trace(struct bpf_program *prog);
LIBBPF_API struct bpf_link *
bpf_program__attach_lsm(struct bpf_program *prog);
LIBBPF_API struct bpf_link *
bpf_program__attach_cgroup(struct bpf_program *prog, int cgroup_fd);
struct bpf_map;
LIBBPF_API struct bpf_link *bpf_map__attach_struct_ops(struct bpf_map *map);
struct bpf_insn;
/*
@ -321,6 +329,7 @@ LIBBPF_API int bpf_program__set_socket_filter(struct bpf_program *prog);
LIBBPF_API int bpf_program__set_tracepoint(struct bpf_program *prog);
LIBBPF_API int bpf_program__set_raw_tracepoint(struct bpf_program *prog);
LIBBPF_API int bpf_program__set_kprobe(struct bpf_program *prog);
LIBBPF_API int bpf_program__set_lsm(struct bpf_program *prog);
LIBBPF_API int bpf_program__set_sched_cls(struct bpf_program *prog);
LIBBPF_API int bpf_program__set_sched_act(struct bpf_program *prog);
LIBBPF_API int bpf_program__set_xdp(struct bpf_program *prog);
@ -347,6 +356,7 @@ LIBBPF_API bool bpf_program__is_socket_filter(const struct bpf_program *prog);
LIBBPF_API bool bpf_program__is_tracepoint(const struct bpf_program *prog);
LIBBPF_API bool bpf_program__is_raw_tracepoint(const struct bpf_program *prog);
LIBBPF_API bool bpf_program__is_kprobe(const struct bpf_program *prog);
LIBBPF_API bool bpf_program__is_lsm(const struct bpf_program *prog);
LIBBPF_API bool bpf_program__is_sched_cls(const struct bpf_program *prog);
LIBBPF_API bool bpf_program__is_sched_act(const struct bpf_program *prog);
LIBBPF_API bool bpf_program__is_xdp(const struct bpf_program *prog);
@ -407,6 +417,8 @@ typedef void (*bpf_map_clear_priv_t)(struct bpf_map *, void *);
LIBBPF_API int bpf_map__set_priv(struct bpf_map *map, void *priv,
bpf_map_clear_priv_t clear_priv);
LIBBPF_API void *bpf_map__priv(const struct bpf_map *map);
LIBBPF_API int bpf_map__set_initial_value(struct bpf_map *map,
const void *data, size_t size);
LIBBPF_API int bpf_map__reuse_fd(struct bpf_map *map, int fd);
LIBBPF_API int bpf_map__resize(struct bpf_map *map, __u32 max_entries);
LIBBPF_API bool bpf_map__is_offload_neutral(const struct bpf_map *map);
@ -444,7 +456,15 @@ struct xdp_link_info {
__u8 attach_mode;
};
struct bpf_xdp_set_link_opts {
size_t sz;
__u32 old_fd;
};
#define bpf_xdp_set_link_opts__last_field old_fd
LIBBPF_API int bpf_set_link_xdp_fd(int ifindex, int fd, __u32 flags);
LIBBPF_API int bpf_set_link_xdp_fd_opts(int ifindex, int fd, __u32 flags,
const struct bpf_xdp_set_link_opts *opts);
LIBBPF_API int bpf_get_link_xdp_id(int ifindex, __u32 *prog_id, __u32 flags);
LIBBPF_API int bpf_get_link_xdp_info(int ifindex, struct xdp_link_info *info,
size_t info_size, __u32 flags);

View File

@ -243,5 +243,14 @@ LIBBPF_0.0.8 {
bpf_link__pin;
bpf_link__pin_path;
bpf_link__unpin;
bpf_link__update_program;
bpf_link_create;
bpf_link_update;
bpf_map__set_initial_value;
bpf_program__attach_cgroup;
bpf_program__attach_lsm;
bpf_program__is_lsm;
bpf_program__set_attach_target;
bpf_program__set_lsm;
bpf_set_link_xdp_fd_opts;
} LIBBPF_0.0.7;

View File

@ -108,6 +108,7 @@ probe_load(enum bpf_prog_type prog_type, const struct bpf_insn *insns,
case BPF_PROG_TYPE_TRACING:
case BPF_PROG_TYPE_STRUCT_OPS:
case BPF_PROG_TYPE_EXT:
case BPF_PROG_TYPE_LSM:
default:
break;
}

View File

@ -132,7 +132,8 @@ done:
return ret;
}
int bpf_set_link_xdp_fd(int ifindex, int fd, __u32 flags)
static int __bpf_set_link_xdp_fd_replace(int ifindex, int fd, int old_fd,
__u32 flags)
{
int sock, seq = 0, ret;
struct nlattr *nla, *nla_xdp;
@ -178,6 +179,14 @@ int bpf_set_link_xdp_fd(int ifindex, int fd, __u32 flags)
nla->nla_len += nla_xdp->nla_len;
}
if (flags & XDP_FLAGS_REPLACE) {
nla_xdp = (struct nlattr *)((char *)nla + nla->nla_len);
nla_xdp->nla_type = IFLA_XDP_EXPECTED_FD;
nla_xdp->nla_len = NLA_HDRLEN + sizeof(old_fd);
memcpy((char *)nla_xdp + NLA_HDRLEN, &old_fd, sizeof(old_fd));
nla->nla_len += nla_xdp->nla_len;
}
req.nh.nlmsg_len += NLA_ALIGN(nla->nla_len);
if (send(sock, &req, req.nh.nlmsg_len, 0) < 0) {
@ -191,6 +200,29 @@ cleanup:
return ret;
}
int bpf_set_link_xdp_fd_opts(int ifindex, int fd, __u32 flags,
const struct bpf_xdp_set_link_opts *opts)
{
int old_fd = -1;
if (!OPTS_VALID(opts, bpf_xdp_set_link_opts))
return -EINVAL;
if (OPTS_HAS(opts, old_fd)) {
old_fd = OPTS_GET(opts, old_fd, -1);
flags |= XDP_FLAGS_REPLACE;
}
return __bpf_set_link_xdp_fd_replace(ifindex, fd,
old_fd,
flags);
}
int bpf_set_link_xdp_fd(int ifindex, int fd, __u32 flags)
{
return __bpf_set_link_xdp_fd_replace(ifindex, fd, 0, flags);
}
static int __dump_link_nlmsg(struct nlmsghdr *nlh,
libbpf_dump_nlmsg_t dump_link_nlmsg, void *cookie)
{

View File

@ -280,7 +280,11 @@ int xsk_umem__create_v0_0_4(struct xsk_umem **umem_ptr, void *umem_area,
fill->consumer = map + off.fr.consumer;
fill->flags = map + off.fr.flags;
fill->ring = map + off.fr.desc;
fill->cached_cons = umem->config.fill_size;
fill->cached_prod = *fill->producer;
/* cached_cons is "size" bigger than the real consumer pointer
* See xsk_prod_nb_free
*/
fill->cached_cons = *fill->consumer + umem->config.fill_size;
map = mmap(NULL, off.cr.desc + umem->config.comp_size * sizeof(__u64),
PROT_READ | PROT_WRITE, MAP_SHARED | MAP_POPULATE, umem->fd,
@ -297,6 +301,8 @@ int xsk_umem__create_v0_0_4(struct xsk_umem **umem_ptr, void *umem_area,
comp->consumer = map + off.cr.consumer;
comp->flags = map + off.cr.flags;
comp->ring = map + off.cr.desc;
comp->cached_prod = *comp->producer;
comp->cached_cons = *comp->consumer;
*umem_ptr = umem;
return 0;
@ -672,6 +678,8 @@ int xsk_socket__create(struct xsk_socket **xsk_ptr, const char *ifname,
rx->consumer = rx_map + off.rx.consumer;
rx->flags = rx_map + off.rx.flags;
rx->ring = rx_map + off.rx.desc;
rx->cached_prod = *rx->producer;
rx->cached_cons = *rx->consumer;
}
xsk->rx = rx;
@ -691,7 +699,11 @@ int xsk_socket__create(struct xsk_socket **xsk_ptr, const char *ifname,
tx->consumer = tx_map + off.tx.consumer;
tx->flags = tx_map + off.tx.flags;
tx->ring = tx_map + off.tx.desc;
tx->cached_cons = xsk->config.tx_size;
tx->cached_prod = *tx->producer;
/* cached_cons is r->size bigger than the real consumer pointer
* See xsk_prod_nb_free
*/
tx->cached_cons = *tx->consumer + xsk->config.tx_size;
}
xsk->tx = tx;

View File

@ -35,3 +35,5 @@ CONFIG_MPLS_ROUTING=m
CONFIG_MPLS_IPTUNNEL=m
CONFIG_IPV6_SIT=m
CONFIG_BPF_JIT=y
CONFIG_BPF_LSM=y
CONFIG_SECURITY=y

View File

@ -11,6 +11,7 @@
static const unsigned int total_bytes = 10 * 1024 * 1024;
static const struct timeval timeo_sec = { .tv_sec = 10 };
static const size_t timeo_optlen = sizeof(timeo_sec);
static int expected_stg = 0xeB9F;
static int stop, duration;
static int settimeo(int fd)
@ -88,7 +89,7 @@ done:
return NULL;
}
static void do_test(const char *tcp_ca)
static void do_test(const char *tcp_ca, const struct bpf_map *sk_stg_map)
{
struct sockaddr_in6 sa6 = {};
ssize_t nr_recv = 0, bytes = 0;
@ -126,14 +127,34 @@ static void do_test(const char *tcp_ca)
err = listen(lfd, 1);
if (CHECK(err == -1, "listen", "errno:%d\n", errno))
goto done;
err = pthread_create(&srv_thread, NULL, server, (void *)(long)lfd);
if (CHECK(err != 0, "pthread_create", "err:%d\n", err))
goto done;
if (sk_stg_map) {
err = bpf_map_update_elem(bpf_map__fd(sk_stg_map), &fd,
&expected_stg, BPF_NOEXIST);
if (CHECK(err, "bpf_map_update_elem(sk_stg_map)",
"err:%d errno:%d\n", err, errno))
goto done;
}
/* connect to server */
err = connect(fd, (struct sockaddr *)&sa6, addrlen);
if (CHECK(err == -1, "connect", "errno:%d\n", errno))
goto wait_thread;
goto done;
if (sk_stg_map) {
int tmp_stg;
err = bpf_map_lookup_elem(bpf_map__fd(sk_stg_map), &fd,
&tmp_stg);
if (CHECK(!err || errno != ENOENT,
"bpf_map_lookup_elem(sk_stg_map)",
"err:%d errno:%d\n", err, errno))
goto done;
}
err = pthread_create(&srv_thread, NULL, server, (void *)(long)lfd);
if (CHECK(err != 0, "pthread_create", "err:%d errno:%d\n", err, errno))
goto done;
/* recv total_bytes */
while (bytes < total_bytes && !READ_ONCE(stop)) {
@ -149,7 +170,6 @@ static void do_test(const char *tcp_ca)
CHECK(bytes != total_bytes, "recv", "%zd != %u nr_recv:%zd errno:%d\n",
bytes, total_bytes, nr_recv, errno);
wait_thread:
WRITE_ONCE(stop, 1);
pthread_join(srv_thread, &thread_ret);
CHECK(IS_ERR(thread_ret), "pthread_join", "thread_ret:%ld",
@ -175,7 +195,7 @@ static void test_cubic(void)
return;
}
do_test("bpf_cubic");
do_test("bpf_cubic", NULL);
bpf_link__destroy(link);
bpf_cubic__destroy(cubic_skel);
@ -197,7 +217,10 @@ static void test_dctcp(void)
return;
}
do_test("bpf_dctcp");
do_test("bpf_dctcp", dctcp_skel->maps.sk_stg_map);
CHECK(dctcp_skel->bss->stg_result != expected_stg,
"Unexpected stg_result", "stg_result (%x) != expected_stg (%x)\n",
dctcp_skel->bss->stg_result, expected_stg);
bpf_link__destroy(link);
bpf_dctcp__destroy(dctcp_skel);

View File

@ -125,6 +125,6 @@ void test_btf_dump() {
if (!test__start_subtest(t->name))
continue;
test_btf_dump_case(i, &btf_dump_test_cases[i]);
test_btf_dump_case(i, &btf_dump_test_cases[i]);
}
}

View File

@ -0,0 +1,244 @@
// SPDX-License-Identifier: GPL-2.0
#include <test_progs.h>
#include "cgroup_helpers.h"
#include "test_cgroup_link.skel.h"
static __u32 duration = 0;
#define PING_CMD "ping -q -c1 -w1 127.0.0.1 > /dev/null"
static struct test_cgroup_link *skel = NULL;
int ping_and_check(int exp_calls, int exp_alt_calls)
{
skel->bss->calls = 0;
skel->bss->alt_calls = 0;
CHECK_FAIL(system(PING_CMD));
if (CHECK(skel->bss->calls != exp_calls, "call_cnt",
"exp %d, got %d\n", exp_calls, skel->bss->calls))
return -EINVAL;
if (CHECK(skel->bss->alt_calls != exp_alt_calls, "alt_call_cnt",
"exp %d, got %d\n", exp_alt_calls, skel->bss->alt_calls))
return -EINVAL;
return 0;
}
void test_cgroup_link(void)
{
struct {
const char *path;
int fd;
} cgs[] = {
{ "/cg1" },
{ "/cg1/cg2" },
{ "/cg1/cg2/cg3" },
{ "/cg1/cg2/cg3/cg4" },
};
int last_cg = ARRAY_SIZE(cgs) - 1, cg_nr = ARRAY_SIZE(cgs);
DECLARE_LIBBPF_OPTS(bpf_link_update_opts, link_upd_opts);
struct bpf_link *links[ARRAY_SIZE(cgs)] = {}, *tmp_link;
__u32 prog_ids[ARRAY_SIZE(cgs)], prog_cnt = 0, attach_flags;
int i = 0, err, prog_fd;
bool detach_legacy = false;
skel = test_cgroup_link__open_and_load();
if (CHECK(!skel, "skel_open_load", "failed to open/load skeleton\n"))
return;
prog_fd = bpf_program__fd(skel->progs.egress);
err = setup_cgroup_environment();
if (CHECK(err, "cg_init", "failed: %d\n", err))
goto cleanup;
for (i = 0; i < cg_nr; i++) {
cgs[i].fd = create_and_get_cgroup(cgs[i].path);
if (CHECK(cgs[i].fd < 0, "cg_create", "fail: %d\n", cgs[i].fd))
goto cleanup;
}
err = join_cgroup(cgs[last_cg].path);
if (CHECK(err, "cg_join", "fail: %d\n", err))
goto cleanup;
for (i = 0; i < cg_nr; i++) {
links[i] = bpf_program__attach_cgroup(skel->progs.egress,
cgs[i].fd);
if (CHECK(IS_ERR(links[i]), "cg_attach", "i: %d, err: %ld\n",
i, PTR_ERR(links[i])))
goto cleanup;
}
ping_and_check(cg_nr, 0);
/* query the number of effective progs and attach flags in root cg */
err = bpf_prog_query(cgs[0].fd, BPF_CGROUP_INET_EGRESS,
BPF_F_QUERY_EFFECTIVE, &attach_flags, NULL,
&prog_cnt);
CHECK_FAIL(err);
CHECK_FAIL(attach_flags != BPF_F_ALLOW_MULTI);
if (CHECK(prog_cnt != 1, "effect_cnt", "exp %d, got %d\n", 1, prog_cnt))
goto cleanup;
/* query the number of effective progs in last cg */
err = bpf_prog_query(cgs[last_cg].fd, BPF_CGROUP_INET_EGRESS,
BPF_F_QUERY_EFFECTIVE, NULL, NULL,
&prog_cnt);
CHECK_FAIL(err);
CHECK_FAIL(attach_flags != BPF_F_ALLOW_MULTI);
if (CHECK(prog_cnt != cg_nr, "effect_cnt", "exp %d, got %d\n",
cg_nr, prog_cnt))
goto cleanup;
/* query the effective prog IDs in last cg */
err = bpf_prog_query(cgs[last_cg].fd, BPF_CGROUP_INET_EGRESS,
BPF_F_QUERY_EFFECTIVE, &attach_flags,
prog_ids, &prog_cnt);
CHECK_FAIL(err);
CHECK_FAIL(attach_flags != BPF_F_ALLOW_MULTI);
if (CHECK(prog_cnt != cg_nr, "effect_cnt", "exp %d, got %d\n",
cg_nr, prog_cnt))
goto cleanup;
for (i = 1; i < prog_cnt; i++) {
CHECK(prog_ids[i - 1] != prog_ids[i], "prog_id_check",
"idx %d, prev id %d, cur id %d\n",
i, prog_ids[i - 1], prog_ids[i]);
}
/* detach bottom program and ping again */
bpf_link__destroy(links[last_cg]);
links[last_cg] = NULL;
ping_and_check(cg_nr - 1, 0);
/* mix in with non link-based multi-attachments */
err = bpf_prog_attach(prog_fd, cgs[last_cg].fd,
BPF_CGROUP_INET_EGRESS, BPF_F_ALLOW_MULTI);
if (CHECK(err, "cg_attach_legacy", "errno=%d\n", errno))
goto cleanup;
detach_legacy = true;
links[last_cg] = bpf_program__attach_cgroup(skel->progs.egress,
cgs[last_cg].fd);
if (CHECK(IS_ERR(links[last_cg]), "cg_attach", "err: %ld\n",
PTR_ERR(links[last_cg])))
goto cleanup;
ping_and_check(cg_nr + 1, 0);
/* detach link */
bpf_link__destroy(links[last_cg]);
links[last_cg] = NULL;
/* detach legacy */
err = bpf_prog_detach2(prog_fd, cgs[last_cg].fd, BPF_CGROUP_INET_EGRESS);
if (CHECK(err, "cg_detach_legacy", "errno=%d\n", errno))
goto cleanup;
detach_legacy = false;
/* attach legacy exclusive prog attachment */
err = bpf_prog_attach(prog_fd, cgs[last_cg].fd,
BPF_CGROUP_INET_EGRESS, 0);
if (CHECK(err, "cg_attach_exclusive", "errno=%d\n", errno))
goto cleanup;
detach_legacy = true;
/* attempt to mix in with multi-attach bpf_link */
tmp_link = bpf_program__attach_cgroup(skel->progs.egress,
cgs[last_cg].fd);
if (CHECK(!IS_ERR(tmp_link), "cg_attach_fail", "unexpected success!\n")) {
bpf_link__destroy(tmp_link);
goto cleanup;
}
ping_and_check(cg_nr, 0);
/* detach */
err = bpf_prog_detach2(prog_fd, cgs[last_cg].fd, BPF_CGROUP_INET_EGRESS);
if (CHECK(err, "cg_detach_legacy", "errno=%d\n", errno))
goto cleanup;
detach_legacy = false;
ping_and_check(cg_nr - 1, 0);
/* attach back link-based one */
links[last_cg] = bpf_program__attach_cgroup(skel->progs.egress,
cgs[last_cg].fd);
if (CHECK(IS_ERR(links[last_cg]), "cg_attach", "err: %ld\n",
PTR_ERR(links[last_cg])))
goto cleanup;
ping_and_check(cg_nr, 0);
/* check legacy exclusive prog can't be attached */
err = bpf_prog_attach(prog_fd, cgs[last_cg].fd,
BPF_CGROUP_INET_EGRESS, 0);
if (CHECK(!err, "cg_attach_exclusive", "unexpected success")) {
bpf_prog_detach2(prog_fd, cgs[last_cg].fd, BPF_CGROUP_INET_EGRESS);
goto cleanup;
}
/* replace BPF programs inside their links for all but first link */
for (i = 1; i < cg_nr; i++) {
err = bpf_link__update_program(links[i], skel->progs.egress_alt);
if (CHECK(err, "prog_upd", "link #%d\n", i))
goto cleanup;
}
ping_and_check(1, cg_nr - 1);
/* Attempt program update with wrong expected BPF program */
link_upd_opts.old_prog_fd = bpf_program__fd(skel->progs.egress_alt);
link_upd_opts.flags = BPF_F_REPLACE;
err = bpf_link_update(bpf_link__fd(links[0]),
bpf_program__fd(skel->progs.egress_alt),
&link_upd_opts);
if (CHECK(err == 0 || errno != EPERM, "prog_cmpxchg1",
"unexpectedly succeeded, err %d, errno %d\n", err, -errno))
goto cleanup;
/* Compare-exchange single link program from egress to egress_alt */
link_upd_opts.old_prog_fd = bpf_program__fd(skel->progs.egress);
link_upd_opts.flags = BPF_F_REPLACE;
err = bpf_link_update(bpf_link__fd(links[0]),
bpf_program__fd(skel->progs.egress_alt),
&link_upd_opts);
if (CHECK(err, "prog_cmpxchg2", "errno %d\n", -errno))
goto cleanup;
/* ping */
ping_and_check(0, cg_nr);
/* close cgroup FDs before detaching links */
for (i = 0; i < cg_nr; i++) {
if (cgs[i].fd > 0) {
close(cgs[i].fd);
cgs[i].fd = -1;
}
}
/* BPF programs should still get called */
ping_and_check(0, cg_nr);
/* leave cgroup and remove them, don't detach programs */
cleanup_cgroup_environment();
/* BPF programs should have been auto-detached */
ping_and_check(0, 0);
cleanup:
if (detach_legacy)
bpf_prog_detach2(prog_fd, cgs[last_cg].fd,
BPF_CGROUP_INET_EGRESS);
for (i = 0; i < cg_nr; i++) {
if (!IS_ERR(links[i]))
bpf_link__destroy(links[i]);
}
test_cgroup_link__destroy(skel);
for (i = 0; i < cg_nr; i++) {
if (cgs[i].fd > 0)
close(cgs[i].fd);
}
cleanup_cgroup_environment();
}

View File

@ -82,6 +82,7 @@ static void get_stack_print_output(void *ctx, int cpu, void *data, __u32 size)
void test_get_stack_raw_tp(void)
{
const char *file = "./test_get_stack_rawtp.o";
const char *file_err = "./test_get_stack_rawtp_err.o";
const char *prog_name = "raw_tracepoint/sys_enter";
int i, err, prog_fd, exp_cnt = MAX_CNT_RAWTP;
struct perf_buffer_opts pb_opts = {};
@ -93,6 +94,10 @@ void test_get_stack_raw_tp(void)
struct bpf_map *map;
cpu_set_t cpu_set;
err = bpf_prog_load(file_err, BPF_PROG_TYPE_RAW_TRACEPOINT, &obj, &prog_fd);
if (CHECK(err >= 0, "prog_load raw tp", "err %d errno %d\n", err, errno))
return;
err = bpf_prog_load(file, BPF_PROG_TYPE_RAW_TRACEPOINT, &obj, &prog_fd);
if (CHECK(err, "prog_load raw tp", "err %d errno %d\n", err, errno))
return;

View File

@ -0,0 +1,61 @@
// SPDX-License-Identifier: GPL-2.0
#include <test_progs.h>
void test_global_data_init(void)
{
const char *file = "./test_global_data.o";
int err = -ENOMEM, map_fd, zero = 0;
__u8 *buff = NULL, *newval = NULL;
struct bpf_object *obj;
struct bpf_map *map;
__u32 duration = 0;
size_t sz;
obj = bpf_object__open_file(file, NULL);
if (CHECK_FAIL(!obj))
return;
map = bpf_object__find_map_by_name(obj, "test_glo.rodata");
if (CHECK_FAIL(!map || !bpf_map__is_internal(map)))
goto out;
sz = bpf_map__def(map)->value_size;
newval = malloc(sz);
if (CHECK_FAIL(!newval))
goto out;
memset(newval, 0, sz);
/* wrong size, should fail */
err = bpf_map__set_initial_value(map, newval, sz - 1);
if (CHECK(!err, "reject set initial value wrong size", "err %d\n", err))
goto out;
err = bpf_map__set_initial_value(map, newval, sz);
if (CHECK(err, "set initial value", "err %d\n", err))
goto out;
err = bpf_object__load(obj);
if (CHECK_FAIL(err))
goto out;
map_fd = bpf_map__fd(map);
if (CHECK_FAIL(map_fd < 0))
goto out;
buff = malloc(sz);
if (buff)
err = bpf_map_lookup_elem(map_fd, &zero, buff);
if (CHECK(!buff || err || memcmp(buff, newval, sz),
"compare .rodata map data override",
"err %d errno %d\n", err, errno))
goto out;
memset(newval, 1, sz);
/* object loaded - should fail */
err = bpf_map__set_initial_value(map, newval, sz);
CHECK(!err, "reject set initial value after load", "err %d\n", err);
out:
free(buff);
free(newval);
bpf_object__close(obj);
}

View File

@ -0,0 +1,309 @@
// SPDX-License-Identifier: GPL-2.0
// Copyright (c) 2018 Facebook
// Copyright (c) 2019 Cloudflare
// Copyright (c) 2020 Isovalent, Inc.
/*
* Test that the socket assign program is able to redirect traffic towards a
* socket, regardless of whether the port or address destination of the traffic
* matches the port.
*/
#define _GNU_SOURCE
#include <fcntl.h>
#include <signal.h>
#include <stdlib.h>
#include <unistd.h>
#include "test_progs.h"
#define BIND_PORT 1234
#define CONNECT_PORT 4321
#define TEST_DADDR (0xC0A80203)
#define NS_SELF "/proc/self/ns/net"
static const struct timeval timeo_sec = { .tv_sec = 3 };
static const size_t timeo_optlen = sizeof(timeo_sec);
static int stop, duration;
static bool
configure_stack(void)
{
char tc_cmd[BUFSIZ];
/* Move to a new networking namespace */
if (CHECK_FAIL(unshare(CLONE_NEWNET)))
return false;
/* Configure necessary links, routes */
if (CHECK_FAIL(system("ip link set dev lo up")))
return false;
if (CHECK_FAIL(system("ip route add local default dev lo")))
return false;
if (CHECK_FAIL(system("ip -6 route add local default dev lo")))
return false;
/* Load qdisc, BPF program */
if (CHECK_FAIL(system("tc qdisc add dev lo clsact")))
return false;
sprintf(tc_cmd, "%s %s %s %s", "tc filter add dev lo ingress bpf",
"direct-action object-file ./test_sk_assign.o",
"section classifier/sk_assign_test",
(env.verbosity < VERBOSE_VERY) ? " 2>/dev/null" : "");
if (CHECK(system(tc_cmd), "BPF load failed;",
"run with -vv for more info\n"))
return false;
return true;
}
static int
start_server(const struct sockaddr *addr, socklen_t len, int type)
{
int fd;
fd = socket(addr->sa_family, type, 0);
if (CHECK_FAIL(fd == -1))
goto out;
if (CHECK_FAIL(setsockopt(fd, SOL_SOCKET, SO_RCVTIMEO, &timeo_sec,
timeo_optlen)))
goto close_out;
if (CHECK_FAIL(bind(fd, addr, len) == -1))
goto close_out;
if (type == SOCK_STREAM && CHECK_FAIL(listen(fd, 128) == -1))
goto close_out;
goto out;
close_out:
close(fd);
fd = -1;
out:
return fd;
}
static int
connect_to_server(const struct sockaddr *addr, socklen_t len, int type)
{
int fd = -1;
fd = socket(addr->sa_family, type, 0);
if (CHECK_FAIL(fd == -1))
goto out;
if (CHECK_FAIL(setsockopt(fd, SOL_SOCKET, SO_SNDTIMEO, &timeo_sec,
timeo_optlen)))
goto close_out;
if (CHECK_FAIL(connect(fd, addr, len)))
goto close_out;
goto out;
close_out:
close(fd);
fd = -1;
out:
return fd;
}
static in_port_t
get_port(int fd)
{
struct sockaddr_storage ss;
socklen_t slen = sizeof(ss);
in_port_t port = 0;
if (CHECK_FAIL(getsockname(fd, (struct sockaddr *)&ss, &slen)))
return port;
switch (ss.ss_family) {
case AF_INET:
port = ((struct sockaddr_in *)&ss)->sin_port;
break;
case AF_INET6:
port = ((struct sockaddr_in6 *)&ss)->sin6_port;
break;
default:
CHECK(1, "Invalid address family", "%d\n", ss.ss_family);
}
return port;
}
static ssize_t
rcv_msg(int srv_client, int type)
{
struct sockaddr_storage ss;
char buf[BUFSIZ];
socklen_t slen;
if (type == SOCK_STREAM)
return read(srv_client, &buf, sizeof(buf));
else
return recvfrom(srv_client, &buf, sizeof(buf), 0,
(struct sockaddr *)&ss, &slen);
}
static int
run_test(int server_fd, const struct sockaddr *addr, socklen_t len, int type)
{
int client = -1, srv_client = -1;
char buf[] = "testing";
in_port_t port;
int ret = 1;
client = connect_to_server(addr, len, type);
if (client == -1) {
perror("Cannot connect to server");
goto out;
}
if (type == SOCK_STREAM) {
srv_client = accept(server_fd, NULL, NULL);
if (CHECK_FAIL(srv_client == -1)) {
perror("Can't accept connection");
goto out;
}
} else {
srv_client = server_fd;
}
if (CHECK_FAIL(write(client, buf, sizeof(buf)) != sizeof(buf))) {
perror("Can't write on client");
goto out;
}
if (CHECK_FAIL(rcv_msg(srv_client, type) != sizeof(buf))) {
perror("Can't read on server");
goto out;
}
port = get_port(srv_client);
if (CHECK_FAIL(!port))
goto out;
/* SOCK_STREAM is connected via accept(), so the server's local address
* will be the CONNECT_PORT rather than the BIND port that corresponds
* to the listen socket. SOCK_DGRAM on the other hand is connectionless
* so we can't really do the same check there; the server doesn't ever
* create a socket with CONNECT_PORT.
*/
if (type == SOCK_STREAM &&
CHECK(port != htons(CONNECT_PORT), "Expected", "port %u but got %u",
CONNECT_PORT, ntohs(port)))
goto out;
else if (type == SOCK_DGRAM &&
CHECK(port != htons(BIND_PORT), "Expected",
"port %u but got %u", BIND_PORT, ntohs(port)))
goto out;
ret = 0;
out:
close(client);
if (srv_client != server_fd)
close(srv_client);
if (ret)
WRITE_ONCE(stop, 1);
return ret;
}
static void
prepare_addr(struct sockaddr *addr, int family, __u16 port, bool rewrite_addr)
{
struct sockaddr_in *addr4;
struct sockaddr_in6 *addr6;
switch (family) {
case AF_INET:
addr4 = (struct sockaddr_in *)addr;
memset(addr4, 0, sizeof(*addr4));
addr4->sin_family = family;
addr4->sin_port = htons(port);
if (rewrite_addr)
addr4->sin_addr.s_addr = htonl(TEST_DADDR);
else
addr4->sin_addr.s_addr = htonl(INADDR_LOOPBACK);
break;
case AF_INET6:
addr6 = (struct sockaddr_in6 *)addr;
memset(addr6, 0, sizeof(*addr6));
addr6->sin6_family = family;
addr6->sin6_port = htons(port);
addr6->sin6_addr = in6addr_loopback;
if (rewrite_addr)
addr6->sin6_addr.s6_addr32[3] = htonl(TEST_DADDR);
break;
default:
fprintf(stderr, "Invalid family %d", family);
}
}
struct test_sk_cfg {
const char *name;
int family;
struct sockaddr *addr;
socklen_t len;
int type;
bool rewrite_addr;
};
#define TEST(NAME, FAMILY, TYPE, REWRITE) \
{ \
.name = NAME, \
.family = FAMILY, \
.addr = (FAMILY == AF_INET) ? (struct sockaddr *)&addr4 \
: (struct sockaddr *)&addr6, \
.len = (FAMILY == AF_INET) ? sizeof(addr4) : sizeof(addr6), \
.type = TYPE, \
.rewrite_addr = REWRITE, \
}
void test_sk_assign(void)
{
struct sockaddr_in addr4;
struct sockaddr_in6 addr6;
struct test_sk_cfg tests[] = {
TEST("ipv4 tcp port redir", AF_INET, SOCK_STREAM, false),
TEST("ipv4 tcp addr redir", AF_INET, SOCK_STREAM, true),
TEST("ipv6 tcp port redir", AF_INET6, SOCK_STREAM, false),
TEST("ipv6 tcp addr redir", AF_INET6, SOCK_STREAM, true),
TEST("ipv4 udp port redir", AF_INET, SOCK_DGRAM, false),
TEST("ipv4 udp addr redir", AF_INET, SOCK_DGRAM, true),
TEST("ipv6 udp port redir", AF_INET6, SOCK_DGRAM, false),
TEST("ipv6 udp addr redir", AF_INET6, SOCK_DGRAM, true),
};
int server = -1;
int self_net;
self_net = open(NS_SELF, O_RDONLY);
if (CHECK_FAIL(self_net < 0)) {
perror("Unable to open "NS_SELF);
return;
}
if (!configure_stack()) {
perror("configure_stack");
goto cleanup;
}
for (int i = 0; i < ARRAY_SIZE(tests) && !READ_ONCE(stop); i++) {
struct test_sk_cfg *test = &tests[i];
const struct sockaddr *addr;
if (!test__start_subtest(test->name))
continue;
prepare_addr(test->addr, test->family, BIND_PORT, false);
addr = (const struct sockaddr *)test->addr;
server = start_server(addr, test->len, test->type);
if (server == -1)
goto cleanup;
/* connect to unbound ports */
prepare_addr(test->addr, test->family, CONNECT_PORT,
test->rewrite_addr);
if (run_test(server, addr, test->len, test->type))
goto close;
close(server);
server = -1;
}
close:
close(server);
cleanup:
if (CHECK_FAIL(setns(self_net, CLONE_NEWNET)))
perror("Failed to setns("NS_SELF")");
close(self_net);
}

View File

@ -226,7 +226,7 @@ static void *server_thread(void *arg)
return ERR_PTR(err);
}
while (!server_done) {
while (true) {
client_fd = accept(fd, (struct sockaddr *)&addr, &len);
if (client_fd == -1 && errno == EAGAIN) {
usleep(50);
@ -272,7 +272,7 @@ void test_tcp_rtt(void)
CHECK_FAIL(run_test(cgroup_fd, server_fd));
server_done = true;
pthread_join(tid, &server_res);
CHECK_FAIL(pthread_join(tid, &server_res));
CHECK_FAIL(IS_ERR(server_res));
close_server_fd:

View File

@ -0,0 +1,86 @@
// SPDX-License-Identifier: GPL-2.0
/*
* Copyright (C) 2020 Google LLC.
*/
#include <test_progs.h>
#include <sys/mman.h>
#include <sys/wait.h>
#include <unistd.h>
#include <malloc.h>
#include <stdlib.h>
#include "lsm.skel.h"
char *CMD_ARGS[] = {"true", NULL};
int heap_mprotect(void)
{
void *buf;
long sz;
int ret;
sz = sysconf(_SC_PAGESIZE);
if (sz < 0)
return sz;
buf = memalign(sz, 2 * sz);
if (buf == NULL)
return -ENOMEM;
ret = mprotect(buf, sz, PROT_READ | PROT_WRITE | PROT_EXEC);
free(buf);
return ret;
}
int exec_cmd(int *monitored_pid)
{
int child_pid, child_status;
child_pid = fork();
if (child_pid == 0) {
*monitored_pid = getpid();
execvp(CMD_ARGS[0], CMD_ARGS);
return -EINVAL;
} else if (child_pid > 0) {
waitpid(child_pid, &child_status, 0);
return child_status;
}
return -EINVAL;
}
void test_test_lsm(void)
{
struct lsm *skel = NULL;
int err, duration = 0;
skel = lsm__open_and_load();
if (CHECK(!skel, "skel_load", "lsm skeleton failed\n"))
goto close_prog;
err = lsm__attach(skel);
if (CHECK(err, "attach", "lsm attach failed: %d\n", err))
goto close_prog;
err = exec_cmd(&skel->bss->monitored_pid);
if (CHECK(err < 0, "exec_cmd", "err %d errno %d\n", err, errno))
goto close_prog;
CHECK(skel->bss->bprm_count != 1, "bprm_count", "bprm_count = %d\n",
skel->bss->bprm_count);
skel->bss->monitored_pid = getpid();
err = heap_mprotect();
if (CHECK(errno != EPERM, "heap_mprotect", "want errno=EPERM, got %d\n",
errno))
goto close_prog;
CHECK(skel->bss->mprotect_count != 1, "mprotect_count",
"mprotect_count = %d\n", skel->bss->mprotect_count);
close_prog:
lsm__destroy(skel);
}

View File

@ -11,7 +11,7 @@ static void nsleep()
{
struct timespec ts = { .tv_nsec = MY_TV_NSEC };
(void)nanosleep(&ts, NULL);
(void)syscall(__NR_nanosleep, &ts, NULL);
}
void test_vmlinux(void)

View File

@ -0,0 +1,62 @@
// SPDX-License-Identifier: GPL-2.0
#include <test_progs.h>
#define IFINDEX_LO 1
#define XDP_FLAGS_REPLACE (1U << 4)
void test_xdp_attach(void)
{
struct bpf_object *obj1, *obj2, *obj3;
const char *file = "./test_xdp.o";
int err, fd1, fd2, fd3;
__u32 duration = 0;
DECLARE_LIBBPF_OPTS(bpf_xdp_set_link_opts, opts,
.old_fd = -1);
err = bpf_prog_load(file, BPF_PROG_TYPE_XDP, &obj1, &fd1);
if (CHECK_FAIL(err))
return;
err = bpf_prog_load(file, BPF_PROG_TYPE_XDP, &obj2, &fd2);
if (CHECK_FAIL(err))
goto out_1;
err = bpf_prog_load(file, BPF_PROG_TYPE_XDP, &obj3, &fd3);
if (CHECK_FAIL(err))
goto out_2;
err = bpf_set_link_xdp_fd_opts(IFINDEX_LO, fd1, XDP_FLAGS_REPLACE,
&opts);
if (CHECK(err, "load_ok", "initial load failed"))
goto out_close;
err = bpf_set_link_xdp_fd_opts(IFINDEX_LO, fd2, XDP_FLAGS_REPLACE,
&opts);
if (CHECK(!err, "load_fail", "load with expected id didn't fail"))
goto out;
opts.old_fd = fd1;
err = bpf_set_link_xdp_fd_opts(IFINDEX_LO, fd2, 0, &opts);
if (CHECK(err, "replace_ok", "replace valid old_fd failed"))
goto out;
err = bpf_set_link_xdp_fd_opts(IFINDEX_LO, fd3, 0, &opts);
if (CHECK(!err, "replace_fail", "replace invalid old_fd didn't fail"))
goto out;
err = bpf_set_link_xdp_fd_opts(IFINDEX_LO, -1, 0, &opts);
if (CHECK(!err, "remove_fail", "remove invalid old_fd didn't fail"))
goto out;
opts.old_fd = fd2;
err = bpf_set_link_xdp_fd_opts(IFINDEX_LO, -1, 0, &opts);
if (CHECK(err, "remove_ok", "remove valid old_fd failed"))
goto out;
out:
bpf_set_link_xdp_fd(IFINDEX_LO, -1, 0);
out_close:
bpf_object__close(obj3);
out_2:
bpf_object__close(obj2);
out_1:
bpf_object__close(obj1);
}

View File

@ -6,6 +6,7 @@
* the kernel BPF logic.
*/
#include <stddef.h>
#include <linux/bpf.h>
#include <linux/types.h>
#include <bpf/bpf_helpers.h>
@ -14,6 +15,15 @@
char _license[] SEC("license") = "GPL";
int stg_result = 0;
struct {
__uint(type, BPF_MAP_TYPE_SK_STORAGE);
__uint(map_flags, BPF_F_NO_PREALLOC);
__type(key, int);
__type(value, int);
} sk_stg_map SEC(".maps");
#define DCTCP_MAX_ALPHA 1024U
struct dctcp {
@ -43,12 +53,18 @@ void BPF_PROG(dctcp_init, struct sock *sk)
{
const struct tcp_sock *tp = tcp_sk(sk);
struct dctcp *ca = inet_csk_ca(sk);
int *stg;
ca->prior_rcv_nxt = tp->rcv_nxt;
ca->dctcp_alpha = min(dctcp_alpha_on_init, DCTCP_MAX_ALPHA);
ca->loss_cwnd = 0;
ca->ce_state = 0;
stg = bpf_sk_storage_get(&sk_stg_map, (void *)tp, NULL, 0);
if (stg) {
stg_result = *stg;
bpf_sk_storage_delete(&sk_stg_map, (void *)tp);
}
dctcp_reset(tp, ca);
}

View File

@ -0,0 +1,48 @@
// SPDX-License-Identifier: GPL-2.0
/*
* Copyright 2020 Google LLC.
*/
#include "vmlinux.h"
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_tracing.h>
#include <errno.h>
char _license[] SEC("license") = "GPL";
int monitored_pid = 0;
int mprotect_count = 0;
int bprm_count = 0;
SEC("lsm/file_mprotect")
int BPF_PROG(test_int_hook, struct vm_area_struct *vma,
unsigned long reqprot, unsigned long prot, int ret)
{
if (ret != 0)
return ret;
__u32 pid = bpf_get_current_pid_tgid() >> 32;
int is_heap = 0;
is_heap = (vma->vm_start >= vma->vm_mm->start_brk &&
vma->vm_end <= vma->vm_mm->brk);
if (is_heap && monitored_pid == pid) {
mprotect_count++;
ret = -EPERM;
}
return ret;
}
SEC("lsm/bprm_committed_creds")
int BPF_PROG(test_void_hook, struct linux_binprm *bprm)
{
__u32 pid = bpf_get_current_pid_tgid() >> 32;
if (monitored_pid == pid)
bprm_count++;
return 0;
}

View File

@ -12,7 +12,6 @@ int bpf_prog1(struct __sk_buff *skb)
__u32 lport = skb->local_port;
__u32 rport = skb->remote_port;
__u8 *d = data;
__u32 len = (__u32) data_end - (__u32) data;
int err;
if (data + 10 > data_end) {

View File

@ -0,0 +1,24 @@
// SPDX-License-Identifier: GPL-2.0
// Copyright (c) 2020 Facebook
#include <linux/bpf.h>
#include <bpf/bpf_helpers.h>
int calls = 0;
int alt_calls = 0;
SEC("cgroup_skb/egress1")
int egress(struct __sk_buff *skb)
{
__sync_fetch_and_add(&calls, 1);
return 1;
}
SEC("cgroup_skb/egress2")
int egress_alt(struct __sk_buff *skb)
{
__sync_fetch_and_add(&alt_calls, 1);
return 1;
}
char _license[] SEC("license") = "GPL";

View File

@ -0,0 +1,26 @@
// SPDX-License-Identifier: GPL-2.0
#include <linux/bpf.h>
#include <bpf/bpf_helpers.h>
#define MAX_STACK_RAWTP 10
SEC("raw_tracepoint/sys_enter")
int bpf_prog2(void *ctx)
{
__u64 stack[MAX_STACK_RAWTP];
int error;
/* set all the flags which should return -EINVAL */
error = bpf_get_stack(ctx, stack, 0, -1);
if (error < 0)
goto loop;
return error;
loop:
while (1) {
error++;
}
}
char _license[] SEC("license") = "GPL";

View File

@ -68,7 +68,7 @@ static struct foo struct3 = {
bpf_map_update_elem(&result_##map, &key, var, 0); \
} while (0)
SEC("static_data_load")
SEC("classifier/static_data_load")
int load_static_data(struct __sk_buff *skb)
{
static const __u64 bar = ~0;

View File

@ -0,0 +1,204 @@
// SPDX-License-Identifier: GPL-2.0
// Copyright (c) 2019 Cloudflare Ltd.
// Copyright (c) 2020 Isovalent, Inc.
#include <stddef.h>
#include <stdbool.h>
#include <string.h>
#include <linux/bpf.h>
#include <linux/if_ether.h>
#include <linux/in.h>
#include <linux/ip.h>
#include <linux/ipv6.h>
#include <linux/pkt_cls.h>
#include <linux/tcp.h>
#include <sys/socket.h>
#include <bpf/bpf_helpers.h>
#include <bpf/bpf_endian.h>
int _version SEC("version") = 1;
char _license[] SEC("license") = "GPL";
/* Fill 'tuple' with L3 info, and attempt to find L4. On fail, return NULL. */
static inline struct bpf_sock_tuple *
get_tuple(struct __sk_buff *skb, bool *ipv4, bool *tcp)
{
void *data_end = (void *)(long)skb->data_end;
void *data = (void *)(long)skb->data;
struct bpf_sock_tuple *result;
struct ethhdr *eth;
__u64 tuple_len;
__u8 proto = 0;
__u64 ihl_len;
eth = (struct ethhdr *)(data);
if (eth + 1 > data_end)
return NULL;
if (eth->h_proto == bpf_htons(ETH_P_IP)) {
struct iphdr *iph = (struct iphdr *)(data + sizeof(*eth));
if (iph + 1 > data_end)
return NULL;
if (iph->ihl != 5)
/* Options are not supported */
return NULL;
ihl_len = iph->ihl * 4;
proto = iph->protocol;
*ipv4 = true;
result = (struct bpf_sock_tuple *)&iph->saddr;
} else if (eth->h_proto == bpf_htons(ETH_P_IPV6)) {
struct ipv6hdr *ip6h = (struct ipv6hdr *)(data + sizeof(*eth));
if (ip6h + 1 > data_end)
return NULL;
ihl_len = sizeof(*ip6h);
proto = ip6h->nexthdr;
*ipv4 = false;
result = (struct bpf_sock_tuple *)&ip6h->saddr;
} else {
return (struct bpf_sock_tuple *)data;
}
if (proto != IPPROTO_TCP && proto != IPPROTO_UDP)
return NULL;
*tcp = (proto == IPPROTO_TCP);
return result;
}
static inline int
handle_udp(struct __sk_buff *skb, struct bpf_sock_tuple *tuple, bool ipv4)
{
struct bpf_sock_tuple ln = {0};
struct bpf_sock *sk;
size_t tuple_len;
int ret;
tuple_len = ipv4 ? sizeof(tuple->ipv4) : sizeof(tuple->ipv6);
if ((void *)tuple + tuple_len > (void *)(long)skb->data_end)
return TC_ACT_SHOT;
sk = bpf_sk_lookup_udp(skb, tuple, tuple_len, BPF_F_CURRENT_NETNS, 0);
if (sk)
goto assign;
if (ipv4) {
if (tuple->ipv4.dport != bpf_htons(4321))
return TC_ACT_OK;
ln.ipv4.daddr = bpf_htonl(0x7f000001);
ln.ipv4.dport = bpf_htons(1234);
sk = bpf_sk_lookup_udp(skb, &ln, sizeof(ln.ipv4),
BPF_F_CURRENT_NETNS, 0);
} else {
if (tuple->ipv6.dport != bpf_htons(4321))
return TC_ACT_OK;
/* Upper parts of daddr are already zero. */
ln.ipv6.daddr[3] = bpf_htonl(0x1);
ln.ipv6.dport = bpf_htons(1234);
sk = bpf_sk_lookup_udp(skb, &ln, sizeof(ln.ipv6),
BPF_F_CURRENT_NETNS, 0);
}
/* workaround: We can't do a single socket lookup here, because then
* the compiler will likely spill tuple_len to the stack. This makes it
* lose all bounds information in the verifier, which then rejects the
* call as unsafe.
*/
if (!sk)
return TC_ACT_SHOT;
assign:
ret = bpf_sk_assign(skb, sk, 0);
bpf_sk_release(sk);
return ret;
}
static inline int
handle_tcp(struct __sk_buff *skb, struct bpf_sock_tuple *tuple, bool ipv4)
{
struct bpf_sock_tuple ln = {0};
struct bpf_sock *sk;
size_t tuple_len;
int ret;
tuple_len = ipv4 ? sizeof(tuple->ipv4) : sizeof(tuple->ipv6);
if ((void *)tuple + tuple_len > (void *)(long)skb->data_end)
return TC_ACT_SHOT;
sk = bpf_skc_lookup_tcp(skb, tuple, tuple_len, BPF_F_CURRENT_NETNS, 0);
if (sk) {
if (sk->state != BPF_TCP_LISTEN)
goto assign;
bpf_sk_release(sk);
}
if (ipv4) {
if (tuple->ipv4.dport != bpf_htons(4321))
return TC_ACT_OK;
ln.ipv4.daddr = bpf_htonl(0x7f000001);
ln.ipv4.dport = bpf_htons(1234);
sk = bpf_skc_lookup_tcp(skb, &ln, sizeof(ln.ipv4),
BPF_F_CURRENT_NETNS, 0);
} else {
if (tuple->ipv6.dport != bpf_htons(4321))
return TC_ACT_OK;
/* Upper parts of daddr are already zero. */
ln.ipv6.daddr[3] = bpf_htonl(0x1);
ln.ipv6.dport = bpf_htons(1234);
sk = bpf_skc_lookup_tcp(skb, &ln, sizeof(ln.ipv6),
BPF_F_CURRENT_NETNS, 0);
}
/* workaround: We can't do a single socket lookup here, because then
* the compiler will likely spill tuple_len to the stack. This makes it
* lose all bounds information in the verifier, which then rejects the
* call as unsafe.
*/
if (!sk)
return TC_ACT_SHOT;
if (sk->state != BPF_TCP_LISTEN) {
bpf_sk_release(sk);
return TC_ACT_SHOT;
}
assign:
ret = bpf_sk_assign(skb, sk, 0);
bpf_sk_release(sk);
return ret;
}
SEC("classifier/sk_assign_test")
int bpf_sk_assign_test(struct __sk_buff *skb)
{
struct bpf_sock_tuple *tuple, ln = {0};
bool ipv4 = false;
bool tcp = false;
int tuple_len;
int ret = 0;
tuple = get_tuple(skb, &ipv4, &tcp);
if (!tuple)
return TC_ACT_SHOT;
/* Note that the verifier socket return type for bpf_skc_lookup_tcp()
* differs from bpf_sk_lookup_udp(), so even though the C-level type is
* the same here, if we try to share the implementations they will
* fail to verify because we're crossing pointer types.
*/
if (tcp)
ret = handle_tcp(skb, tuple, ipv4);
else
ret = handle_udp(skb, tuple, ipv4);
return ret == 0 ? TC_ACT_OK : TC_ACT_SHOT;
}

Some files were not shown because too many files have changed in this diff Show More