Valentin Schneider 68f4ff04db sched, smp: Trace smp callback causing an IPI ... Context ======= The newly-introduced ipi_send_cpumask tracepoint has a "callback" parameter which so far has only been fed with NULL. While CSD_TYPE_SYNC/ASYNC and CSD_TYPE_IRQ_WORK share a similar backing struct layout (meaning their callback func can be accessed without caring about the actual CSD type), CSD_TYPE_TTWU doesn't even have a function attached to its struct. This means we need to check the type of a CSD before eventually dereferencing its associated callback. This isn't as trivial as it sounds: the CSD type is stored in __call_single_node.u_flags, which get cleared right before the callback is executed via csd_unlock(). This implies checking the CSD type before it is enqueued on the call_single_queue, as the target CPU's queue can be flushed before we get to sending an IPI. Furthermore, send_call_function_single_ipi() only has a CPU parameter, and would need to have an additional argument to trickle down the invoked function. This is somewhat silly, as the extra argument will always be pushed down to the function even when nothing is being traced, which is unnecessary overhead. Changes ======= send_call_function_single_ipi() is only used by smp.c, and is defined in sched/core.c as it contains scheduler-specific ops (set_nr_if_polling() of a CPU's idle task). Split it into two parts: the scheduler bits remain in sched/core.c, and the actual IPI emission is moved into smp.c. This lets us define an __always_inline helper function that can take the related callback as parameter without creating useless register pressure in the non-traced path which only gains a (disabled) static branch. Do the same thing for the multi IPI case. Signed-off-by: Valentin Schneider <vschneid@redhat.com> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org> Link: https://lore.kernel.org/r/20230307143558.294354-8-vschneid@redhat.com		2023-03-24 11:01:29 +01:00
..
bpf	bpf-for-netdev	2023-03-06 20:28:00 -08:00
cgroup	Networking changes for 6.3.	2023-02-21 18:24:12 -08:00
configs	mm, slob: rename CONFIG_SLOB to CONFIG_SLOB_DEPRECATED	2022-12-01 00:09:20 +01:00
debug	kdb: use srcu console list iterator	2022-12-02 11:25:00 +01:00
dma	swiotlb: mark swiotlb_memblock_alloc() as __init	2023-02-22 06:44:48 -08:00
entry	entry: kmsan: introduce kmsan_unpoison_entry_regs()	2022-10-03 14:03:25 -07:00
events	perf: Fix check before add_event_to_groups() in perf_group_detach()	2023-03-15 21:49:47 +01:00
futex	- Prevent the leaking of a debug timer in futex_waitv()	2023-01-01 11:15:05 -08:00
gcov	gcov: add support for checksum field	2022-12-21 14:31:52 -08:00
irq	A set of updates for the interrupt susbsystem:	2023-03-05 11:19:16 -08:00
kcsan	kcsan: test: don't put the expect array on the stack	2023-01-02 08:59:33 -08:00
livepatch	Livepatching changes for 6.3	2023-02-23 14:00:10 -08:00
locking	RCU pull request for v6.3	2023-02-21 10:45:51 -08:00
module	modules-6.3-rc1	2023-02-23 14:05:08 -08:00
power	Merge branches 'powercap', 'pm-domains', 'pm-em' and 'pm-opp'	2023-02-15 20:06:26 +01:00
printk	printk changes for 6.3	2023-02-23 13:49:45 -08:00
rcu	Merge branch 'stall.2023.01.09a' into HEAD	2023-02-02 16:40:07 -08:00
sched	sched, smp: Trace smp callback causing an IPI	2023-03-24 11:01:29 +01:00
time	Updates for timekeeping, timers and clockevent/source drivers:	2023-02-21 09:45:13 -08:00
trace	Tracing fixes for 6.3:	2023-03-19 10:46:02 -07:00
.gitignore
acct.c	acct: fix potential integer overflow in encode_comp_t()	2022-11-30 16:13:18 -08:00
async.c
audit_fsnotify.c	audit: fix potential double free on error path from fsnotify_add_inode_mark	2022-08-22 18:50:06 -04:00
audit_tree.c	audit: use fsnotify group lock helpers	2022-04-25 14:37:28 +02:00
audit_watch.c	audit_init_parent(): constify path	2022-09-01 17:39:30 -04:00
audit.c	audit: use time_after to compare time	2022-08-29 19:47:03 -04:00
audit.h	audit: remove selinux_audit_rule_update() declaration	2022-09-07 11:30:15 -04:00
auditfilter.c
auditsc.c	capability: just use a 'u64' instead of a 'u32[2]' array	2023-03-01 10:01:22 -08:00
backtracetest.c
bounds.c	mm: multi-gen LRU: minimal implementation	2022-09-26 19:46:09 -07:00
capability.c	capability: just use a 'u64' instead of a 'u32[2]' array	2023-03-01 10:01:22 -08:00
cfi.c	cfi: Switch to -fsanitize=kcfi	2022-09-26 10:13:13 -07:00
compat.c	sched_getaffinity: don't assume 'cpumask_size()' is fully initialized	2023-03-14 19:32:38 -07:00
configs.c
context_tracking.c	context_tracking: Fix noinstr vs KASAN	2023-01-13 11:48:18 +01:00
cpu_pm.c	cpuidle, cpu_pm: Remove RCU fiddling from cpu_pm_{enter,exit}()	2023-01-13 11:48:15 +01:00
cpu.c	cpu/hotplug: Do not bail-out in DYING/STARTING sections	2022-12-02 12:43:02 +01:00
crash_core.c	mm: remove 'First tail page' members from struct page	2023-02-02 22:32:59 -08:00
crash_dump.c
cred.c	cred: Do not default to init_cred in prepare_kernel_cred()	2022-11-01 10:04:52 -07:00
delayacct.c	delayacct: support re-entrance detection of thrashing accounting	2022-09-26 19:46:07 -07:00
dma.c
exec_domain.c
exit.c	arm64 updates for 6.3:	2023-02-21 15:27:48 -08:00
extable.c	context_tracking: Take NMI eqs entrypoints over RCU	2022-07-05 13:32:59 -07:00
fail_function.c	kernel/fail_function: fix memory leak with using debugfs_lookup()	2023-02-08 13:36:22 +01:00
fork.c	fork: allow CLONE_NEWTIME in clone3 flags	2023-03-08 12:31:34 +01:00
freezer.c	freezer,sched: Rewrite core freezer logic	2022-09-07 21:53:50 +02:00
gen_kheaders.sh	kheaders: use standard naming for the temporary directory	2023-01-22 23:43:34 +09:00
groups.c	security: Add LSM hook to setgroups() syscall	2022-07-15 18:21:49 +00:00
hung_task.c	hung_task: print message when hung_task_warnings gets down to zero.	2023-02-09 17:03:20 -08:00
iomem.c
irq_work.c	irq_work: Trace self-IPIs sent via arch_irq_work_raise()	2023-03-24 11:01:27 +01:00
jump_label.c	jump_label: Prevent key->enabled int overflow	2022-12-01 15:53:05 -08:00
kallsyms_internal.h	kallsyms: Reduce the memory occupied by kallsyms_seqs_of_names[]	2022-11-12 18:47:36 -08:00
kallsyms_selftest.c	kallsyms: Fix scheduling with interrupts disabled in self-test	2023-01-13 15:09:08 -08:00
kallsyms_selftest.h	kallsyms: Add self-test facility	2022-11-15 00:42:02 -08:00
kallsyms.c	kallsyms: Add self-test facility	2022-11-15 00:42:02 -08:00
kcmp.c
Kconfig.freezer
Kconfig.hz
Kconfig.locks
Kconfig.preempt	Revert "signal, x86: Delay calling signals in atomic on RT enabled kernels"	2022-03-31 10:36:55 +02:00
kcov.c	mm: replace vma->vm_flags direct modifications with modifier calls	2023-02-09 16:51:39 -08:00
kexec_core.c	There is no particular theme here - mainly quick hits all over the tree.	2023-02-23 17:55:40 -08:00
kexec_elf.c
kexec_file.c	kexec: introduce sysctl parameters kexec_load_limit_*	2023-02-02 22:50:05 -08:00
kexec_internal.h	panic, kexec: make __crash_kexec() NMI safe	2022-09-11 21:55:06 -07:00
kexec.c	kexec: introduce sysctl parameters kexec_load_limit_*	2023-02-02 22:50:05 -08:00
kheaders.c
kmod.c
kprobes.c	x86/kprobes: Fix arch_check_optimized_kprobe check within optimized_kprobe range	2023-02-21 08:49:16 +09:00
ksysfs.c	kernels/ksysfs.c: export kernel address bits	2023-01-20 14:30:45 +01:00
kthread.c	kthread_worker: check all delayed works when destroy kthread worker	2023-02-02 22:50:02 -08:00
latencytop.c	latencytop: use the last element of latency_record of system	2022-09-11 21:55:12 -07:00
Makefile	kernel hardening fixes for v6.2-rc1	2022-12-23 12:00:24 -08:00
module_signature.c
notifier.c	kernel/notifier: Remove CONFIG_SRCU	2023-02-02 16:26:06 -08:00
nsproxy.c	fs/exec: switch timens when a task gets a new mm	2022-10-25 15:15:52 -07:00
padata.c	Kbuild updates for v6.2	2022-12-19 12:33:32 -06:00
panic.c	panic: fix the panic_print NMI backtrace setting	2023-03-02 21:54:23 -08:00
params.c	kernel/params.c: Use kstrtobool() instead of strtobool()	2023-01-25 14:07:21 -08:00
pid_namespace.c	- Daniel Verkamp has contributed a memfd series ("mm/memfd: add	2023-02-23 17:09:35 -08:00
pid_sysctl.h	mm/memfd: add MFD_NOEXEC_SEAL and MFD_EXEC	2023-01-18 17:12:37 -08:00
pid.c	gfs2: Add glockfd debugfs file	2022-06-29 13:07:16 +02:00
profile.c	kernel/profile.c: simplify duplicated code in profile_setup()	2022-09-11 21:55:12 -07:00
ptrace.c	rseq: Introduce extensible rseq ABI	2022-12-27 12:52:10 +01:00
range.c
reboot.c	kernel/reboot: Add SYS_OFF_MODE_RESTART_PREPARE mode	2022-10-04 15:59:36 +02:00
regset.c
relay.c	mm: replace vma->vm_flags direct modifications with modifier calls	2023-02-09 16:51:39 -08:00
resource_kunit.c
resource.c	dax/kmem: Fix leak of memory-hotplug resources	2023-02-17 14:58:01 -08:00
rseq.c	rseq: Extend struct rseq with per-memory-map concurrency ID	2022-12-27 12:52:12 +01:00
scftorture.c	scftorture: Fix distribution of short handler delays	2022-04-11 17:07:29 -07:00
scs.c	scs: add support for dynamic shadow call stacks	2022-11-09 18:06:35 +00:00
seccomp.c	seccomp: fix kernel-doc function name warning	2023-01-13 17:01:06 -08:00
signal.c	sched: Introduce per-memory-map concurrency ID	2022-12-27 12:52:11 +01:00
smp.c	sched, smp: Trace smp callback causing an IPI	2023-03-24 11:01:29 +01:00
smpboot.c	smpboot: use atomic_try_cmpxchg in cpu_wait_death and cpu_report_death	2022-09-11 21:55:10 -07:00
smpboot.h
softirq.c	context_tracking: Take IRQ eqs entrypoints over RCU	2022-07-05 13:32:59 -07:00
stackleak.c	stackleak: add on/off stack variants	2022-05-08 01:33:09 -07:00
stacktrace.c
static_call_inline.c	static_call: Add call depth tracking support	2022-10-17 16:41:16 +02:00
static_call.c	static_call: Don't make __static_call_return0 static	2022-04-05 09:59:38 +02:00
stop_machine.c	Scheduler changes in this cycle were:	2022-05-24 11:11:13 -07:00
sys_ni.c	kernel/sys_ni: add compat entry for fadvise64_64	2022-08-20 15:17:45 -07:00
sys.c	- Daniel Verkamp has contributed a memfd series ("mm/memfd: add	2023-02-23 17:09:35 -08:00
sysctl-test.c	kernel/sysctl-test: use SYSCTL_{ZERO/ONE_HUNDRED} instead of i_{zero/one_hundred}	2022-09-08 16:56:45 -07:00
sysctl.c	sysctl: fix proc_dobool() usability	2023-02-21 13:34:07 -08:00
task_work.c	task_work: use try_cmpxchg in task_work_add, task_work_cancel_match and task_work_run	2022-09-11 21:55:10 -07:00
taskstats.c	genetlink: start to validate reserved header bytes	2022-08-29 12:47:15 +01:00
torture.c	torture: Fix hang during kthread shutdown phase	2023-01-05 12:10:35 -08:00
tracepoint.c	tracepoint: Allow livepatch module add trace event	2023-02-18 14:34:36 -05:00
tsacct.c	taskstats: version 12 with thread group and exe info	2022-04-29 14:38:03 -07:00
ucount.c	ucounts: Split rlimit and ucount values and max values	2022-05-18 18:24:57 -05:00
uid16.c
uid16.h
umh.c	umh: simplify the capability pointer logic	2023-03-03 16:18:19 -08:00
up.c
user_namespace.c	userns: fix a struct's kernel-doc notation	2023-02-02 22:50:04 -08:00
user-return-notifier.c
user.c	kernel/user: Allow user_struct::locked_vm to be usable for iommufd	2022-11-30 20:16:49 -04:00
usermode_driver.c	blob_to_mnt(): kern_unmount() is needed to undo kern_mount()	2022-05-19 23:25:47 -04:00
utsname_sysctl.c	kernel/utsname_sysctl.c: Fix hostname polling	2022-10-23 12:01:01 -07:00
utsname.c
watch_queue.c	watch_queue: fix IOC_WATCH_QUEUE_SET_SIZE alloc error paths	2023-03-08 11:44:45 +01:00
watchdog_hld.c	Revert "printk: add functions to prefer direct printing"	2022-06-23 18:41:40 +02:00
watchdog.c	powerpc updates for 6.0	2022-08-06 16:38:17 -07:00
workqueue_internal.h
workqueue.c	workqueue: Fold rebind_worker() within rebind_workers()	2023-01-13 07:50:40 -10:00