linux/tools/perf/Documentation
Yang Jihong 55c40e5052 perf kwork top: Introduce new top utility
Some common tools for collecting statistics on CPU usage, such as top,
obtain statistics from timer interrupt sampling, and then periodically
read statistics from /proc/stat.

This method has some deviations:

1. In the tick interrupt, the time between the last tick and the current
   tick is counted in the current task. However, the task may be running
   only part of the time.
2. For each task, the top tool periodically reads the /proc/{PID}/status
   information. For tasks with a short life cycle, it may be missed.

In conclusion, the top tool cannot accurately collect statistics on the
CPU usage and running time of tasks.

The statistical method based on sched_switch tracepoint can accurately
calculate the CPU usage of all tasks. This method is applicable to
scenarios where performance comparison data is of high precision.

Example usage:

  # perf kwork

   Usage: perf kwork [<options>] {record|report|latency|timehist|top}

      -D, --dump-raw-trace  dump raw trace in ASCII
      -f, --force           don't complain, do it
      -k, --kwork <kwork>   list of kwork to profile (irq, softirq, workqueue, sched, etc)
      -v, --verbose         be more verbose (show symbol address, etc)

  # perf kwork -k sched record -- perf bench sched messaging -g 1 -l 10000
  # Running 'sched/messaging' benchmark:
  # 20 sender and receiver processes per group
  # 1 groups == 40 processes run

       Total time: 14.074 [sec]
  [ perf record: Woken up 1 times to write data ]
  [ perf record: Captured and wrote 15.886 MB perf.data (129472 samples) ]
  # perf kwork top

  Total  : 115708.178 ms, 8 cpus
  %Cpu(s):   9.78% id
  %Cpu0   [|||||||||||||||||||||||||||     90.55%]
  %Cpu1   [|||||||||||||||||||||||||||     90.51%]
  %Cpu2   [||||||||||||||||||||||||||      88.57%]
  %Cpu3   [|||||||||||||||||||||||||||     91.18%]
  %Cpu4   [|||||||||||||||||||||||||||     91.09%]
  %Cpu5   [|||||||||||||||||||||||||||     90.88%]
  %Cpu6   [||||||||||||||||||||||||||      88.64%]
  %Cpu7   [|||||||||||||||||||||||||||     90.28%]

        PID    %CPU           RUNTIME  COMMMAND
    ----------------------------------------------------
       4113   22.23       3221.547 ms  sched-messaging
       4105   21.61       3131.495 ms  sched-messaging
       4119   21.53       3120.937 ms  sched-messaging
       4103   21.39       3101.614 ms  sched-messaging
       4106   21.37       3095.209 ms  sched-messaging
       4104   21.25       3077.269 ms  sched-messaging
       4115   21.21       3073.188 ms  sched-messaging
       4109   21.18       3069.022 ms  sched-messaging
       4111   20.78       3010.033 ms  sched-messaging
       4114   20.74       3007.073 ms  sched-messaging
       4108   20.73       3002.137 ms  sched-messaging
       4107   20.47       2967.292 ms  sched-messaging
       4117   20.39       2955.335 ms  sched-messaging
       4112   20.34       2947.080 ms  sched-messaging
       4118   20.32       2942.519 ms  sched-messaging
       4121   20.23       2929.865 ms  sched-messaging
       4110   20.22       2930.078 ms  sched-messaging
       4122   20.15       2919.542 ms  sched-messaging
       4120   19.77       2866.032 ms  sched-messaging
       4116   19.72       2857.660 ms  sched-messaging
       4127   16.19       2346.334 ms  sched-messaging
       4142   15.86       2297.600 ms  sched-messaging
       4141   15.62       2262.646 ms  sched-messaging
       4136   15.41       2231.408 ms  sched-messaging
       4130   15.38       2227.008 ms  sched-messaging
       4129   15.31       2217.692 ms  sched-messaging
       4126   15.21       2201.711 ms  sched-messaging
       4139   15.19       2200.722 ms  sched-messaging
       4137   15.10       2188.633 ms  sched-messaging
       4134   15.06       2182.082 ms  sched-messaging
       4132   15.02       2177.530 ms  sched-messaging
       4131   14.73       2131.973 ms  sched-messaging
       4125   14.68       2125.439 ms  sched-messaging
       4128   14.66       2122.255 ms  sched-messaging
       4123   14.65       2122.113 ms  sched-messaging
       4135   14.56       2107.144 ms  sched-messaging
       4133   14.51       2103.549 ms  sched-messaging
       4124   14.27       2066.671 ms  sched-messaging
       4140   14.17       2052.251 ms  sched-messaging
       4138   13.81       2000.361 ms  sched-messaging
          0   11.42       1652.009 ms  swapper/2
          0   11.35       1641.694 ms  swapper/6
          0    9.71       1405.108 ms  swapper/7
          0    9.48       1372.338 ms  swapper/1
          0    9.44       1366.013 ms  swapper/0
          0    9.11       1318.382 ms  swapper/5
          0    8.90       1287.582 ms  swapper/4
          0    8.81       1274.356 ms  swapper/3
       4100    2.61        379.328 ms  perf
       4101    1.16        169.487 ms  perf-exec
        151    0.65         94.741 ms  systemd-resolve
        249    0.36         53.030 ms  sd-resolve
        153    0.14         21.405 ms  systemd-timesyn
          1    0.10         16.200 ms  systemd
         16    0.09         15.785 ms  rcu_preempt
       4102    0.06          9.727 ms  perf
       4095    0.03          5.464 ms  kworker/7:1
         98    0.02          3.231 ms  jbd2/sda-8
        353    0.02          4.115 ms  sshd
         75    0.02          3.889 ms  kworker/2:1
         73    0.01          1.552 ms  kworker/5:1
         64    0.01          1.591 ms  kworker/4:1
         74    0.01          1.952 ms  kworker/3:1
         61    0.01          2.608 ms  kcompactd0
        397    0.01          1.602 ms  kworker/1:1
         69    0.01          1.817 ms  kworker/1:1H
         10    0.01          2.553 ms  kworker/u16:0
       2909    0.01          2.684 ms  kworker/0:2
       1211    0.00          0.426 ms  kworker/7:0
         97    0.00          0.153 ms  kworker/7:1H
         51    0.00          0.100 ms  ksoftirqd/7
        120    0.00          0.856 ms  systemd-journal
         76    0.00          1.414 ms  kworker/6:1
         46    0.00          0.246 ms  ksoftirqd/6
         45    0.00          0.164 ms  migration/6
         41    0.00          0.098 ms  ksoftirqd/5
         40    0.00          0.207 ms  migration/5
         86    0.00          1.339 ms  kworker/4:1H
         36    0.00          0.252 ms  ksoftirqd/4
         35    0.00          0.090 ms  migration/4
         31    0.00          0.156 ms  ksoftirqd/3
         30    0.00          0.073 ms  migration/3
         26    0.00          0.180 ms  ksoftirqd/2
         25    0.00          0.085 ms  migration/2
         21    0.00          0.106 ms  ksoftirqd/1
         20    0.00          0.118 ms  migration/1
        302    0.00          1.440 ms  systemd-logind
         17    0.00          0.132 ms  migration/0
         15    0.00          0.255 ms  ksoftirqd/0

Reviewed-by: Ian Rogers <irogers@google.com>
Signed-off-by: Yang Jihong <yangjihong1@huawei.com>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Kan Liang <kan.liang@linux.intel.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Ravi Bangoria <ravi.bangoria@amd.com>
Cc: Sandipan Das <sandipan.das@amd.com>
Link: https://lore.kernel.org/r/20230812084917.169338-10-yangjihong1@huawei.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
2023-09-12 17:31:59 -03:00
..
android.txt
arm-coresight.txt perf docs: Fix man page build wrt perf-arm-coresight.txt 2022-10-25 17:40:48 -03:00
asciidoc.conf perf docs: Allow man page date to be specified 2019-09-27 09:26:14 -03:00
asciidoctor-extensions.rb perf Documentation: Support for asciidoctor 2018-04-26 13:47:10 -03:00
build-docdep.perl perf doc: Fix doc.dep 2021-08-02 09:56:17 -03:00
build-xed.txt perf script: Add --insn-trace for instruction decoding 2018-10-24 15:29:50 -03:00
Build.txt perf tools: Add doc about how to build perf with Asan and UBSan 2019-03-19 16:52:04 -03:00
callchain-overhead-calculation.txt
cat-texi.perl perf doc: Fix perfman.info build 2021-08-02 09:56:17 -03:00
db-export.txt perf db-export: Add brief documentation 2019-06-05 09:47:57 -03:00
examples.txt perf tools: Replace lkml.org links with lore 2021-02-11 12:54:27 -03:00
guest-files.txt perf tools: Tidy guest option documentation 2022-08-11 18:50:17 -03:00
guestmount.txt perf tools: Tidy guest option documentation 2022-08-11 18:50:17 -03:00
intel-bts.txt
intel-hybrid.txt perf record: Fix manpage formatting of description of support to hybrid systems 2022-08-27 11:55:17 -03:00
intel-pt.txt perf intel-pt: Update intel-pt.txt file with new location of the documentation 2020-03-11 11:00:33 -03:00
itrace.txt perf intel-pt: Synthesize cycle events 2023-02-17 11:02:44 -03:00
jit-interface.txt
jitdump-specification.txt perf doc: Fix typos all over the place 2021-09-27 09:32:28 -03:00
Makefile perf doc: Add support for KBUILD_BUILD_TIMESTAMP 2023-05-15 17:49:01 -03:00
manpage-1.72.xsl
manpage-base.xsl
manpage-bold-literal.xsl
manpage-normal.xsl
manpage-suppress-sp.xsl
perf-annotate.txt perf symbol: Add command line support for addr2line path 2023-04-04 09:39:56 -03:00
perf-archive.txt
perf-arm-spe.txt perf docs: Add man page entry for Arm SPE 2022-04-14 08:54:03 -03:00
perf-bench.txt perf bench uprobe: Add benchmark to test uprobe overhead 2023-07-20 11:31:19 -03:00
perf-buildid-cache.txt perf record: Disable debuginfod by default 2022-01-15 17:41:25 -03:00
perf-buildid-list.txt perf buildid-list: Add a "-m" option to show kernel and modules build-ids 2022-07-18 16:35:34 -03:00
perf-c2c.txt perf c2c: Add report option to show false sharing in adjacent cachelines 2023-02-16 09:33:45 -03:00
perf-config.txt perf parse-events: Remove BPF event support 2023-08-15 16:41:48 -03:00
perf-daemon.txt perf daemon: Add examples to man page 2021-02-11 10:19:52 -03:00
perf-data.txt perf data: Add JSON export 2021-04-29 10:30:58 -03:00
perf-diff.txt perf tools: Make quiet mode consistent between tools 2022-10-27 16:37:26 -03:00
perf-dlfilter.txt perf dlfilter: Add al_cleanup() 2023-08-15 16:41:49 -03:00
perf-evlist.txt perf tools: Fix documentation of verbose options 2021-03-06 16:54:26 -03:00
perf-ftrace.txt perf docs: Fix format of unordered lists 2023-08-16 08:37:49 -03:00
perf-help.txt
perf-inject.txt perf inject: Clarify build-id options a little bit 2022-10-04 08:55:22 -03:00
perf-intel-pt.txt perf intel-pt: Synthesize cycle events 2023-02-17 11:02:44 -03:00
perf-iostat.txt perf docs: Fix accidental em-dashes 2021-08-10 11:05:21 -03:00
perf-kallsyms.txt perf tools: Fix documentation of verbose options 2021-03-06 16:54:26 -03:00
perf-kmem.txt perf kmem: Improve man page for record options 2021-10-08 16:10:02 -03:00
perf-kvm.txt perf kvm: Update documentation to reflect new changes 2023-03-15 16:53:43 -03:00
perf-kwork.txt perf kwork top: Introduce new top utility 2023-09-12 17:31:59 -03:00
perf-list.txt perf tools docs: Use canonical ftrace path 2023-02-02 16:32:19 -03:00
perf-lock.txt perf lock contention: Add --output option 2023-07-01 10:48:48 -07:00
perf-mem.txt perf mem/c2c: Document that SPE is used for mem and c2c on ARM 2023-01-27 15:00:34 -03:00
perf-probe.txt perf docs: Fix a typo in 'perf probe' man page: l20th -> 120th 2023-01-19 09:49:59 -03:00
perf-record.txt perf record: Track sideband events for all CPUs when tracing selected CPUs 2023-09-12 17:31:43 -03:00
perf-report.txt perf symbol: Add command line support for addr2line path 2023-04-04 09:39:56 -03:00
perf-sched.txt perf sched timehist: Add support for filtering on CPU 2020-01-06 11:46:09 -03:00
perf-script-perl.txt perf tools docs: Use canonical ftrace path 2023-02-02 16:32:19 -03:00
perf-script-python.txt perf tools docs: Use canonical ftrace path 2023-02-02 16:32:19 -03:00
perf-script.txt perf script: Add new output field 'dsoff' to print dso offset 2023-05-12 15:21:49 -03:00
perf-stat.txt perf stat: Document --metric-no-threshold and threshold colors 2023-06-05 16:04:14 -03:00
perf-test.txt perf test: Add Symbols test 2023-01-22 18:09:56 -03:00
perf-timechart.txt perf timechart: Document missing --force option 2017-11-16 14:50:06 -03:00
perf-top.txt perf top: Add --branch-history option 2023-04-04 09:39:56 -03:00
perf-trace.txt perf tools: Fix documentation of verbose options 2021-03-06 16:54:26 -03:00
perf-version.txt perf version: Add man page 2018-04-02 13:52:23 -03:00
perf.data-directory-format.txt perf record: Put a copy of kcore into the perf.data directory 2019-11-06 15:43:05 -03:00
perf.data-file-format.txt perf doc: Fix typo in perf.data-file-format.txt 2023-07-28 19:01:16 -03:00
perf.txt perf docs: Add man page entry for Arm SPE 2022-04-14 08:54:03 -03:00
perfconfig.example
security.txt perf docs: Introduce security.txt file to document related issues 2020-05-28 10:03:26 -03:00
tips.txt perf tools: Fix typos / broken sentences 2019-07-02 16:08:16 -03:00
topdown.txt perf doc: Refresh topdown documentation 2023-02-19 08:10:15 -03:00