Hi All,
this patch set adds kernel and user space support to reveal
short lived bpf programs.
The kernel stores addr+bpf_prog_name information into perf ring buffer.
Later 'perf report' can properly attribute the cpu time to the program.
The following command was run before and after: 'perf record ./test_progs; perf
report'
Before patch set:
# Overhead Command Shared Object Symbol
# ........ .......... .................
.......................................
#
10.73% test_progs [kernel.kallsyms] [k] __htab_map_lookup_elem
8.16% test_progs [kernel.kallsyms] [k] bpf_skb_set_tunnel_key
7.90% test_progs [kernel.kallsyms] [k] memcmp
3.10% test_progs [kernel.kallsyms] [k] lookup_nulls_elem_raw
2.57% test_progs [kernel.kallsyms] [k] dst_release
2.45% test_progs [kernel.kallsyms] [k] do_check
1.89% test_progs [kernel.kallsyms] [k] bpf_xdp_adjust_head
1.52% test_progs [kernel.kallsyms] [k] 0x00007fffa002cc5a
1.22% test_progs [kernel.kallsyms] [k] check_helper_mem_access
0.99% test_progs [kernel.kallsyms] [k] 0x00007fffa001a6a8
0.99% test_progs [kernel.kallsyms] [k] kmem_cache_alloc_trace
0.97% test_progs [kernel.kallsyms] [k] 0x00007fffa001a652
0.95% test_progs [kernel.kallsyms] [k] 0x00007fffa0042d52
0.95% test_progs [kernel.kallsyms] [k] read_tsc
0.81% test_progs [kernel.kallsyms] [k] 0x00007fffa001a660
0.77% test_progs [kernel.kallsyms] [k] 0x00007fffa0042d69
0.74% test_progs [kernel.kallsyms] [k] percpu_array_map_lookup_elem
0.69% test_progs [kernel.kallsyms] [k] 0x00007fffa001a64e
After:
# Overhead Command Shared Object Symbol
# ........ .......... .................
.............................................
#
18.13% test_progs [kernel.kallsyms] [k]
bpf_prog_1accc788e7f04c38_balancer_ingres
10.73% test_progs [kernel.kallsyms] [k] __htab_map_lookup_elem
7.94% test_progs [kernel.kallsyms] [k] bpf_prog_20a05d8a586cf0e8_F
7.80% test_progs [kernel.kallsyms] [k] bpf_skb_set_tunnel_key
4.64% test_progs [kernel.kallsyms] [k] __sysfs_match_string
3.61% test_progs [kernel.kallsyms] [k] bpf_prog_9d89afa51f1dc0d7_F
3.51% test_progs [kernel.kallsyms] [k] bpf_prog_73b45270911a0294_F
3.41% test_progs [kernel.kallsyms] [k] bpf_prog_79be7b7bad8026bf_F
3.26% test_progs [kernel.kallsyms] [k] memcmp
3.10% test_progs [kernel.kallsyms] [k] lookup_nulls_elem_raw
2.89% test_progs [kernel.kallsyms] [k] bpf_prog_57bf3d413b9b7455_F
2.57% test_progs [kernel.kallsyms] [k] dst_discard
2.45% test_progs [kernel.kallsyms] [k] do_check
2.39% test_progs [kernel.kallsyms] [k] bpf_prog_576cbdaac1a4d2f6_F
1.82% test_progs [kernel.kallsyms] [k] bpf_prog_53ade2ecbddaa85b_F
1.70% test_progs [kernel.kallsyms] [k] bpf_xdp_adjust_head
1.32% test_progs [kernel.kallsyms] [k] bpf_prog_0edc54822404a598_F
Important considerations:
- Before and after the number of cpu samples are the same. No samples are lost.
But perf cannot find the symbol by IP, so a lot of small
0x00007fffa001a64e-like
symbols appear towards the end of 'perf report' and none of them look hot.
In reallity these IP addresses belong to few bpf programs that
were active at that time.
- newly loaded bpf program can be JITed into address space of unloaded prog.
7fffa001aXXX address at time X can belong to a program A, but similar
7fffa001aYYY address at time Y can belong to a different program B.
- event->mmap.pid == -1 is an existing indicator of kernel event.
event->mmap.tid == BPF_FS_MAGIC is an extension to indicate bpf related event.
Alternatively it's possible to introduce new 'enum perf_event_type' command
specificially for bpf prog load/unload, but existing RECORD_MMAP
is very close, so the choice made by this patchset is to extend it.
- steps to land the set:
Patches 1 and 2 need to go via bpf-next tree,
since we're adding support for better program names exactly in the same lines.
Patch 3 can go in parallel into perf tree. It has no effect without kernel
support and kernel support has not effect on old perf.
Peter, Arnaldo, Daniel, please review.
Alexei Starovoitov (3):
perf/core: introduce perf_event_mmap_bpf_prog
bpf: emit RECORD_MMAP events for bpf prog load/unload
tools/perf: recognize and process RECORD_MMAP events for bpf progs
include/linux/perf_event.h | 1 +
kernel/bpf/core.c | 22 +++++++++++++++++--
kernel/events/core.c | 44 +++++++++++++++++++++++++++++++++-----
tools/perf/util/machine.c | 27 +++++++++++++++++++++++
tools/perf/util/symbol.c | 13 +++++++++++
tools/perf/util/symbol.h | 1 +
6 files changed, 101 insertions(+), 7 deletions(-)
--
2.17.1
