Em Fri, Nov 02, 2018 at 03:42:49PM +0000, Edward Cree escreveu:
> On 02/11/18 15:02, Arnaldo Carvalho de Melo wrote:
> > Yeah, didn't work as well:
>
> > And the -vv in 'perf trace' didn't seem to map to further details in the
> > output of the verifier debug:
> Yeah for log_level 2 you probably need to make source-level changes to either
> perf or libbpf (I think the latter). It's annoying that essentially no tools
> plumb through an option for that, someone should fix them ;-)
>
> > libbpf: -- BEGIN DUMP LOG ---
> > libbpf:
> > 0: (bf) r6 = r1
> > 1: (bf) r1 = r10
> > 2: (07) r1 += -328
> > 3: (b7) r7 = 64
> > 4: (b7) r2 = 64
> > 5: (bf) r3 = r6
> > 6: (85) call bpf_probe_read#4
> > 7: (79) r1 = *(u64 *)(r10 -320)
> > 8: (15) if r1 == 0x101 goto pc+4
> > R0=inv(id=0) R1=inv(id=0) R6=ctx(id=0,off=0,imm=0) R7=inv64 R10=fp0,call_-1
> > 9: (55) if r1 != 0x2 goto pc+22
> > R0=inv(id=0) R1=inv2 R6=ctx(id=0,off=0,imm=0) R7=inv64 R10=fp0,call_-1
> > 10: (bf) r1 = r6
> > 11: (07) r1 += 16
> > 12: (05) goto pc+2
> > 15: (79) r3 = *(u64 *)(r1 +0)
> > dereference of modified ctx ptr R1 off=16 disallowed
> Aha, we at least got a different error message this time.
> And indeed llvm has done that optimisation, rather than the more obvious
> 11: r3 = *(u64 *)(r1 +16)
> because it wants to have lots of reads share a single insn. You may be able
> to defeat that optimisation by adding compiler barriers, idk. Maybe someone
> with llvm knowledge can figure out how to stop it (ideally, llvm would know
> when it's generating for bpf backend and not do that). -O0? ¯\_(ツ)_/¯
set env: NR_CPUS=4
set env: LINUX_VERSION_CODE=0x41300
set env: CLANG_EXEC=/usr/local/bin/clang
unset env: CLANG_OPTIONS
set env: KERNEL_INC_OPTIONS= -nostdinc -isystem
/usr/lib/gcc/x86_64-redhat-linux/7/include
-I/home/acme/git/linux/arch/x86/include -I./arch/x86/include/generated
-I/home/acme/git/linux/include -I./include
-I/home/acme/git/linux/arch/x86/include/uapi
-I./arch/x86/include/generated/uapi -I/home/acme/git/linux/include/uapi
-I./include/generated/uapi -include
/home/acme/git/linux/include/linux/kconfig.h
set env: PERF_BPF_INC_OPTIONS=-I/home/acme/lib/perf/include/bpf
set env: WORKING_DIR=/lib/modules/4.19.0-rc8-00014-gc0cff31be705/build
set env:
CLANG_SOURCE=/home/acme/git/perf/tools/perf/examples/bpf/augmented_raw_syscalls.c
llvm compiling command template: $CLANG_EXEC -D__KERNEL__
-D__NR_CPUS__=$NR_CPUS -DLINUX_VERSION_CODE=$LINUX_VERSION_CODE $CLANG_OPTIONS
$PERF_BPF_INC_OPTIONS $KERNEL_INC_OPTIONS -Wno-unused-value -Wno-pointer-sign
-working-directory $WORKING_DIR -c "$CLANG_SOURCE" -target bpf $CLANG_EMIT_LLVM
-O2 -o - $LLVM_OPTIONS_PIPE
llvm compiling command : /usr/local/bin/clang -D__KERNEL__ -D__NR_CPUS__=4
-DLINUX_VERSION_CODE=0x41300 -I/home/acme/lib/perf/include/bpf -nostdinc
-isystem /usr/lib/gcc/x86_64-redhat-linux/7/include
-I/home/acme/git/linux/arch/x86/include -I./arch/x86/include/generated
-I/home/acme/git/linux/include -I./include
-I/home/acme/git/linux/arch/x86/include/uapi
-I./arch/x86/include/generated/uapi -I/home/acme/git/linux/include/uapi
-I./include/generated/uapi -include
/home/acme/git/linux/include/linux/kconfig.h -Wno-unused-value
-Wno-pointer-sign -working-directory
/lib/modules/4.19.0-rc8-00014-gc0cff31be705/build -c
/home/acme/git/perf/tools/perf/examples/bpf/augmented_raw_syscalls.c -target
bpf -O2 -o -
So it is using -O2, lets try with -O0...
So I added this to my ~/.perfconfig, i.e. the default clang command line
template replacing -O2 with -O0.
[root@seventh perf]# cat ~/.perfconfig
[llvm]
clang-bpf-cmd-template = "$CLANG_EXEC -D__KERNEL__
-D__NR_CPUS__=$NR_CPUS -DLINUX_VERSION_CODE=$LINUX_VERSION_CODE $CLANG_OPTIONS
$PERF_BPF_INC_OPTIONS $KERNEL_INC_OPTIONS -Wno-unused-value -Wno-pointer-sign
-working-directory $WORKING_DIR -c \"$CLANG_SOURCE\" -target bpf
$CLANG_EMIT_LLVM -O0 -o - $LLVM_OPTIONS_PIPE"
# dump-obj = true
[root@seventh perf]#
And got an explosion:
# trace -vv -e tools/perf/examples/bpf/augmented_raw_syscalls.c sleep 1
bpf: builtin compilation failed: -95, try external compiler
Kernel build dir is set to /lib/modules/4.19.0-rc8-00014-gc0cff31be705/build
set env: KBUILD_DIR=/lib/modules/4.19.0-rc8-00014-gc0cff31be705/build
unset env: KBUILD_OPTS
include option is set to -nostdinc -isystem
/usr/lib/gcc/x86_64-redhat-linux/7/include
-I/home/acme/git/linux/arch/x86/include -I./arch/x86/include/generated
-I/home/acme/git/linux/include -I./include
-I/home/acme/git/linux/arch/x86/include/uapi
-I./arch/x86/include/generated/uapi -I/home/acme/git/linux/include/uapi
-I./include/generated/uapi -include
/home/acme/git/linux/include/linux/kconfig.h
set env: NR_CPUS=4
set env: LINUX_VERSION_CODE=0x41300
set env: CLANG_EXEC=/usr/local/bin/clang
unset env: CLANG_OPTIONS
set env: KERNEL_INC_OPTIONS= -nostdinc -isystem
/usr/lib/gcc/x86_64-redhat-linux/7/include
-I/home/acme/git/linux/arch/x86/include -I./arch/x86/include/generated
-I/home/acme/git/linux/include -I./include
-I/home/acme/git/linux/arch/x86/include/uapi
-I./arch/x86/include/generated/uapi -I/home/acme/git/linux/include/uapi
-I./include/generated/uapi -include
/home/acme/git/linux/include/linux/kconfig.h
set env: PERF_BPF_INC_OPTIONS=-I/home/acme/lib/perf/include/bpf
set env: WORKING_DIR=/lib/modules/4.19.0-rc8-00014-gc0cff31be705/build
set env:
CLANG_SOURCE=/home/acme/git/perf/tools/perf/examples/bpf/augmented_raw_syscalls.c
llvm compiling command template: $CLANG_EXEC -D__KERNEL__
-D__NR_CPUS__=$NR_CPUS -DLINUX_VERSION_CODE=$LINUX_VERSION_CODE $CLANG_OPTIONS
$PERF_BPF_INC_OPTIONS $KERNEL_INC_OPTIONS -Wno-unused-value -Wno-pointer-sign
-working-directory $WORKING_DIR -c "$CLANG_SOURCE" -target bpf $CLANG_EMIT_LLVM
-O0 -o - $LLVM_OPTIONS_PIPE
llvm compiling command : /usr/local/bin/clang -D__KERNEL__ -D__NR_CPUS__=4
-DLINUX_VERSION_CODE=0x41300 -I/home/acme/lib/perf/include/bpf -nostdinc
-isystem /usr/lib/gcc/x86_64-redhat-linux/7/include
-I/home/acme/git/linux/arch/x86/include -I./arch/x86/include/generated
-I/home/acme/git/linux/include -I./include
-I/home/acme/git/linux/arch/x86/include/uapi
-I./arch/x86/include/generated/uapi -I/home/acme/git/linux/include/uapi
-I./include/generated/uapi -include
/home/acme/git/linux/include/linux/kconfig.h -Wno-unused-value
-Wno-pointer-sign -working-directory
/lib/modules/4.19.0-rc8-00014-gc0cff31be705/build -c
/home/acme/git/perf/tools/perf/examples/bpf/augmented_raw_syscalls.c -target
bpf -O0 -o -
clang-7: /home/acme/git/llvm/lib/Target/BPF/MCTargetDesc/BPFAsmBackend.cpp:74:
virtual void {anonymous}::BPFAsmBackend::applyFixup(const llvm::MCAssembler&,
const llvm::MCFixup&, const llvm::MCValue&, llvm::MutableArrayRef<char>,
uint64_t, bool, const llvm::MCSubtargetInfo*) const: Assertion `Value == 0'
failed.
Stack dump:
0. Program arguments: /usr/local/bin/clang-7 -cc1 -triple bpf -emit-obj
-mrelax-all -disable-free -main-file-name augmented_raw_syscalls.c
-mrelocation-model static -mthread-model posix -mdisable-fp-elim -fmath-errno
-masm-verbose -mconstructor-aliases -dwarf-column-info -debugger-tuning=gdb
-coverage-notes-file /home/acme/git/perf/-.gcno -nostdsysteminc -nobuiltininc
-resource-dir /usr/local/lib/clang/8.0.0 -working-directory
/lib/modules/4.19.0-rc8-00014-gc0cff31be705/build -isystem
/usr/lib/gcc/x86_64-redhat-linux/7/include -include
/home/acme/git/linux/include/linux/kconfig.h -D __KERNEL__ -D __NR_CPUS__=4 -D
LINUX_VERSION_CODE=0x41300 -I /home/acme/lib/perf/include/bpf -I
/home/acme/git/linux/arch/x86/include -I ./arch/x86/include/generated -I
/home/acme/git/linux/include -I ./include -I
/home/acme/git/linux/arch/x86/include/uapi -I ./arch/x86/include/generated/uapi
-I /home/acme/git/linux/include/uapi -I ./include/generated/uapi -O0
-Wno-unused-value -Wno-pointer-sign -fdebug-compilation-dir /home/acme/git/perf
-ferror-limit 19 -fmessage-length 203 -fobjc-runtime=gcc
-fdiagnostics-show-option -fcolor-diagnostics -o - -x c
/home/acme/git/perf/tools/perf/examples/bpf/augmented_raw_syscalls.c -faddrsig
1. <eof> parser at end of file
2. Code generation
#0 0x000000000408ff98 llvm::sys::PrintStackTrace(llvm::raw_ostream&)
(/usr/local/bin/clang-7+0x408ff98)
#1 0x000000000409002b (/usr/local/bin/clang-7+0x409002b)
#2 0x000000000408e059 llvm::sys::RunSignalHandlers()
(/usr/local/bin/clang-7+0x408e059)
#3 0x000000000408fa5b (/usr/local/bin/clang-7+0x408fa5b)
#4 0x00007fcd33ab81c0 __restore_rt (/lib64/libpthread.so.0+0x121c0)
#5 0x00007fcd32817750 __GI_raise (/lib64/libc.so.6+0x34750)
#6 0x00007fcd32818d31 __GI_abort (/lib64/libc.so.6+0x35d31)
#7 0x00007fcd3281005a __assert_fail_base (/lib64/libc.so.6+0x2d05a)
#8 0x00007fcd328100d2 (/lib64/libc.so.6+0x2d0d2)
#9 0x0000000002602d13 (/usr/local/bin/clang-7+0x2602d13)
#10 0x0000000003bf6da7 llvm::MCAssembler::layout(llvm::MCAsmLayout&)
(/usr/local/bin/clang-7+0x3bf6da7)
#11 0x0000000003bf6e2c llvm::MCAssembler::Finish()
(/usr/local/bin/clang-7+0x3bf6e2c)
#12 0x0000000003c524e8 llvm::MCObjectStreamer::FinishImpl()
(/usr/local/bin/clang-7+0x3c524e8)
#13 0x0000000003c2db4b llvm::MCELFStreamer::FinishImpl()
(/usr/local/bin/clang-7+0x3c2db4b)
#14 0x0000000003c5ba43 llvm::MCStreamer::Finish()
(/usr/local/bin/clang-7+0x3c5ba43)
#15 0x0000000004c281b9 llvm::AsmPrinter::doFinalization(llvm::Module&)
(/usr/local/bin/clang-7+0x4c281b9)
#16 0x00000000038d637b llvm::FPPassManager::doFinalization(llvm::Module&)
(/usr/local/bin/clang-7+0x38d637b)
#17 0x00000000038d683d (/usr/local/bin/clang-7+0x38d683d)
#18 0x00000000038d6d6b llvm::legacy::PassManagerImpl::run(llvm::Module&)
(/usr/local/bin/clang-7+0x38d6d6b)
#19 0x00000000038d6f9b llvm::legacy::PassManager::run(llvm::Module&)
(/usr/local/bin/clang-7+0x38d6f9b)
#20 0x000000000436b091 (/usr/local/bin/clang-7+0x436b091)
#21 0x000000000436df07 clang::EmitBackendOutput(clang::DiagnosticsEngine&,
clang::HeaderSearchOptions const&, clang::CodeGenOptions const&,
clang::TargetOptions const&, clang::LangOptions const&, llvm::DataLayout
const&, llvm::Module*, clang::BackendAction,
std::unique_ptr<llvm::raw_pwrite_stream,
std::default_delete<llvm::raw_pwrite_stream> >)
(/usr/local/bin/clang-7+0x436df07)
#22 0x0000000004f6ee11 (/usr/local/bin/clang-7+0x4f6ee11)
#23 0x00000000062af3a3 clang::ParseAST(clang::Sema&, bool, bool)
(/usr/local/bin/clang-7+0x62af3a3)
#24 0x0000000004a73873 clang::ASTFrontendAction::ExecuteAction()
(/usr/local/bin/clang-7+0x4a73873)
#25 0x0000000004f6ced8 clang::CodeGenAction::ExecuteAction()
(/usr/local/bin/clang-7+0x4f6ced8)
#26 0x0000000004a7329d clang::FrontendAction::Execute()
(/usr/local/bin/clang-7+0x4a7329d)
#27 0x0000000004a04ccf
clang::CompilerInstance::ExecuteAction(clang::FrontendAction&)
(/usr/local/bin/clang-7+0x4a04ccf)
#28 0x0000000004bc512e
clang::ExecuteCompilerInvocation(clang::CompilerInstance*)
(/usr/local/bin/clang-7+0x4bc512e)
#29 0x0000000001d88a6c cc1_main(llvm::ArrayRef<char const*>, char const*,
void*) (/usr/local/bin/clang-7+0x1d88a6c)
#30 0x0000000001d7e31e (/usr/local/bin/clang-7+0x1d7e31e)
#31 0x0000000001d7ef94 main (/usr/local/bin/clang-7+0x1d7ef94)
#32 0x00007fcd32803fea __libc_start_main (/lib64/libc.so.6+0x20fea)
#33 0x0000000001d7beea _start (/usr/local/bin/clang-7+0x1d7beea)
clang-7: error: unable to execute command: Aborted (core dumped)
clang-7: error: clang frontend command failed due to signal (use -v to see
invocation)
clang version 8.0.0 (http://llvm.org/git/clang.git
8587270a739ee30c926a76d5657e65e85b560f6e) (http://llvm.org/git/llvm.git
0566eefef9c3777bd780ec4cbb9efa764633b76c)
Target: bpf
Thread model: posix
InstalledDir: /usr/local/bin
clang-7: note: diagnostic msg: PLEASE submit a bug report to
https://bugs.llvm.org/ and include the crash backtrace, preprocessed source,
and associated run script.
clang-7: note: diagnostic msg:
********************
PLEASE ATTACH THE FOLLOWING FILES TO THE BUG REPORT:
Preprocessed source(s) and associated run script(s) are located at:
clang-7: note: diagnostic msg: /tmp/augmented_raw_syscalls-7444d9.c
clang-7: note: diagnostic msg: /tmp/augmented_raw_syscalls-7444d9.sh
clang-7: note: diagnostic msg:
********************
ERROR: unable to compile tools/perf/examples/bpf/augmented_raw_syscalls.c
Hint: Check error message shown above.
Hint: You can also pre-compile it into .o using:
clang -target bpf -O2 -c
tools/perf/examples/bpf/augmented_raw_syscalls.c
with proper -I and -D options.
event syntax error: 'tools/perf/examples/bpf/augmented_raw_syscalls.c'
\___ Failed to load
tools/perf/examples/bpf/augmented_raw_syscalls.c from source: Error when
compiling BPF scriptlet
(add -v to see detail)
Run 'perf list' for a list of valid events
Usage: perf trace [<options>] [<command>]
or: perf trace [<options>] -- <command> [<options>]
or: perf trace record [<options>] [<command>]
or: perf trace record [<options>] -- <command> [<options>]
-e, --event <event> event/syscall selector. use 'perf list' to list
available events
[root@seventh perf]#
Trying with -O1...
> Alternatively, your prog looks short enough that maybe you could kick the C
> habit and write it directly in eBPF asm, that way no-one is optimising things
> behind your back. (I realise this option won't appeal to everyone ;-)
Nah, if that is what it takes... Would be better with simple looking C
code tho :-)
> The reason the verifier disallows this, iirc, is because it needs to be able
> to rewrite the offsets on ctx accesses (see convert_ctx_accesses()) in case
> underlying kernel struct doesn't match the layout of the ctx ABI. To do this
> it needs the ctx offset to live entirely in the insn doing the access,
> otherwise different paths could lead to the same insn accessing different ctx
> offsets with different fixups needed — can't be done.
>
> -Ed
