This series adds private-stack support to the RV64 BPF JIT.
The verifier rejects a struct_ops program that requests a private stack
with -EACCES when the JIT does not report
bpf_jit_supports_private_stack(), and sched_ext requests one for its
dispatch callback, so no sched_ext scheduler can load on riscv64 today.
Allocation, the 16-byte guard regions at both ends, the guard value and
the lifetime handling follow arm64, x86-64 and powerpc64. What is
specific to riscv64 is where the pointers end up:
- Every BPF stack access goes through BPF_REG_FP (S5), so pointing S5
at the top of the per-CPU area and leaving SP on the kernel stack is
all the JIT has to do. S5 is callee-saved and each subprog derives
its own, so bpf2bpf calls need no extra save and restore.
- The outgoing stack-argument area stays in the kernel frame: callers
write it relative to SP while callees read it relative to RV_REG_FP.
Unlike x86-64, no register here is shared with the private-stack
frame pointer, so no verifier-side exclusion is needed and stack
arguments keep working together with a private stack. arm64 already
does the same, and patch 3 measures the three ranges at run time and
pins them in the disassembly.
- The per-CPU lookup is emitted after the tail-call entry point. A
program using a private stack can still be entered as a tail-call
target, at RV_TAILCALL_OFFSET, so it has to set up its own S5 there;
and since that offset is a constant added to the target's bpf_func,
everything in front of it must be the same size in every program.
The base address is materialized with absolute moves, as on arm64,
x86-64 and powerpc64. Those three reach the per-CPU offset through gs:,
TPIDR or r13 and only have to move the base into a register; riscv64 has
no such register, so the address of __per_cpu_offset is loaded here too.
A PC-relative pair is not an option for the base itself: the dynamic
per-CPU area is vmalloc'd, further from the JIT region than auipc
reaches, which is why the whole setup is ten instructions on the sv39
kernel tested here. __per_cpu_offset is a kernel-image symbol and could
use the emit_addr() form that insn_is_mov_percpu_addr() already uses, at
the same two instructions; unifying the two needs an extra_pass argument
and an error return in bpf_jit_build_prologue(), which also touches
bpf_jit_comp32.c, so it is left as is. Both constants are fixed before
the offset-convergence loop, so the prologue length is stable across
passes.
RV32 does not declare support and keeps the existing behaviour; the
prologue signature is unchanged, so bpf_jit_comp32.c needs no edit.
Testing
=======
QEMU virt rv64, an 8 vCPU SMP kernel and a 1 vCPU UP kernel, source tree
clean at the head of this series, selftests built from these patches, no
private-stack related failures. On UP, kfunc_call/subprog,
kfunc_call/subprog_lskel and scx_simple are not run: all three fail the
same way on the unpatched base.
- verifier_private_stack, struct_ops_private_stack,
private_stack_runtime
- tailcalls, stack_arg, subprogs, fentry_fexit, kfunc_call,
percpu_alloc, async_stack_depth, arena_atomics, arena_ldsx
- may_goto_priv_stack, may_goto_far, verifier_may_goto_1,
verifier_may_goto_2
- scx_simple on SMP
Chen Pei (3):
riscv: bpf: Add BPF private stack support
selftests/bpf: Enable private stack tests for riscv64
selftests/bpf: Add a runtime test for private stack and stack
arguments
arch/riscv/net/bpf_jit.h | 1 +
arch/riscv/net/bpf_jit_comp64.c | 42 +++++-
arch/riscv/net/bpf_jit_core.c | 84 +++++++++--
.../selftests/bpf/private_stack_runtime.h | 12 ++
.../bpf/prog_tests/private_stack_runtime.c | 83 +++++++++++
.../bpf/prog_tests/struct_ops_private_stack.c | 3 +-
.../bpf/progs/private_stack_runtime.c | 140 ++++++++++++++++++
.../bpf/progs/verifier_private_stack.c | 41 ++++-
8 files changed, 392 insertions(+), 14 deletions(-)
create mode 100644 tools/testing/selftests/bpf/private_stack_runtime.h
create mode 100644
tools/testing/selftests/bpf/prog_tests/private_stack_runtime.c
create mode 100644 tools/testing/selftests/bpf/progs/private_stack_runtime.c
--
2.50.1
