Howdy,
After some dramatic debugging, I think I managed to isolate a problem
that looks like a funny eBPF runtime regression. It seems to be
introduced somewhere after 4.14.
The eBPF in question is running on network sockets with
SO_ATTACH_BPF. The BPF_PROG_TYPE_SOCKET_FILTER code:
uint64_t a = bpf_ktime_get_ns();
uint64_t b = bpf_ktime_get_ns();
uint64_t delta = b - a;
if ((int64_t)delta > 0) {
Depending on a context, the "delta" variable is set to
a wrong value. The compiled bytecode seems fine:
Disassembly of section socket1:
bpf_prog1:
; {
0: 85 00 00 00 05 00 00 00 call 5
; uint64_t a = bpf_ktime_get_ns();
1: bf 07 00 00 00 00 00 00 r7 = r0
; uint64_t b = bpf_ktime_get_ns();
2: 85 00 00 00 05 00 00 00 call 5
3: bf 06 00 00 00 00 00 00 r6 = r0
; uint64_t delta = b - a;
4: bf 68 00 00 00 00 00 00 r8 = r6
5: 1f 78 00 00 00 00 00 00 r8 -= r7
; if ((int64_t)delta > 0) {
6: b7 01 00 00 01 00 00 00 r1 = 1
7: 6d 81 0a 00 00 00 00 00 if r1 s> r8 goto +10 XXX
The code runs fine from root. From unprivileged user though, the
value of "delta" is a wrapped negative. Both "a" and "b" are fine in
both root and non-root cases. Technically bpf_ktime_get_ns() can
go backwards, but this isn't the case here. It does seem like the
problem is with the behaiviour of ... the subtraction operation when
running from SO_ATTACH_BPF executed by non-root?
Code:
https://gist.github.com/majek/d0bb75a8c62cc35bec2b342054084aab
git clone https://gist.github.com/majek/d0bb75a8c62cc35bec2b342054084aab
cd d0bb75a8c62cc35bec2b342054084aab
make
Then:
$ sudo ./ebpf-bug
0 -> 0 0x0000000000000000
1 -> 12585651690481 0x00000b72534c5bf1
2 -> 12585651690697 0x00000b72534c5cc9
3 -> 216 0x00000000000000d8
$ ./ebpf-bug
0 -> 1 0x0000000000000001
1 -> 12581437028489 0x00000b715815b889
2 -> 12581437028689 0x00000b715815b951
3 -> 18446731492272523127 0xfffff48ea7ea4777
"1" shows "a"
"2" shows "b"
"3" shows "detla"
As you see "delta" is off the scale for unprivileged user run. I
don't see any reason why root vs non-root should make any difference
for this code.
For completeness, this was tested with jit disabled:
$ cat /proc/version
Linux version 5.0.0-rc6+ (marek@mrbreeze) (gcc version 7.3.0)
$ sudo sysctl -a|grep -i jit
net.core.bpf_jit_enable = 0
net.core.bpf_jit_harden = 0
The same test on 4.14 seems fine:
$ cat /proc/version
Linux version 4.14.83-cloudflare-2018.11.4 (gcc version 8.2.0 (GCC))
$ sudo ./ebpf-bug
0 -> 0 0x0000000000000000
1 -> 7435203111991321 0x001a6a472052b819
2 -> 7435203111991429 0x001a6a472052b885
3 -> 108 0x000000000000006c
$ ./ebpf-bug
0 -> 0 0x0000000000000000
1 -> 7435205114618775 0x001a6a4797b06397
2 -> 7435205114618883 0x001a6a4797b06403
3 -> 108 0x000000000000006c
Cheers,
Marek
