matanper opened a new pull request, #69587:
URL: https://github.com/apache/doris/pull/69587
### What problem does this PR solve?
Issue Number: close #68892
Related PR: None
Problem Summary:
An eviction scan walks an LRU queue from the head while holding the cache
lock
(`find_evict_candidates`, and the loop in
`try_reserve_from_other_queue_by_time_interval`). It skips entries that are
not
releasable (held or downloading) and stops only when enough space is freed
or the
queue ends. Nothing bounds how many entries one scan examines, and the
skipped
entries stay at the head, so every later reservation walks over them again.
With a full cache and many held blocks (concurrent reads plus write-through
from
S3 file writers, one reservation per block), lock hold time grows with the
number
of held entries at the queue head, and every cache operation queues behind
it.
In production on 4.1.4 (cloud mode) we measured cache lock waits averaging
hundreds
of ms with p99 in seconds; S3 upload threads that copy into the cache backed
up on
the lock and Stream Loads stalled. With this change, on the same load, lock
waits
dropped to about 80 us on average.
The fix, applied to both scan loops:
- Entries a scan skips because they are in use are moved to the end of their
queue
after the scan (not during it, so the scan never reaches them again), and
the
move is recorded as MOVETOBACK in the LRU recorder like other touches.
Their
access time is not changed. `enable_file_cache_evict_rotate_in_use`
(mutable,
default true).
- One scan of one queue examines at most `file_cache_evict_max_scan_entries`
entries (mutable, default 4096; <= 0 means unbounded). A scan that reaches
the
bound after it already has enough space succeeds as before. If the
reservation
is still short, it fails, and `split_range_into_cells` turns that into a
SKIP_CACHE block, as it already does for any reservation failure.
Candidates
already selected are still evicted. One reservation scans at most a fixed
number
of queues, so its work under the lock is bounded by a small multiple of
the cap.
- bvars `file_cache_evict_scanned_entries`,
`file_cache_evict_rotated_in_use` and
`file_cache_evict_scan_truncated`.
Benchmark: `BM_FileCacheReserveWithHeldHead`
(`be/benchmark/benchmark_file_cache_evict_scan.hpp`,
run with `benchmark_test`) times one reservation into a full in-memory cache
whose LRU head holds
1K / 8K / 64K in-use blocks, in four modes: before (no bound, no rotation,
the pre-fix path),
bound only, rotation only, and after. Results will be added here.
### Release note
Add BE configs `file_cache_evict_max_scan_entries` (default 4096) and
`enable_file_cache_evict_rotate_in_use` (default true), which bound file
cache
eviction scans under the cache lock and move in-use entries out of the way of
later scans.
### Check List (For Author)
- Test: Unit Test added
(`be/test/io/cache/block_file_cache_evict_scan_test.cpp`:
rotation in the self-queue and time-interval scans, no rotation when
disabled,
and the scan bound turning the block into SKIP_CACHE), plus a Google
Benchmark.
The same change, minus the time-interval part, has run in production on a
4.1.4 fork.
- Behavior changed: Yes. An eviction scan that hits the bound without freeing
enough space fails the reservation (the block is not cached) instead of
walking
the rest of the queue, and in-use entries skipped by a scan move to the
end of
their queue.
- Does this need documentation: No
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
