liaoxin01 opened a new pull request, #69099:
URL: https://github.com/apache/doris/pull/69099
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
File cache LRU dump/restore (`file_cache_background_lru_dump_tail_record_num
> 0`) is meant to keep the LRU order across BE restarts. In practice the
restored order can be worse than having no order at all, and hot data is
evicted first after a restart:
1. **Restored blocks end up at the cold end.** `initialize_unlocked()`
restores the dump before `_storage->init()` starts the async cache load. Every
block that is not in the dump is then appended to the hot end of the LRU queue
by `load_cache_info_into_memory_from_db/fs` (`LRUQueue::add` inserts at
`queue.end()`). Eviction scans from `queue.begin()`, so the whole restored set,
which is supposed to be the hot part, is evicted before any block missing from
the dump. Blocks touched by queries during the async load are pushed toward the
cold end the same way.
2. **The dump is often incomplete, and the missing part is the hottest
part.** Restore runs before the LRU log replay thread is started, so the ADD
events generated by restore pile up in the LRU log queue, which is capped by
`file_cache_background_lru_log_queue_max_size` (500000) and drops events once
full. The dump is written cold-to-hot, so once a queue has more than 500k
entries the ADD events of the hottest entries are dropped. Later MOVETOBACK
events for those blocks are ignored because they are not in the shadow queue,
so they never come back. Restore also bumps the update counter past
`file_cache_background_lru_dump_update_cnt_threshold`, so the next periodic
dump overwrites the dump file without them, and after the next restart they
fall into case 1.
Besides that, the feature has a steady cost: every LRU change allocates and
enqueues a log record under the cache lock, the shadow queue keeps a second
copy of the LRU (up to 5M entries per queue per cache path), and the dump file
is rewritten every minute.
This PR disables the feature by default by changing the default of
`file_cache_background_lru_dump_tail_record_num` from `5000000` to `0`. With
`0`, LRU log recording, periodic dump and restore on startup are all skipped.
The feature can still be enabled explicitly. UTs and the `test_lru_persist`
docker suite that exercise LRU dump now enable it explicitly.
For clusters that turn it off, it is recommended to remove
`lru_dump_*.tail*` under each cache path, so a stale dump is not restored if
the feature is enabled again later.
### Release note
Disable file cache LRU dump/restore by default.
`file_cache_background_lru_dump_tail_record_num` now defaults to 0.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [ ] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [x] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [x] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [x] Yes. File cache LRU dump/restore is disabled by default. After a
restart, the LRU order follows the cache load order unless
`file_cache_background_lru_dump_tail_record_num` is set to a positive value.
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
