deardeng opened a new pull request, #67315:
URL: https://github.com/apache/doris/pull/67315
run_background_monitor() ended its loop with
update_shadow_queue_element_count_metrics(), which takes _mutex_lru_log. The
LRU log replay thread holds that lock for as long as it takes to drain the log
queue, so a slow consumer froze the monitor with it:
check_disk_resource_limit(), check_need_evict_cache_in_advance() and every
gauge stopped running, and the disk resource limit mode stayed at whatever
value it happened to hold. Gauges were observed frozen for up to 40 minutes in
production.
The call was redundant from the start. #64798 added the shadow queue element
count gauge and published it in two places: inside replay_queue_event(), under
the same lock that mutates the shadow queue, and again from the monitor every
file_cache_background_monitor_interval_ms as a periodic refresh. Nothing
outside replay_queue_event() mutates a shadow queue, so that refresh could only
rewrite a value that had just been published and could not have changed since.
What it did add was a dependency from the disk protection loop onto a lock
owned by a background consumer.
Drop the call, and update_shadow_queue_element_count_metrics() with it: it
existed only for that refresh, and leaving a public method that takes
_mutex_lru_log invites the next background loop to reintroduce the coupling.
The gauge is still published by replay, now on the replay interval instead of
the monitor interval. Its test is replaced by one asserting that replay
publishes the gauge on its own.
How long replay holds the lock is a separate problem, addressed separately.
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
### Release note
None
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [x] No.
- [ ] Yes. <!-- Explain the behavior change -->
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
