0AyanamiRei opened a new pull request, #65420:
URL: https://github.com/apache/doris/pull/65420
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary:
The existing S3 GET/PUT token bucket rate limiter uses fixed per-BE QPS
settings. This is not suitable for heterogeneous deployments because a BE with
many CPU cores and a BE with few CPU cores receive the same S3 request budget.
This PR adds CPU-aware S3 rate limiter settings for GET and PUT requests.
When the new per-core settings are left as 0, Doris keeps the existing absolute
limiter behavior. When a per-core setting is positive, Doris derives the
effective per-BE S3 QPS from BE CPU cores and caps it with
s3_token_per_second_max when configured.
The PR also centralizes effective limiter config calculation, reuses it for
initialization and dynamic rebuild, and guards CPU-derived QPS calculation from
overflow before applying the max cap.
### Release note
Add BE configs to support CPU-aware S3 GET/PUT request rate limiting.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [x] Unit Test
- ./run-be-ut.sh --run
--filter=S3UTILTest.effective_s3_rate_limiter_config_caps_overflowing_per_core_get_config
- ./run-be-ut.sh --run --filter='S3UTILTest.*'
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [x] Yes. New per-core S3 limiter configs allow Doris to derive GET/PUT
QPS from BE CPU cores when enabled; existing configs remain effective when
per-core configs are 0.
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
