Gabriel39 opened a new pull request, #69768: URL: https://github.com/apache/doris/pull/69768
### What problem does this PR solve? Warm external-table scans can stall on a congested file-cache disk even when object storage can serve the same immutable range faster. This backports [7af14bb9d998cb1188dec1ab93552330e6553e90](https://github.com/apache/doris/commit/7af14bb9d998cb1188dec1ab93552330e6553e90) to `branch-4.2`. ### Changes - Add a shared BE latency history with 50 samples per path, a 5 ms local threshold and minimum-latency feedback for the unselected path, following the StarCache 4.1.4 external-read policy. - Expose `enable_file_cache_io_adaptor` as a session variable, default `true`, and `file_cache_io_adaptor_skip_read_factor` as the sole BE tuning setting, default `1.0`. An omitted Thrift field keeps older-FE queries on their existing policy. - Adapt complete disk-cache hits before synchronous/asynchronous write-mode dispatch. Preserve cached blocks, avoid duplicate writeback and fall back to the complete cached range on remote errors or short reads. Keep mixed misses, inflight memory, background/index/warmup and internal-table policies. - Add 12 Profile counters covering lookup/local/remote costs, bypass requests and bytes, minimum feedback, remote failures and local fallback, with merge, delta and Profile-reporting coverage. - For this branch, allocate Thrift field **232** because 230/231 are already used, and reuse the existing physical remote-read byte counter for sampling. ### Validation On this `branch-4.2` backport: - 15 controller tests passed with ASan/UBSan and leak detection enabled. - All nine affected C++ translation units passed syntax checks against this branch's headers and generated sources using compatible build flags. - C++/Java Thrift generation, FE Checkstyle and clang-format 16 checks passed. - Two FE session-default, serialization, isolation and forwarding tests passed. - The full BE build and reader integration tests have not been rerun locally on this backport; `run buildall` is requested for this PR. Previously verified on the source commit: Release BE build, 46 focused BE regression tests, two opt-in cache-path benchmarks, and comparison against the actual StarCache policy object (43,018 matching decisions per factor, factors 1/2/3). The source benchmark used five alternating on/off rounds, 200 × 64 KiB reads per group, and calibration with 50 actual service-time samples on each path. It used file-backed reads and injected service delays: | Injected local / remote latency | Disabled median | Enabled median | Bypass requests | | --- | ---: | ---: | ---: | | 8 ms / 1 ms | 1711.451 ms | 236.368 ms | 200/200 | | 1 ms / 8 ms | 219.364 ms | 218.181 ms | 0/200 | Every returned byte matched and both scenarios performed zero cache writes. These results measure controlled cache routing; production-query and runtime-filter timing improvements require separate measurement. ### Release note External-table scans can adaptively bypass slower local file-cache reads. The session setting `enable_file_cache_io_adaptor` controls this behavior and defaults to `true`. ### Check List (For Author) - Test: - [x] Unit Test - [x] Manual test (source benchmark described above) - Behavior changed: - [x] Yes: eligible external scans may read remote storage despite a disk-cache hit. - Does this need documentation? - [x] Yes: document the session setting and Profile counters. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
