Gabriel39 opened a new pull request, #69779: URL: https://github.com/apache/doris/pull/69779
### What problem does this PR solve? Warm external-table scans can stall on a congested file-cache disk even when object storage serves the same immutable range faster. This applies the single-commit feature from #69768 to `master`. ### Implementation - Share 50 latency samples per path across the BE. Keep local reads below a 5 ms average; otherwise compare the local average multiplied by the skip factor with the remote average. Feed the remembered minimum into the unselected path. Only actual observations age that minimum. - Forward session variable `enable_file_cache_io_adaptor` (default `true`). Keep `file_cache_io_adaptor_skip_read_factor=1.0` as the sole BE tuning setting. Optional Thrift field **1007** has the same meaning in both proposed branches and was checked against all active releases and master; an omitted field preserves old-FE behavior. - Apply to complete downloaded disk-cache hits before synchronous/asynchronous write-mode dispatch. Preserve cached blocks and avoid duplicate writeback. Keep mixed misses, inflight memory, background/index/warmup and internal-table behavior. - After a remote error or short read, overwrite the complete range from local cache and disable optional remote reads for that reader's lifetime. Required misses still read remote storage; other readers retain independent eligibility. Cancellation propagates without disabling future adaptation. - Preserve asynchronous LRU recency on the full-hit path: collect eligible touches under the cache mutex and enqueue after unlocking. The capacity test warms A/B, repeatedly reads A through both write modes, inserts C and verifies that cold B is evicted. ### Policy boundary The controller does not force IO solely to collect fresh samples. Remembered-minimum feedback cannot discover a disk recovery that has never been observed. The new no-fast-history test documents this limitation alongside the existing fast-minimum recovery test. No additional switches, probe intervals or global failure backoff are introduced. ### Observability All 12 counters participate in statistics merge, delta reporting and Profile aggregation: | Profile counters | Meaning | | --- | --- | | `IOAdaptorLookupCount`, `IOAdaptorLookupTime` | Full-coverage lookup attempts and cost | | `IOAdaptorLocalReadCount`, `IOAdaptorLocalReadTime` | Local attempts and service time, including fallback | | `IOAdaptorBypassCount`, `IOAdaptorBypassBytes` | Requested remote bypasses and bytes | | `IOAdaptorRemoteReadBytes`, `IOAdaptorRemoteReadTime` | Returned bypass bytes and remote cost | | `IOAdaptorLocalFeedbackCount`, `IOAdaptorRemoteFeedbackCount` | Synthetic minimum feedback; these are not IO operations | | `IOAdaptorRemoteFailureCount`, `IOAdaptorFallbackLocalCount` | Failed/short bypasses and local fallbacks | Requested and returned remote bytes are tracked separately so short reads remain measurable. ### Master adaptation and validation - Keep master's cache-loading API, existing query-option IDs and Profile fields. Register the session variable with the current `VarAttrDef` API. - Extract local-cache reads and remote sampling into focused helpers while retaining routing, accounting, cancellation and fallback order. - **51 focused BE tests passed under ASan** using `run-be-ut.sh`, including both write modes, real capacity eviction, failure/short-read suppression, reader isolation, required misses, cancellation, cache repair, scanner propagation and all Profile accounting. - **Four FE tests passed** using `run-fe-ut.sh`, covering defaults, forwarding/session isolation, explicit true/false wire values and independence from field 232. - Thrift C++/Java generation, FE Checkstyle, clang-format 16, all build-hygiene gates and changed-line clang-tidy checks passed. Lint findings were resolved by extracting helpers; no new lint suppressions were added. - Local ASan validation used the default Unity/PCH mode and matching dependency versions. An existing unused-constant warning in an unchanged scanner required a per-file local build adjustment; that configuration is excluded from the PR. ### Controlled source-branch measurements On `branch-4.2`, 49 focused Release BE tests, four FE tests, 16 standalone controller tests with ASan/UBSan and leak detection, and both opt-in controlled benchmarks passed. The new wire, repeated-failure and LRU regression tests failed before their fixes and passed afterward. Five alternating on/off rounds used 200 x 64 KiB file-backed requests per group, calibrated with 50 actual service-time samples on each path: | Injected local / remote latency | Disabled median | Enabled median | Enabled bypasses | | --- | ---: | ---: | ---: | | 8 ms / 1 ms | 1619.270 ms | 212.894 ms | 200/200 | | 1 ms / 8 ms | 213.492 ms | 213.307 ms | 0/200 | Every returned byte matched and neither scenario wrote cache data. These are controlled routing measurements from the source branch; master was validated with ASan, and production-query gains require separate measurement. ### Release note External-table scans can adaptively bypass slower disk-cache reads. Session variable `enable_file_cache_io_adaptor` controls this behavior and defaults to `true`. ### Check List (For Author) - Test: BE/FE unit tests and controlled local benchmarks described above. - Behavior changed: Yes; eligible external cache hits may read remote storage. - Does this need documentation: Yes; session setting and Profile counters. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
