asdf2014 opened a new pull request, #65482: URL: https://github.com/apache/doris/pull/65482
### What problem does this PR solve? Related PR: N/A Problem Summary: For hourly batch-load workloads, every load bumps the tablet version of the hot partition, so its query cache entries never hit again: each "last N days" aggregation recomputes the whole hot partition every hour, and BE CPU spikes during load windows. This PR adds **incremental merge** for the query cache (experimental, default off). When a cache entry's version falls behind, BE reuses it instead of discarding it: it scans only the delta rowsets in `(cached_version, current_version]`, emits their partial aggregation state side by side with the cached partial blocks so the existing upstream merge aggregation combines both, and writes the merged entry back under the new version. The execution plan is unchanged.  **Measured impact** (80M-row duplicate-key table, 200k-row hourly appends, group by a non-distribution column): a stale query drops from **~307 ms** (full recompute) to **~19 ms** with incremental merge, about **16x** and on par with an exact hit (~17 ms); the cost no longer grows with the base data size (8M rows: 48 ms full vs 19 ms incremental; 80M rows: 307 ms vs 19 ms). Full numbers in the Verification section below. Design highlights: 1. Correctness algebra: `S(v2) = S(v1) ⊎ Δ` for append-only data plus the homomorphism `partial(A ⊎ B) ≡ merge(partial(A), partial(B))`. Every precondition guards one of the two properties; any violation falls back to a full recompute silently, so results are always correct. 2. FE authorizes via a new optional thrift field (`TQueryCacheParam.allow_incremental`) only for non-finalize aggregations directly above the olap scan on an append-only index: DUP_KEYS, or merge-on-write UNIQUE_KEYS. Merge-on-read UNIQUE and AGG tables always fall back. 3. BE re-validates per tablet at decision time (fallback reasons are visible in the profile): cloud mode, version order, the compaction threshold (`query_cache_max_incremental_merge_count`, default 8, 0 disables), keys type, delta capturability on the version graph, delete predicates in the delta, and for merge-on-write tables a delete-bitmap window check that rejects loads rewriting pre-existing keys. A rare backfill therefore costs exactly one full recompute, which re-bases the entry, and the next pure-append load is incremental again. 4. A fragment-level `QueryCacheRuntime` makes one idempotent decision per instance (HIT / INCREMENTAL / MISS), pins the entry and pre-captures the delta read source. This also fixes three pre-existing defects: the scan/cache-source double-lookup race (could write back an empty poisoned entry), row-binlog scans polluting the cache, and `build_cache_key` failures aborting the query instead of degrading to uncached execution. 5. Entropy control: every merge appends the delta blocks to the entry, so after `query_cache_max_incremental_merge_count` merges the next query recomputes in full and compacts the entry; oversized entries keep the delta-scan benefit but skip the write back. Verification: - FE UT: `QueryCacheNormalizerTest` 13/13 (8 assertions on the incremental authorization matrix: switch, plan shapes, DUP / MoW / MoR / AGG / nested aggregation). - BE UT: 34/34 across the decision layer, the incremental scenarios (MoW pure-append / history-rewrite / irrelevant bitmap entries, version gap, capture error via debug point, delete predicates) and the operator layer. llvm-cov: zero uncovered changed lines; `query_cache.cpp` at 100% line coverage. - Regression: `query_cache_incremental` passes on a real 1FE+1BE cluster; every step is checked against a cache-off baseline. BE metrics confirm the incremental path actually fires (`stale_hit_total` +40 across the suite, fallbacks exactly at the three designed spots). - Benchmark (80M-row DUP table, 200k-row hourly appends, group by a non-distribution column, 5 rounds each): | scenario | latency (median) | |---|---| | no cache, full scan | ~446 ms | | exact hit | ~17 ms | | stale + incremental merge (this PR) | ~19 ms | | stale + full recompute (switch off) | ~307 ms | A stale query becomes as cheap as an exact hit, and its cost no longer grows with the base data size (8M rows: 48ms full vs 19ms incremental; 80M rows: 307ms vs 19ms). ### Release note Add experimental incremental merge for the query cache: a stale entry can be reused by scanning only the delta rowsets and merging them with the cached partial aggregation state. Controlled by the session variable `enable_query_cache_incremental` (default off) and the BE config `query_cache_max_incremental_merge_count` (default 8). ### Check List (For Author) - Test - [x] Regression test - [x] Unit Test - Behavior changed: - [x] No. - Does this need documentation? - [x] Yes. Document PR: https://github.com/apache/doris-website/pull/3975 ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
