Authorlove opened a new pull request, #69403: URL: https://github.com/apache/doris/pull/69403
### What problem does this PR solve? Issue Number: None Related PR: https://github.com/apache/doris/pull/69313 and https://github.com/apache/doris-thirdparty/pull/420 (separate compaction fixes; this PR is independent) Problem Summary: After DELETE or an overwrite becomes visible, CLucene BM25 still reads physical index document counts, token totals and term document frequencies. Consequently identical visible rows can score and rank differently until compaction. Collect BM25 statistics from the query's captured visible collection for CLucene V2/V3 DUP and merge-on-write UNIQUE tables. The rowset list, version-bounded delete bitmap and delete predicates are shared before scanner splitting; delete-sign rows are excluded using the storage reader. Each tablet's statistics are initialized once and published with precomputed IDF/average-length caches, so scanner partitioning cannot change the scoring population. This retains the existing tablet-scoped collection definition; user search/range predicates do not redefine the corpus. Exact document token lengths are cached by immutable physical index/field/properties in the bounded searcher LRU. Frequency-bearing indexes reconstruct them from postings; frequency-free historical indexes read and tokenize the original string/string-array column, preserving NULL/empty and ignore_above behavior. Quantized norms are not treated as exact lengths. Each query applies its own visibility mask and corrects document frequency using the smaller visible/excluded set. The cache publishes only complete loads, retries failures, and allows waiting queries to cancel without cancelling another query's successful load. The visibility read also exposed a shared reader bug: an empty EOF block could retain the preceding block's row IDs. Clear those IDs and only fetch locations for nonempty blocks, preserving the cardinality assertion. This change does not modify the CLucene dependency or compaction implementation. SNII, MOR/AGG and row-binlog scoring retain their existing paths. Dynamic VARIANT shapes that cannot be read as a proven string/string array are not silently approximated by the raw-length fallback. No persisted format or Thrift/protobuf field changes are introduced, and scanner parameter additions preserve positional initializer compatibility. ### Release note BM25 on CLucene V2/V3 DUP and merge-on-write UNIQUE tables now uses the query's visible document collection after deletes and overwrites, without waiting for compaction. Scores, ordering and score-threshold results can change accordingly. Cold queries may scan postings or raw indexed text to populate exact-length caches; these bounded caches reserve roughly 8 bytes per physical document per indexed field. Queries still read visibility-related columns. This is not a claim of performance-neutral behavior. The related compaction PRs address persisted statistics separately. They are not prerequisites for this query-time correction. ### Check List (For Author) - Test - [ ] Same-base Release build comparison, cold/warm query measurements, and final independent-tree SQL validation (in progress). - [x] ASAN BE built with the official current dependency bundle and LDB 0.25. ASAN runs included the separate compaction commits before this change was rebased independently; realtime source is identical. - [x] New realtime SQL suite passes all 20 combinations (V2/V3 × phrase support × five history modes), with automatic compaction disabled and no compaction trigger. - [x] Comparisons cover repeated queries, subsequent deletes, length-41 reinserts, NULL/empty values, all-deleted snapshots, inserts after empty snapshots, score thresholds, TopK/offset and forced parallel scanner splitting. - [x] The same new SQL suite fails on unpatched 4.1.5 at the expected score comparison. For two visible documents, clean scores are 0.28530657 / 0.20084909; the old history-bearing table scores 0.177374 / 0.192449. The patched BE produces the clean scores immediately. - [x] 74 targeted ASAN BE unit tests passed: live-mask token/df collection, concurrent cache publication, failed-load retry, empty-cache results, cancelled waiters, repeated tokens, length 41 and nullable arrays. - [x] Existing 20-combination compaction regression passes after the shared EOF row-ID fix. - [x] Final formatting, build hygiene and source/test changed-line clang-tidy checks passed. - Targeted unit tests used the complete two selected suites plus required test harness through `run-be-ut.sh`, with temporary CMake source selection restored afterward. The unrelated full unit build has pre-existing obsolete JSON-test initializers. - Local SQL validation uses an isolated patched ASAN BE with a released 4.1.5 FE; a complete master FE/BE CI run is still required. Optional Java/CDC extensions are excluded from the local BE build. The dictionary build utility needs a narrow local LSan suppression for libomp's process-exit `___kmp_allocate_align` allocation. - Behavior changed: - [x] Yes. Supported BM25 queries honor visible rows before compaction. Missing required index/data information fails instead of publishing partial statistics. - Does this need documentation? - [x] The behavior, scope and cold-query/cache cost are described above; production rollout still requires capacity review. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
