airborne12 opened a new pull request, #69565:
URL: https://github.com/apache/doris/pull/69565
### What problem does this PR solve?
Related PR: #69292
Problem Summary: Backport #69292 to `branch-4.1`. It contains the commit of
#69563, the backport of #69399, which it needs: binding runs before CTE
inlining, which copies scans with virtual columns. Merge #69563 first.
`score()` only worked as `ORDER BY score() LIMIT n` directly above the scan
whose MATCH defines it. A minimum score filter without that TopN evaluated the
unbound `score()` placeholder as zero and silently dropped every row:
```sql
SELECT id FROM documents WHERE content MATCH 'apple' AND score() > 0.00001;
-- returned nothing
```
`BindScoreToOlapScan`, which replaces `CheckScoreUsage`, binds `score()` to
the scan of its SELECT at the start of rewrite and rejects the uses it cannot
bind. The minimum score stays a predicate of the scan, and
`PushDownScoreTopNIntoOlapScan` still pushes a TopN on the score together with
the minimum. In the BE, an expression on the score reads the score column
instead of evaluating `Score()`, and MATCH projections do not add to the score.
See #69292 for the supported and rejected uses.
#### Branch adaptations
- The `SegmentIterator` of the branch collects the columns of pushed-down
expressions in `_extract_common_expr_columns` instead of per-column states, so
the score column is marked there, and the new BE test checks
`_common_expr_columns` and `_is_common_expr_column`. The branch reads the
columns collected there before it evaluates the expressions, also virtual ones,
and a virtual slot uses a column that is already read.
- The branch evaluates MATCH projections with `evaluate_inverted_index`
directly, and that loop runs without the score collector.
- `PushDownScoreTopNIntoOlapScan` keeps the overflow check of the branch for
`limit + offset`, which fails like master does.
- `SearchDslParser` and `CollectionSimilarity` are included from their
packages on the branch.
- The branch has no SNII storage format, so `BindScoreToOlapScanTest` and
`test_score_filter_matrix` cover V2 and V3, and the `.out` of
`test_score_filter_matrix` is regenerated without the SNII cases.
- The branch does not have #68672, which keeps predicates on virtual columns
out of the condition cache. The minimum score is now a predicate of the scan,
and a score depends on every rowset, so a cached result goes stale when a
rowset is added: without the change, `test_score_filter_matrix` lost the rows
of the first rowset. `VirtualSlotRef::get_digest` returns 0 as on master, the
part of #68672 that this needs, with the `VirtualSlotRef` tests of #68672 that
the branch can run.
### Release note
`score()` works in the select list, window functions and ORDER BY of a
SELECT on one table whose WHERE clause has MATCH or SEARCH, with an optional
minimum score `score() > x` or `score() >= x`, without `ORDER BY score()
LIMIT`. A scored SELECT in a subquery, view, CTE or set operation still needs
`ORDER BY score() LIMIT`. Other uses of `score()` are rejected instead of
evaluating it as 0.
### Check List (For Author)
- Test
- [x] Regression test
- [x] Unit Test
- [ ] Manual test
- Behavior changed:
- [x] Yes. A minimum score works without `ORDER BY score() LIMIT`, and
unsupported uses of `score()` fail planning instead of evaluating it as 0. A
minimum score without a TopN needs the BE change, so BEs must be upgraded
before FEs.
- Does this need documentation?
- [x] Yes. The `score()` documentation should describe where `score()`
can be used.
### Validation
Validated together with the other two backports (#69399, #69419 and #69292
on `branch-4.1` at 3f911d030b), in a local ASAN build:
- PASS: FE unit tests `LogicalPlanDeepCopierTest`, `CTEInlineTest`,
`InferPredicateByReplaceTest`, `InferPredicatesTest`,
`UnequalPredicateInferTest`, `BindScoreToOlapScanTest`, `SearchExpressionTest`,
`TopNRuntimeFilterTest` and `MaterializedViewUtilsTest` (228 tests), and FE
Checkstyle.
- PASS: `./build.sh --be` and `./build.sh --fe`.
- PASS: BE unit tests `SegmentIteratorScoreCommonExprTest`,
`SegmentIteratorApplyIndexExprTest` and `VirtualSlotRefTest` (31 tests);
clang-format 16 on the changed BE files.
- PASS: the 16 regression suites of the branch that use `score()`, including
`test_cte_inline_virtual_column`, `infer_match_predicate`,
`test_score_filter_usage` and `test_score_filter_matrix`. The framework
regenerated the `.out` of `test_score_filter_matrix` on this branch: it drops
the 46 SNII cases and keeps the 94 V2 and V3 cases of master unchanged. The
other `.out` files are unchanged.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
