airborne12 opened a new pull request, #69292:
URL: https://github.com/apache/doris/pull/69292
### What problem does this PR solve?
Issue Number: N/A
Related PR: N/A
Problem Summary:
A predicate such as the following misses the score TopN rewrite:
```sql
SELECT id FROM documents
WHERE content MATCH 'apple' AND score() > 0.00001;
```
Without a score virtual column, the scalar `score()` placeholder evaluates
to zero and can silently discard matching rows. The existing rewrite requires a
score projection, `ORDER BY score`, and `LIMIT`.
This change binds a supported minimum score predicate to the existing score
virtual column and scan range descriptor. An explicit zero score limit enables
full scoring before ordinary filtering and sorting. Physical translation
retains the hidden score slot when the query does not project it. The existing
score TopN path shares range validation and keeps its optimization when
eligible.
The supported shape is one top-level AND comparison, `score() > literal` or
`score() >= literal`, with an existing MATCH or SEARCH scoring context.
Reversed comparisons and FLOAT/DOUBLE casts are supported. Score projections,
ordinary sorts, multiple ordering keys, LIMIT, and OFFSET also work with that
predicate. Unsupported raw score contexts, upper bounds, equality, and lossy
score casts fail analysis. SEARCH SQL rendering is added so DEBUG plan display
can render the expression.
The change reuses BE scoring and protocol fields. V2, V3, and SNII keep
their existing BM25 values and norm encodings; their numerical scores are not
made identical.
Validation completed locally:
- ASAN BE/FE build, final FE packaging, and 141 FE unit tests across five
affected classes.
- Six regression suites: `test_score_filter_usage`,
`test_score_filter_matrix`, `test_bm25_score_range_filter`,
`test_search_score_topn_predicates`, `test_search_score_topn_delete_predicate`,
and `test_search_score_cache`.
- 146 independently checked matrix result groups covering V2/V3/SNII, strict
and inclusive boundaries, projections, sorting, casts, deletes, MoW updates,
cache/WAND settings, and score filtering before ANN candidate truncation.
Result files were generated and replayed with the standard runner.
- V2 and SNII MATCH/SEARCH probes, each consuming all 50,000 qualifying rows
from 100,000 documents and checking scores against the existing TopN path.
- Full FE reactor Checkstyle on the candidate merged with current master,
and the complete committed-diff English check.
Cloud execution and mixed-version execution have not been tested.
### Release note
Support one `score() > literal` or `score() >= literal` filter with MATCH or
SEARCH without requiring ORDER BY or LIMIT. Reject unsupported score contexts
and lossy score casts instead of evaluating an unmaterialized or altered score
expression.
### Check List (For Author)
- Test
- [x] Regression test
- [x] Unit Test
- [x] Manual test (100,000-document probes described above)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason
- Behavior changed:
- [ ] No.
- [x] Yes. Minimum score filtering can run without SQL TopN; unsupported
score usage raises an analysis error.
- Does this need documentation?
- [ ] No.
- [x] Yes. The score() documentation should describe standalone minimum
score predicates. A documentation PR has not been opened.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
