airborne12 opened a new pull request, #69269:
URL: https://github.com/apache/doris/pull/69269
### What problem does this PR solve?
Related PR: #69052
Backport regexp and wildcard SEARCH scoring to `branch-4.1`. Scored
expansions currently return constant scores. The fix uses existing `TermScorer`
instances to compute BM25 for each expanded term and sums the matching
contributions. Unscored queries retain the bitmap path.
This uses the compatible original change from
`34a123b85da6ffc4990157820211dd08462744ec` with the branch's `query_v2`
interfaces. Legacy indexes without total-term-frequency metadata retain the
unit-average-length fallback. The backport also adds independent term-score
assertions and SQL regression coverage.
### Release note
Scored SEARCH regexp and wildcard expansions use BM25 relevance scores.
### Validation
Tested head: `07de3d3acc1adeddae8ba85c6da295aa047f0b6f`, including current
`branch-4.1` base `c7f95adaaaec4bc0caeab9ee831a9009745d56d0`.
- ASAN UT: 636 tests from 36 related suites passed; zero failures, errors or
skips. The two new expansion-scoring tests fail against the original production
implementation and pass with this fix, including equality with independently
collected `TermQuery` contributions.
- Full ASAN and Release FE/BE builds completed. After synchronizing the
newer Boolean-scoring base, both BE types were rebuilt and the unchanged FE
artifacts were reused. Both binaries retain GLIBC compatibility (maximum
required GLIBC 2.10); ASAN instrumentation was verified in the ASAN binary.
- ASAN and Release each passed 16 related SEARCH/inverted-index suites, with
zero failed suites, fatal scripts or skips. Coverage includes
scoring/cache/TopN/range predicates, Boolean branches, regexp and array regexp,
phrase/prefix/edge, custom analyzers and raw indexes.
- Each binary passed 24 direct score-comparison SQL queries across V2/V3,
standard/keyword indexes and WAND on/off. Regexp and wildcard scores match
explicit expanded-term queries. The new regression has 48 framework-generated
golden cases, replayed on both binaries, including NULL input, empty matches,
ordering and unscored queries.
- Runtime logs contain no sanitizer errors. All task-owned test instances
were stopped. Full merged-tree clang-format 16 and English-description checks
passed.
Additional static-check limit: clang-tidy 16 cannot parse the compiler-20
JNI assertions; clang-tidy 20 is blocked by an unmatched NOLINTEND in the
unchanged baseline. An overall clang-tidy pass is not claimed.
### Check List (For Author)
- Test
- [x] Regression test
- [x] Unit Test
- [x] Manual test (score-comparison matrix described above)
- Behavior changed:
- [x] Yes. Scored regexp/wildcard expansions return BM25 contributions.
- Does this need documentation?
- [x] No. No syntax or setting is added.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
