guptashivam04 opened a new pull request, #70547:
URL: https://github.com/apache/doris/pull/70547
### What problem does this PR solve?
Issue Number: None
Related PR: #70488 (master fix), #64782 (same fix, closed without merge)
Problem Summary:
`search()` has no row-by-row path, but a SEARCH term on a numeric (BKD)
field goes through `read_from_index` without `skip_try`. When the value matches
more than `inverted_index_skip_threshold` percent of a segment (50 by default),
the iterator returns `INVERTED_INDEX_BYPASS`, the segment iterator falls back
to row evaluation, and the query fails:
```
SELECT count(*) FROM t WHERE search('v.account_id:8069')
ERROR: [INTERNAL_ERROR]SearchExpr should not be executed without inverted
index
```
It often passes on a cold run and fails once the exact hit count comes from
the inverted index query cache, so it looks random to users. An account or
tenant id on a table loaded per account is enough to hit it.
This sets `skip_try` for the direct index reader leaf in
`FunctionSearch::build_leaf_query`, so SEARCH always reads the index. MATCH and
plain predicates keep the bypass. Workaround on unpatched versions: `SET
inverted_index_skip_threshold = 100`.
### Release note
Fix `search()` failing with "SearchExpr should not be executed without
inverted index" when a term on a numeric field matches more than
`inverted_index_skip_threshold` percent of a segment.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [x] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
Regression test `search/test_search_bkd_skip_threshold`, same checks and
expected output as master: one value covers 70% of the segment, on a variant
subcolumn and on an INT column, alone, with a text term, with OR, in standard
mode and with ORDER BY ... LIMIT. Unlike the master version, it warms both
numeric leaves once before the checks, because on 4.1 the first lookup of a
value passes on the BKD estimate and only later lookups see the exact count
from the query cache. With the warm-up, each dense check fails on its own
without the fix. Ran locally against a 4.1 cluster: fails before the fix with
"SearchExpr should not be executed without inverted index", passes after, and
still passes with `default_variant_max_subcolumns_count` set to 1 globally (the
test pins it to 2048).
`FunctionSearchTest.TestBuildLeafQueryVariantNestedIntUsesDirectIndexReader`
now checks `skip_try`; compiled locally, not run locally.
Timing on 2M rows (cache off, median): 20% match 18 ms before vs 14 ms
after, 70% match fails before and takes 17 ms after.
- Behavior changed:
- [ ] No.
- [x] Yes. SEARCH no longer follows `inverted_index_skip_threshold` for
numeric fields; queries that failed now return results.
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR should
merge into -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
