This is an automated email from the ASF dual-hosted git repository.
airborne12 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 011cf6fd08e [fix](search) Keep numeric SEARCH leaves on the inverted
index (#70488)
011cf6fd08e is described below
commit 011cf6fd08e2b1839f3d7e34537473b83212dd4c
Author: guptashivam04 <[email protected]>
AuthorDate: Fri Nov 13 08:54:48 2026 +0530
[fix](search) Keep numeric SEARCH leaves on the inverted index (#70488)
### Background
`search()` only runs through the inverted index; there is no row-by-row
path, and `vsearch.cpp` turns any index bypass into "SEARCH cannot fall
back to row execution" (added in #66052). For numeric fields (BKD
readers), `InvertedIndexIterator::read_from_index` first estimates the
hit count and returns `INVERTED_INDEX_BYPASS` when it is over
`inverted_index_skip_threshold` percent of the segment (50 by default),
so the predicate can be evaluated on rows instead. That bypass is useful
for normal predicates and MATCH, not for SEARCH.
### What problem does this PR solve?
Issue Number: None
Related PR: #64782 (same fix for the pre-refactor code, closed without
merge), #68758 (moved the scalar leaf into `ScalarLeafCompiler`, bug
kept)
PR Type: Bugfix
Problem Summary:
A SEARCH term on a numeric field fails when the value matches more than
half of a segment, for example an account id on a table loaded per
account:
```
SELECT count(*) FROM t WHERE search('v.account_id:8069')
ERROR: [E-6001]SEARCH cannot fall back to row execution: [E-6004]hit count:
350255, bkd inverted reached limit 50% , segment num rows:500000
```
`ScalarLeafCompiler::compile` now sets `skip_try` so SEARCH leaves
always read the index. MATCH and plain predicates keep the bypass.
### Details
- **Frequency:** Any SEARCH term on a numeric column or numeric variant
subcolumn whose value covers more than `inverted_index_skip_threshold`
percent of a segment. Common for tenant or account ids. On 4.1 it often
shows up only on the second run, once the exact hit count comes from the
query cache.
- **Root cause:** `ScalarLeafCompiler::compile` calls `read_from_index`
with `skip_try = false`, so the BKD hit-ratio check applies to SEARCH
even though SEARCH cannot evaluate rows.
- **Why existing tests missed it:** No SEARCH test runs a numeric term
that covers more than half a segment with the default threshold;
`test_search_inverted_is_null_pushdown` sets
`inverted_index_skip_threshold=0`, which turns the bypass off.
- **Fix:** Set `param.skip_try = true` before `read_from_index` in
`ScalarLeafCompiler::compile`. This is the only `read_from_index` call
on the SEARCH path.
- **Other paths and branches:** branch-4.1 has the same bug in
`FunctionSearch::build_leaf_query`; a separate PR picks this there.
Numeric range clauses (`field:[a TO b]`) are still unsupported in
SEARCH; that is not changed here.
### Testing
- New regression test `search/test_search_bkd_skip_threshold`: a MOW
table where one value covers 70% of the segment, on a variant subcolumn
and on a plain INT column, alone, combined with a text term, in standard
mode and with ORDER BY ... LIMIT, each run 3 times with the query cache
on. Fails before the fix with "SEARCH cannot fall back to row
execution", passes after. Ran locally against a cluster built from this
branch, and against the same commit without the fix.
-
`FunctionSearchTest.TestBuildLeafQueryVariantNestedIntUsesDirectIndexReader`
now checks that the scalar leaf passes `skip_try = true` to the
iterator. Compiled locally with the BE unit-test flags; not run locally
(no BE UT build), left to CI.
- Manual timing on 2M rows (index query cache and SQL cache off, median
of 11): a 20% numeric match 10 ms before vs 12 ms after (noise), the 70%
match fails before and takes 16 ms after; text SEARCH and plain filters
unchanged.
### Release note
Fix `search()` failing with "SEARCH cannot fall back to row execution"
when a term on a numeric field matches more than
`inverted_index_skip_threshold` percent of a segment.
### Check List (For Author)
- Behavior changed: Yes (see Details). SEARCH no longer follows
`inverted_index_skip_threshold` for numeric fields; queries that failed
now return results.
- Does this need documentation: No
- Pre-submission status: Partially verified: the BE unit test was
compiled but not run locally.
---------
Co-authored-by: Shivam Gupta <[email protected]>
---
.../exprs/function/search/function_search_leaf.cpp | 3 +
.../exprs/function/search/function_search_test.cpp | 4 ++
.../data/search/test_search_bkd_skip_threshold.out | 26 +++++++
.../search/test_search_bkd_skip_threshold.groovy | 84 ++++++++++++++++++++++
4 files changed, 117 insertions(+)
diff --git a/be/src/exprs/function/search/function_search_leaf.cpp
b/be/src/exprs/function/search/function_search_leaf.cpp
index f9e3de40532..cdb2e5f9824 100644
--- a/be/src/exprs/function/search/function_search_leaf.cpp
+++ b/be/src/exprs/function/search/function_search_leaf.cpp
@@ -177,6 +177,9 @@ Status ScalarLeafCompiler::compile(const logical::Node&
leaf, const SearchLeafCo
param.query_type = segment_v2::InvertedIndexQueryType::EQUAL_QUERY;
param.num_rows = ctx.num_rows;
param.roaring = std::make_shared<roaring::Roaring>();
+ // SEARCH has no row-by-row fallback, so a BKD leaf must not take the
+ // inverted_index_skip_threshold bypass: a bypassed leaf would fail the
whole query.
+ param.skip_try = true;
RETURN_IF_ERROR(_iterator->read_from_index(segment_v2::IndexParam
{¶m}));
// Physical NULL postings are excluded by the shared SEARCH leaf mask.
Never
diff --git a/be/test/exprs/function/search/function_search_test.cpp
b/be/test/exprs/function/search/function_search_test.cpp
index cec322d67ef..69c0e5b2cfe 100644
--- a/be/test/exprs/function/search/function_search_test.cpp
+++ b/be/test/exprs/function/search/function_search_test.cpp
@@ -177,6 +177,7 @@ public:
if (i_param->query_value.get_type() == TYPE_INT) {
last_int_value = i_param->query_value.get<TYPE_INT>();
}
+ last_skip_try = i_param->skip_try;
if (i_param->roaring != nullptr) {
i_param->roaring->add(3);
}
@@ -196,6 +197,7 @@ public:
PrimitiveType last_query_value_type = PrimitiveType::TYPE_NULL;
bool last_bool_value = false;
Int32 last_int_value = 0;
+ bool last_skip_try = false;
};
class RecordingDirectInvertedIndexIterator final : public
segment_v2::InvertedIndexIterator {
@@ -3379,6 +3381,8 @@ TEST_F(FunctionSearchTest,
TestBuildLeafQueryVariantNestedIntUsesDirectIndexRead
EXPECT_EQ(InvertedIndexQueryType::EQUAL_QUERY, iterator.last_query_type);
EXPECT_EQ(TYPE_INT, iterator.last_query_value_type);
EXPECT_EQ(3, iterator.last_int_value);
+ // SEARCH has no row fallback, so the BKD skip-threshold bypass must be
disabled.
+ EXPECT_TRUE(iterator.last_skip_try);
}
TEST_F(FunctionSearchTest, TestMultiPhraseQueryCase) {
diff --git a/regression-test/data/search/test_search_bkd_skip_threshold.out
b/regression-test/data/search/test_search_bkd_skip_threshold.out
new file mode 100644
index 00000000000..77a2bdab1d1
--- /dev/null
+++ b/regression-test/data/search/test_search_bkd_skip_threshold.out
@@ -0,0 +1,26 @@
+-- This file is automatically generated. You should know what you did if you
want to edit this
+-- !dense_variant --
+7
+
+-- !dense_column --
+7
+
+-- !dense_and_text --
+4
+
+-- !dense_or --
+9
+
+-- !sparse_variant --
+2
+
+-- !sparse_column --
+2
+
+-- !dense_topn --
+7
+6
+
+-- !dense_standard_mode --
+4
+
diff --git
a/regression-test/suites/search/test_search_bkd_skip_threshold.groovy
b/regression-test/suites/search/test_search_bkd_skip_threshold.groovy
new file mode 100644
index 00000000000..5823d9f2115
--- /dev/null
+++ b/regression-test/suites/search/test_search_bkd_skip_threshold.groovy
@@ -0,0 +1,84 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// A numeric (BKD) leaf that matches more than inverted_index_skip_threshold
percent of a
+// segment used to return INVERTED_INDEX_BYPASS. SEARCH has no row-by-row
path, so the
+// query failed with "SEARCH cannot fall back to row execution". SEARCH must
always use
+// the index.
+suite("test_search_bkd_skip_threshold", "p0") {
+ sql """ set enable_common_expr_pushdown = true """
+ sql """ set enable_inverted_index_query_cache = true """
+ sql """ set inverted_index_skip_threshold = 50 """
+ sql """ set default_variant_enable_typed_paths_to_sparse = false """
+ sql """ set default_variant_enable_doc_mode = false """
+ // Keep both v.n and v.p as indexed subcolumns; a fuzzy budget of 1 would
push one to sparse.
+ sql """ set default_variant_max_subcolumns_count = 2048 """
+
+ sql "DROP TABLE IF EXISTS test_search_bkd_skip_threshold_t"
+ sql """
+ CREATE TABLE test_search_bkd_skip_threshold_t (
+ `id` INT NOT NULL,
+ `k` INT NULL,
+ `v` variant NULL,
+ INDEX idx_k (k) USING INVERTED,
+ INDEX idx_v (v) USING INVERTED PROPERTIES(
+ "parser" = "unicode", "lower_case" = "true", "support_phrase"
= "true")
+ ) ENGINE=OLAP
+ UNIQUE KEY(`id`)
+ DISTRIBUTED BY HASH(`id`) BUCKETS 1
+ PROPERTIES (
+ "replication_allocation" = "tag.location.default: 1",
+ "enable_unique_key_merge_on_write" = "true",
+ "disable_auto_compaction" = "true"
+ )
+ """
+
+ // 7 of 10 rows have n = 5 and k = 5, so a BKD lookup for 5 hits 70% of
the segment.
+ sql """INSERT INTO test_search_bkd_skip_threshold_t VALUES
+ (1, 5, parse_to_variant('{"n": 5, "p": "software engineer"}')),
+ (2, 5, parse_to_variant('{"n": 5, "p": "sales"}')),
+ (3, 5, parse_to_variant('{"n": 5, "p": "software developer"}')),
+ (4, 5, parse_to_variant('{"n": 5, "p": "sales"}')),
+ (5, 5, parse_to_variant('{"n": 5, "p": "software engineer"}')),
+ (6, 5, parse_to_variant('{"n": 5, "p": "sales"}')),
+ (7, 5, parse_to_variant('{"n": 5, "p": "software tester"}')),
+ (8, 6, parse_to_variant('{"n": 6, "p": "software engineer"}')),
+ (9, 6, parse_to_variant('{"n": 6, "p": "sales"}')),
+ (10, 7, parse_to_variant('{"n": 7, "p": "software engineer"}'))
+ """
+
+ // Run each query twice first: an early run can pass on the BKD estimate,
while later runs
+ // read the exact hit count from the query cache, which is over the
threshold.
+ def queries = ["v.n:5", "k:5", "v.n:5 AND v.p:software", "v.n:5 OR v.n:6",
"v.n:6", "k:6"]
+ for (int round = 0; round < 2; round++) {
+ for (String q : queries) {
+ sql "SELECT count(*) FROM test_search_bkd_skip_threshold_t WHERE
search('${q}')"
+ }
+ }
+
+ qt_dense_variant """SELECT count(*) FROM test_search_bkd_skip_threshold_t
WHERE search('v.n:5')"""
+ qt_dense_column """SELECT count(*) FROM test_search_bkd_skip_threshold_t
WHERE search('k:5')"""
+ qt_dense_and_text """SELECT count(*) FROM test_search_bkd_skip_threshold_t
+ WHERE search('v.n:5 AND v.p:software')"""
+ qt_dense_or """SELECT count(*) FROM test_search_bkd_skip_threshold_t WHERE
search('v.n:5 OR v.n:6')"""
+ qt_sparse_variant """SELECT count(*) FROM test_search_bkd_skip_threshold_t
WHERE search('v.n:6')"""
+ qt_sparse_column """SELECT count(*) FROM test_search_bkd_skip_threshold_t
WHERE search('k:6')"""
+ qt_dense_topn """SELECT id FROM test_search_bkd_skip_threshold_t
+ WHERE search('v.n:5') ORDER BY id DESC LIMIT 2"""
+ qt_dense_standard_mode """SELECT count(*) FROM
test_search_bkd_skip_threshold_t
+ WHERE search('v.n:5 AND v.p:software',
'{"mode":"standard"}')"""
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
