This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 40ff995f890 [improvement](inverted index) Short-circuit pushed-down
conjuncts once the row bitmap is empty (#67171)
40ff995f890 is described below
commit 40ff995f890802d5c6877b3e6f390f117d7c5629
Author: Jack <[email protected]>
AuthorDate: Mon Aug 31 09:13:42 2026 +0800
[improvement](inverted index) Short-circuit pushed-down conjuncts once the
row bitmap is empty (#67171)
### What problem does this PR solve?
Problem Summary:
On log-search workloads a query typically carries several pushed-down
conjuncts, e.g. two cheap untokenized MATCH terms plus one expensive
`msg MATCH_PHRASE_PREFIX '...'`:
```sql
SELECT * FROM logs
WHERE _ctime_ >= ... AND _ctime_ <= ...
AND _namespace_ MATCH 'ns' -- untokenized term, ~free
AND _pod_name_ MATCH 'pod-xyz' -- untokenized term, ~free
AND msg MATCH_PHRASE_PREFIX '...' -- whole-segment postings +
positions
ORDER BY _ctime_ DESC LIMIT 1000
```
`SegmentIterator` evaluated ALL pushed-down conjuncts against the index
first and only intersected their bitmaps into `_row_bitmap` afterwards,
so the phrase conjunct paid its full whole-segment postings/positions
cost even when an earlier selective conjunct had already emptied the
candidate bitmap. Reproduced on a production log table: one query
returning 0 rows burned 11,291 CPU seconds and 124 GiB of local index
reads per physical SQL, virtually all of it inside phrase evaluation
(profile: 2,657 segments, InvertedIndexSearcherSearchExecTime = 99.7% of
scan cost, ScanRows = 0).
The empty-bitmap short circuit already exists on the neighbouring paths:
column predicates (`continue_apply` in `_apply_inverted_index`) and
compound AND inside a single expression (`VCompoundPred` COMPOUND_AND
early exit). But top-level AND conjuncts are flattened into separate
expr contexts and take `_apply_index_expr`, which had neither
progressive intersection nor a short circuit, so the most common query
shape never benefited.
Fix: intersect each consumed index result into `_row_bitmap` as soon as
it is produced, and stop evaluating further conjuncts once the bitmap is
empty. A skipped conjunct stays pushed down, so the row-level path keeps
its exact semantics over the now-empty candidate set (zero rows read,
zero cost). Consumed conjuncts are erased from
`_common_expr_ctxs_push_down` only after the ANN range-search pass,
which iterates the same list. A new profile counter
`InvertedIndexConjunctsShortCircuited` reports how many conjuncts were
skipped.
Behavior notes:
- Index results are now intersected before the ANN range-search pass, so
ANN range search executes on the already-narrowed bitmap. AND semantics
are commutative, and a smaller candidate set is what its small-candidate
fallback is designed for.
- Condition cache digest semantics unchanged: a short-circuited (thus
still pushed-down) conjunct keeps the digest cleared, so partial index
results are never cached as full coverage.
### Release note
Inverted index: once earlier pushed-down conjuncts empty the candidate
row bitmap, remaining conjuncts (e.g. expensive MATCH_PHRASE_PREFIX) are
no longer evaluated against the index.
### Check List (For Author)
- Test <!-- At least one of them must be included. -->
- [ ] Regression test
- [x] Unit Test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change.
- [ ] No code files have been changed.
- [ ] Other reason <!-- Add your reason? -->
- Behavior changed:
- [ ] No.
- [x] Yes. <!-- Index results are intersected progressively (before the
ANN pass); short-circuited conjuncts fall back to the row-level path
over an empty candidate set. Same query results, strictly less index
work. -->
- Does this need documentation?
- [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
https://github.com/apache/doris-website/pull/1214 -->
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
---
Review round 2 (automated review, addressed in the third commit):
1. Dead work after the short circuit is now bypassed on an empty
candidate
bitmap: virtual-column MATCH projections skip their whole-segment index
evaluation, the ANN range-search pass breaks out before
loading/searching,
and ANN TopN falls back at entry (all three previously ran even with
zero
surviving rows when the small-candidate fallback thresholds are
disabled).
2. The consumed-list clear no longer drops the proven all-false
condition-cache entry: `_index_conjuncts_proved_empty` keeps the digest
alive for the exhausted case (the empty result is correct for the full
conjunction), while a counter-case test pins the normal fully-consumed
path to its existing digest-zeroing behavior.
---
be/src/exec/operator/olap_scan_operator.cpp | 2 +
be/src/exec/operator/olap_scan_operator.h | 1 +
be/src/exec/scan/olap_scanner.cpp | 2 +
be/src/storage/olap_common.h | 3 +
be/src/storage/segment/segment_iterator.cpp | 63 +++-
be/src/storage/segment/segment_iterator.h | 8 +
...egment_iterator_conjunct_short_circuit_test.cpp | 327 +++++++++++++++++++++
7 files changed, 405 insertions(+), 1 deletion(-)
diff --git a/be/src/exec/operator/olap_scan_operator.cpp
b/be/src/exec/operator/olap_scan_operator.cpp
index b1fb21f764a..d8fcac7579f 100644
--- a/be/src/exec/operator/olap_scan_operator.cpp
+++ b/be/src/exec/operator/olap_scan_operator.cpp
@@ -290,6 +290,8 @@ Status OlapScanLocalState::_init_profile() {
_segment_profile, "InvertedIndexSearcherCacheMiss", TUnit::UNIT,
1);
_inverted_index_downgrade_count_counter =
ADD_COUNTER_WITH_LEVEL(_segment_profile,
"InvertedIndexDowngradeCount", TUnit::UNIT, 1);
+ _inverted_index_conjuncts_short_circuited_counter = ADD_COUNTER_WITH_LEVEL(
+ _segment_profile, "InvertedIndexConjunctsShortCircuited",
TUnit::UNIT, 1);
_inverted_index_analyzer_timer =
ADD_TIMER_WITH_LEVEL(_segment_profile,
"InvertedIndexAnalyzerTime", 1);
_inverted_index_lookup_timer =
diff --git a/be/src/exec/operator/olap_scan_operator.h
b/be/src/exec/operator/olap_scan_operator.h
index 88440fe5bd8..11d40452f93 100644
--- a/be/src/exec/operator/olap_scan_operator.h
+++ b/be/src/exec/operator/olap_scan_operator.h
@@ -265,6 +265,7 @@ private:
RuntimeProfile::Counter* _inverted_index_searcher_cache_hit_counter =
nullptr;
RuntimeProfile::Counter* _inverted_index_searcher_cache_miss_counter =
nullptr;
RuntimeProfile::Counter* _inverted_index_downgrade_count_counter = nullptr;
+ RuntimeProfile::Counter* _inverted_index_conjuncts_short_circuited_counter
= nullptr;
RuntimeProfile::Counter* _inverted_index_analyzer_timer = nullptr;
RuntimeProfile::Counter* _inverted_index_lookup_timer = nullptr;
diff --git a/be/src/exec/scan/olap_scanner.cpp
b/be/src/exec/scan/olap_scanner.cpp
index ed049b8c7ef..4fafd8ffb30 100644
--- a/be/src/exec/scan/olap_scanner.cpp
+++ b/be/src/exec/scan/olap_scanner.cpp
@@ -881,6 +881,8 @@ void OlapScanner::_collect_profile_before_close() {
stats.inverted_index_searcher_cache_miss);
COUNTER_UPDATE(local_state->_inverted_index_downgrade_count_counter,
stats.inverted_index_downgrade_count);
+
COUNTER_UPDATE(local_state->_inverted_index_conjuncts_short_circuited_counter,
+ stats.inverted_index_conjuncts_short_circuited);
COUNTER_UPDATE(local_state->_inverted_index_analyzer_timer,
stats.inverted_index_analyzer_timer);
COUNTER_UPDATE(local_state->_inverted_index_lookup_timer,
stats.inverted_index_lookup_timer);
diff --git a/be/src/storage/olap_common.h b/be/src/storage/olap_common.h
index 55d84154e68..0b05f8fdeb4 100644
--- a/be/src/storage/olap_common.h
+++ b/be/src/storage/olap_common.h
@@ -325,6 +325,9 @@ struct OlapReaderStatistics {
int64_t inverted_index_searcher_cache_hit = 0;
int64_t inverted_index_searcher_cache_miss = 0;
int64_t inverted_index_downgrade_count = 0;
+ // Pushed-down conjuncts skipped (never index-evaluated) because the row
+ // bitmap was already empty when their turn came.
+ int64_t inverted_index_conjuncts_short_circuited = 0;
int64_t inverted_index_analyzer_timer = 0;
int64_t inverted_index_lookup_timer = 0;
// See snii_query_stats.h: one field here instead of one per SNII counter.
diff --git a/be/src/storage/segment/segment_iterator.cpp
b/be/src/storage/segment/segment_iterator.cpp
index 08ec84d700f..1a8ef62134a 100644
--- a/be/src/storage/segment/segment_iterator.cpp
+++ b/be/src/storage/segment/segment_iterator.cpp
@@ -858,7 +858,9 @@ Status
SegmentIterator::_get_row_ranges_by_column_conditions() {
}
}
_opts.condition_cache_digest =
- _common_expr_ctxs_push_down.empty() ? 0 :
_opts.condition_cache_digest;
+ _common_expr_ctxs_push_down.empty() &&
!_index_conjuncts_proved_empty
+ ? 0
+ : _opts.condition_cache_digest;
_opts.stats->rows_inverted_index_filtered += (input_rows -
_row_bitmap.cardinality());
for (uint32_t cid = 0; cid < _schema->num_read_columns(); ++cid) {
bool result_true =
_check_all_conditions_passed_inverted_index_for_column(cid);
@@ -925,6 +927,13 @@ Status SegmentIterator::_apply_ann_topn_predicate() {
if (_ann_topn_runtime == nullptr) {
return Status::OK();
}
+ if (_row_bitmap.isEmpty()) {
+ // Zero candidates: nothing for TopN to select, and the residual
+ // conjuncts that used to veto this path may have been consumed by the
+ // proved-empty short circuit. Fall back before touching the ANN index
+ // (the small-candidate fallback may be disabled by its thresholds).
+ return Status::OK();
+ }
VLOG_DEBUG << fmt::format("Try apply ann topn: {}",
_ann_topn_runtime->debug_string());
// AnnTopNRuntime keeps VSlotRef::column_id(), which is the read-schema
ordinal.
@@ -1237,7 +1246,23 @@ Status SegmentIterator::_apply_index_expr() {
!_opts.runtime_state->query_options().__isset.enable_ann_index_result_cache ||
_opts.runtime_state->query_options().enable_ann_index_result_cache;
+ // Intersect each consumed index result into _row_bitmap right away so a
+ // selective conjunct short-circuits the remaining (potentially expensive,
+ // e.g. MATCH_PHRASE_PREFIX) ones. A skipped conjunct stays pushed down and
+ // keeps its semantics on the row-level path, which then sees zero rows.
+ // Consumed conjuncts are erased only after the ANN pass below, which
+ // iterates the same list.
+ std::vector<const VExprContext*> consumed_by_index;
+ bool bitmap_exhausted = false;
+ size_t considered_conjuncts = 0;
for (const auto& expr_ctx : _common_expr_ctxs_push_down) {
+ if (_row_bitmap.isEmpty()) {
+ _opts.stats->inverted_index_conjuncts_short_circuited +=
+ _common_expr_ctxs_push_down.size() - considered_conjuncts;
+ bitmap_exhausted = true;
+ break;
+ }
+ ++considered_conjuncts;
if (Status st = expr_ctx->evaluate_inverted_index(num_rows());
!st.ok()) {
if (_downgrade_without_index(st) || st.code() ==
ErrorCode::NOT_IMPLEMENTED_ERROR) {
continue;
@@ -1249,12 +1274,25 @@ Status SegmentIterator::_apply_index_expr() {
return st;
}
}
+ if (expr_ctx->all_expr_inverted_index_evaluated()) {
+ const auto* result =
expr_ctx->get_index_context()->get_index_result_for_expr(
+ expr_ctx->root().get());
+ if (result != nullptr) {
+ _row_bitmap &= *result->get_data_bitmap();
+ consumed_by_index.push_back(expr_ctx.get());
+ }
+ }
}
// Evaluate inverted index for virtual column MATCH expressions
(projections).
// Unlike common exprs which filter rows, these only compute index result
bitmaps
// for later materialization via fast_execute().
for (auto& [cid, expr_ctx] : _virtual_column_exprs) {
+ if (_row_bitmap.isEmpty()) {
+ // Zero surviving rows: the projection column is never
materialized,
+ // so its whole-segment index evaluation would be pure waste.
+ break;
+ }
if (expr_ctx->get_index_context() == nullptr) {
continue;
}
@@ -1272,6 +1310,13 @@ Status SegmentIterator::_apply_index_expr() {
// Apply ann range search
for (const auto& expr_ctx : _common_expr_ctxs_push_down) {
+ if (_row_bitmap.isEmpty()) {
+ // A range search intersects into the bitmap; with zero candidates
+ // it cannot add rows, so loading and searching the ANN index
+ // (bypassing the small-candidate fallback when its thresholds are
+ // disabled) would be pure waste.
+ break;
+ }
segment_v2::AnnIndexStats ann_index_stats;
size_t origin_rows = _row_bitmap.cardinality();
bool ann_range_search_executed = false;
@@ -1302,6 +1347,22 @@ Status SegmentIterator::_apply_index_expr() {
_opts.stats->ann_index_range_cache_hits +=
ann_index_stats.range_cache_hits.value();
}
+ if (bitmap_exhausted) {
+ // Zero surviving rows satisfy every remaining conjunct, so the whole
+ // list is consumed -- mirroring the column-predicate short circuit.
+ // This keeps the "all conditions consumed by the index" contract, and
+ // _index_conjuncts_proved_empty keeps the condition-cache digest
+ // alive: the all-false result is correct for the full conjunction, so
+ // clearing the list must not degrade it to "nothing left to cache".
+ _index_conjuncts_proved_empty = true;
+ _common_expr_ctxs_push_down.clear();
+ } else if (!consumed_by_index.empty()) {
+ std::erase_if(_common_expr_ctxs_push_down, [&](const VExprContextSPtr&
ctx) {
+ return std::find(consumed_by_index.begin(),
consumed_by_index.end(), ctx.get()) !=
+ consumed_by_index.end();
+ });
+ }
+
return Status::OK();
}
diff --git a/be/src/storage/segment/segment_iterator.h
b/be/src/storage/segment/segment_iterator.h
index dfe2fd842eb..5602c6280ab 100644
--- a/be/src/storage/segment/segment_iterator.h
+++ b/be/src/storage/segment/segment_iterator.h
@@ -484,6 +484,14 @@ private:
bool _count_fastpath_hit = false;
bool _count_emit_shortcut = false;
uint64_t _count_emit_rows_remaining = 0;
+
+ // An indexed conjunct prefix emptied _row_bitmap, proving the WHOLE
+ // pushed-down conjunction false. Set by the _apply_index_expr short
+ // circuit when it consumes (clears) the remaining conjuncts, and read
+ // where an empty conjunct list would otherwise zero the condition-cache
+ // digest: the all-false result stays valid for the full conjunction, so
+ // it must remain cacheable.
+ bool _index_conjuncts_proved_empty = false;
// Batch size for shortcut emission: VStatisticsIterator's
// MAX_ROW_SIZE_IN_COUNT, the largest default-rows block shape already
// proven through every consumer above the segment iterator by the plain
diff --git
a/be/test/storage/segment/segment_iterator_conjunct_short_circuit_test.cpp
b/be/test/storage/segment/segment_iterator_conjunct_short_circuit_test.cpp
new file mode 100644
index 00000000000..eaf745934b2
--- /dev/null
+++ b/be/test/storage/segment/segment_iterator_conjunct_short_circuit_test.cpp
@@ -0,0 +1,327 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// White-box tests for progressive intersection and short-circuit of
pushed-down
+// conjuncts in SegmentIterator::_apply_index_expr. Once an earlier conjunct's
+// index result empties _row_bitmap, the remaining (potentially expensive, e.g.
+// MATCH_PHRASE_PREFIX) conjuncts must not be evaluated against the index at
+// all; skipped conjuncts stay pushed down so the row-level path keeps their
+// semantics over the (now empty) candidate set. Uses the established
+// `#define private public` convention of segment_iterator_limit_opt_test.cpp.
+#include <gtest/gtest.h>
+
+#include <cstdint>
+#include <memory>
+#include <vector>
+
+#include "common/status.h"
+#include "core/data_type/data_type_number.h"
+#include "exprs/vexpr.h"
+#include "exprs/vexpr_context.h"
+#include "runtime/runtime_state.h"
+#include "storage/index/ann/ann_topn_runtime.h"
+#include "storage/index/inverted/inverted_index_reader.h"
+#include "storage/olap_common.h"
+#include "storage/tablet/tablet_schema.h"
+
+#if defined(__clang__)
+#pragma clang diagnostic push
+#pragma clang diagnostic ignored "-Wkeyword-macro"
+#endif
+#define private public
+#define protected public
+#include "storage/segment/segment.h"
+#include "storage/segment/segment_iterator.h"
+#undef private
+#undef protected
+#if defined(__clang__)
+#pragma clang diagnostic pop
+#endif
+
+namespace doris::segment_v2 {
+
+namespace {
+
+// A test VExpr that counts evaluations and registers a fixed-cardinality index
+// result bitmap, mimicking how real match exprs publish results.
+class BitmapEvalExpr : public VExpr {
+public:
+ explicit BitmapEvalExpr(std::vector<uint32_t> rows) :
_rows(std::move(rows)) {
+ _data_type = std::make_shared<DataTypeUInt8>();
+ }
+
+ const std::string& expr_name() const override {
+ static const std::string kName = "BitmapEvalExpr";
+ return kName;
+ }
+
+ Status execute(VExprContext*, Block*, int*) const override { return
Status::OK(); }
+
+ Status execute_column_impl(VExprContext* context, const Block* block,
const Selector* selector,
+ size_t count, ColumnPtr& result_column) const
override {
+ return Status::OK();
+ }
+
+ Status evaluate_inverted_index(VExprContext* context, uint32_t
segment_num_rows) override {
+ ++_eval_count;
+ auto data = std::make_shared<roaring::Roaring>();
+ for (uint32_t row : _rows) {
+ data->add(row);
+ }
+ InvertedIndexResultBitmap result(std::move(data),
std::make_shared<roaring::Roaring>());
+ context->get_index_context()->set_index_result_for_expr(this,
std::move(result));
+ return Status::OK();
+ }
+
+ Status evaluate_ann_range_search(
+ const segment_v2::AnnRangeSearchRuntime&,
+ const std::vector<std::unique_ptr<segment_v2::IndexIterator>>&,
+ const std::vector<std::unique_ptr<segment_v2::ColumnIterator>>&,
size_t,
+ roaring::Roaring&, segment_v2::AnnIndexStats&, bool,
+ AnnRangeSearchEvaluationResult& result) override {
+ ++_ann_eval_count;
+ result.executed = false;
+ return Status::OK();
+ }
+
+ int eval_count() const { return _eval_count; }
+ int ann_eval_count() const { return _ann_eval_count; }
+
+private:
+ std::vector<uint32_t> _rows;
+ int _eval_count = 0;
+ int _ann_eval_count = 0;
+};
+
+TabletSchemaSPtr make_tablet_schema() {
+ TabletSchemaPB schema_pb;
+ schema_pb.set_keys_type(KeysType::DUP_KEYS);
+ auto* col = schema_pb.add_column();
+ col->set_unique_id(0);
+ col->set_name("k0");
+ col->set_type("INT");
+ col->set_is_key(true);
+ col->set_is_nullable(false);
+ auto tablet_schema = std::make_shared<TabletSchema>();
+ tablet_schema->init_from_pb(schema_pb);
+ return tablet_schema;
+}
+
+// Minimal Segment stub: _apply_index_expr only consults num_rows().
+std::shared_ptr<Segment> make_stub_segment(uint32_t num_rows,
+ const TabletSchemaSPtr&
tablet_schema) {
+ auto seg = std::make_shared<Segment>(0, RowsetId(), tablet_schema,
InvertedIndexFileInfo());
+ seg->_num_rows = num_rows;
+ return seg;
+}
+
+// VExprContext with a BitmapEvalExpr root and an index context to publish
into.
+VExprContextSPtr make_bitmap_ctx(const std::shared_ptr<BitmapEvalExpr>& expr) {
+ auto ctx = std::make_shared<VExprContext>(expr);
+ std::vector<std::unique_ptr<IndexIterator>> index_iters;
+ std::vector<IndexFieldNameAndTypePair> storage_types;
+ std::unordered_map<ColumnId, std::unordered_map<const VExpr*, bool>>
status_map;
+ ColumnIteratorOptions column_iter_opts;
+ auto index_ctx = std::make_shared<IndexExecContext>(index_iters,
storage_types, status_map,
+ nullptr, nullptr,
column_iter_opts);
+ ctx->set_index_context(index_ctx);
+ return ctx;
+}
+
+} // namespace
+
+class SegmentIteratorConjunctShortCircuitTest : public testing::Test {
+protected:
+ void SetUp() override {
+ _tablet_schema = make_tablet_schema();
+ _segment = make_stub_segment(100, _tablet_schema);
+ _read_schema = std::make_shared<ReadSchema>(_tablet_schema->columns());
+ _iter = std::make_unique<SegmentIterator>(_segment, _read_schema);
+
+ TQueryOptions query_options;
+ query_options.__set_enable_fallback_on_missing_inverted_index(true);
+ _runtime_state.set_query_options(query_options);
+
+ _iter->_opts.runtime_state = &_runtime_state;
+ _iter->_opts.stats = &_stats;
+ }
+
+ std::shared_ptr<Segment> _segment;
+ std::shared_ptr<TabletSchema> _tablet_schema;
+ ReadSchemaSPtr _read_schema;
+ std::unique_ptr<SegmentIterator> _iter;
+ RuntimeState _runtime_state;
+ OlapReaderStatistics _stats;
+};
+
+// Once an earlier conjunct's index result empties the row bitmap, remaining
+// conjuncts must not be evaluated against the index. With zero surviving
+// rows every remaining conjunct is trivially satisfied, so the whole
+// pushed-down list is consumed -- mirroring the column-predicate path and
+// keeping both the "all conditions consumed" contract (debug point
+// segment_iterator.apply_inverted_index) and the condition-cache digest
+// intact.
+TEST_F(SegmentIteratorConjunctShortCircuitTest,
empty_bitmap_short_circuits_remaining) {
+ _iter->_row_bitmap.addRange(0, 100);
+
+ auto empty_expr = std::make_shared<BitmapEvalExpr>(std::vector<uint32_t>
{});
+ auto expensive_expr =
std::make_shared<BitmapEvalExpr>(std::vector<uint32_t> {0, 1, 2, 3, 4});
+ auto empty_ctx = make_bitmap_ctx(empty_expr);
+ auto expensive_ctx = make_bitmap_ctx(expensive_expr);
+ _iter->_common_expr_ctxs_push_down = {empty_ctx, expensive_ctx};
+
+ ASSERT_TRUE(_iter->_apply_index_expr().ok());
+
+ // The empty result is intersected as soon as it is produced ...
+ EXPECT_TRUE(_iter->_row_bitmap.isEmpty());
+ // ... so the expensive conjunct is never evaluated against the index ...
+ EXPECT_EQ(empty_expr->eval_count(), 1);
+ EXPECT_EQ(expensive_expr->eval_count(), 0);
+ // ... and the whole list is consumed: zero surviving rows satisfy every
+ // remaining conjunct, exactly like the column-predicate short circuit.
+ EXPECT_TRUE(_iter->_common_expr_ctxs_push_down.empty());
+ // The skip is visible in reader statistics (profile:
+ // InvertedIndexConjunctsShortCircuited).
+ EXPECT_EQ(_stats.inverted_index_conjuncts_short_circuited, 1);
+}
+
+// Once the candidate bitmap is empty, virtual-column MATCH projections must
+// not run their whole-segment index evaluation either: with zero surviving
+// rows their result column is never materialized, so the postings walk is
+// pure waste (SELECT msg MATCH_PHRASE_PREFIX '...' WHERE ns MATCH 'no-hit').
+TEST_F(SegmentIteratorConjunctShortCircuitTest,
empty_bitmap_skips_virtual_column_projections) {
+ _iter->_row_bitmap.addRange(0, 100);
+
+ auto empty_expr = std::make_shared<BitmapEvalExpr>(std::vector<uint32_t>
{});
+ auto projection_expr =
std::make_shared<BitmapEvalExpr>(std::vector<uint32_t> {1, 2});
+ _iter->_common_expr_ctxs_push_down = {make_bitmap_ctx(empty_expr)};
+ _iter->_virtual_column_exprs[0] = make_bitmap_ctx(projection_expr);
+
+ ASSERT_TRUE(_iter->_apply_index_expr().ok());
+
+ EXPECT_TRUE(_iter->_row_bitmap.isEmpty());
+ EXPECT_EQ(projection_expr->eval_count(), 0);
+}
+
+// The ANN range-search pass iterates the same conjunct list; on an empty
+// bitmap a range search cannot produce rows, so with the small-candidate
+// fallback thresholds disabled it must still not load or search the index.
+TEST_F(SegmentIteratorConjunctShortCircuitTest,
empty_bitmap_skips_ann_range_search) {
+ _iter->_row_bitmap.addRange(0, 100);
+
+ auto empty_expr = std::make_shared<BitmapEvalExpr>(std::vector<uint32_t>
{});
+ auto ann_expr = std::make_shared<BitmapEvalExpr>(std::vector<uint32_t> {1,
2});
+ // The short circuit keeps the second conjunct away from the inverted-index
+ // loop; only the ANN pass below it iterates the (uncleared) list.
+ _iter->_common_expr_ctxs_push_down = {make_bitmap_ctx(empty_expr),
make_bitmap_ctx(ann_expr)};
+
+ ASSERT_TRUE(_iter->_apply_index_expr().ok());
+
+ EXPECT_TRUE(_iter->_row_bitmap.isEmpty());
+ EXPECT_EQ(ann_expr->ann_eval_count(), 0);
+}
+
+// An indexed prefix that empties the bitmap proves the WHOLE conjunction
+// false, which is exactly what the condition cache wants to remember. The
+// consumed-list clear must therefore keep the digest alive (an empty result
+// is correct for the full conjunction) instead of letting the empty list
+// zero it, which would silently drop the all-false cache entry that repeated
+// scans relied on before the short circuit existed.
+TEST_F(SegmentIteratorConjunctShortCircuitTest,
exhausted_prefix_preserves_condition_cache_digest) {
+ _iter->_row_bitmap.addRange(0, 100);
+ _iter->_opts.condition_cache_digest = 12345;
+ TQueryOptions query_options;
+ query_options.__set_enable_fallback_on_missing_inverted_index(true);
+ query_options.__set_enable_inverted_index_query(true);
+ _runtime_state.set_query_options(query_options);
+
+ auto empty_expr = std::make_shared<BitmapEvalExpr>(std::vector<uint32_t>
{});
+ auto residual_expr =
std::make_shared<BitmapEvalExpr>(std::vector<uint32_t> {0, 1});
+ _iter->_common_expr_ctxs_push_down = {make_bitmap_ctx(empty_expr),
+ make_bitmap_ctx(residual_expr)};
+
+ ASSERT_TRUE(_iter->_get_row_ranges_by_column_conditions().ok());
+
+ EXPECT_TRUE(_iter->_row_bitmap.isEmpty());
+ EXPECT_TRUE(_iter->_common_expr_ctxs_push_down.empty());
+ EXPECT_EQ(_iter->_opts.condition_cache_digest, 12345)
+ << "the all-false result is valid for the full conjunction and
must stay cacheable";
+}
+
+// Counter-case guarding against an over-wide digest fix: when every conjunct
+// is consumed normally (no short circuit, surviving rows remain), the digest
+// still zeroes exactly as before -- the preserved digest is only for the
+// proved-empty conjunction.
+TEST_F(SegmentIteratorConjunctShortCircuitTest,
fully_consumed_conjuncts_still_zero_digest) {
+ _iter->_row_bitmap.addRange(0, 100);
+ _iter->_opts.condition_cache_digest = 12345;
+ TQueryOptions query_options;
+ query_options.__set_enable_fallback_on_missing_inverted_index(true);
+ query_options.__set_enable_inverted_index_query(true);
+ _runtime_state.set_query_options(query_options);
+
+ auto first_expr = std::make_shared<BitmapEvalExpr>(std::vector<uint32_t>
{1, 2, 3});
+ auto second_expr = std::make_shared<BitmapEvalExpr>(std::vector<uint32_t>
{2, 3, 4});
+ _iter->_common_expr_ctxs_push_down = {make_bitmap_ctx(first_expr),
+ make_bitmap_ctx(second_expr)};
+
+ ASSERT_TRUE(_iter->_get_row_ranges_by_column_conditions().ok());
+
+ EXPECT_EQ(_iter->_row_bitmap.cardinality(), 2);
+ EXPECT_TRUE(_iter->_common_expr_ctxs_push_down.empty());
+ EXPECT_EQ(_iter->_opts.condition_cache_digest, 0);
+}
+
+// ANN TopN falls back before touching the index once the candidate bitmap is
+// empty: there is nothing to select, and the veto that residual conjuncts
+// used to provide is gone after the consumed-list clear. The fixture leaves
+// _index_iterators empty, so reaching past the guard would index out of
+// bounds -- passing proves the guard runs before any ANN state is touched.
+TEST_F(SegmentIteratorConjunctShortCircuitTest, empty_bitmap_skips_ann_topn) {
+ auto order_expr = std::make_shared<BitmapEvalExpr>(std::vector<uint32_t>
{});
+ _iter->_ann_topn_runtime =
+ std::make_shared<AnnTopNRuntime>(true, 10,
make_bitmap_ctx(order_expr));
+
+ ASSERT_TRUE(_iter->_apply_ann_topn_predicate().ok());
+}
+
+// A conjunct whose index result does not empty the bitmap must not stop
+// evaluation of the following conjuncts; consumed results are intersected
+// progressively.
+TEST_F(SegmentIteratorConjunctShortCircuitTest,
progressive_intersection_keeps_going) {
+ _iter->_row_bitmap.addRange(0, 100);
+
+ auto first_expr = std::make_shared<BitmapEvalExpr>(std::vector<uint32_t>
{1, 2, 3, 50, 60});
+ auto second_expr = std::make_shared<BitmapEvalExpr>(std::vector<uint32_t>
{2, 3, 4});
+ auto first_ctx = make_bitmap_ctx(first_expr);
+ auto second_ctx = make_bitmap_ctx(second_expr);
+ _iter->_common_expr_ctxs_push_down = {first_ctx, second_ctx};
+
+ ASSERT_TRUE(_iter->_apply_index_expr().ok());
+
+ EXPECT_EQ(first_expr->eval_count(), 1);
+ EXPECT_EQ(second_expr->eval_count(), 1);
+ // {1,2,3,50,60} intersected with {2,3,4} is {2,3}.
+ EXPECT_EQ(_iter->_row_bitmap.cardinality(), 2);
+ EXPECT_TRUE(_iter->_row_bitmap.contains(2));
+ EXPECT_TRUE(_iter->_row_bitmap.contains(3));
+ // Both conjuncts were fully consumed by the index; nothing was skipped.
+ EXPECT_TRUE(_iter->_common_expr_ctxs_push_down.empty());
+ EXPECT_EQ(_stats.inverted_index_conjuncts_short_circuited, 0);
+}
+
+} // namespace doris::segment_v2
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
