This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-4.2 by this push:
new e4263cbafc1 branch-4.2: [fix](be) Score MATCH on keyword inverted
indexes like SEARCH #69790 (#70008)
e4263cbafc1 is described below
commit e4263cbafc1418914b9ebd83aa276774adb2ec15
Author: Jack <[email protected]>
AuthorDate: Thu Nov 5 11:28:55 2026 +0800
branch-4.2: [fix](be) Score MATCH on keyword inverted indexes like SEARCH
#69790 (#70008)
Backport of #69790 to branch-4.2.
### What problem does this PR solve?
Issue Number: None
Related PR: #69790, #69051
Problem Summary: score() of a MATCH on an untokenized (keyword) inverted
index is always 0, while SEARCH on the same field scores BM25:
INDEX idx_redirect (redirect) USING INVERTED
-- one row ('BP')
SELECT score() FROM t WHERE redirect MATCH 'BP'; -- 0
SELECT score() FROM t WHERE SEARCH('redirect:BP'); -- 0.2876821
Elasticsearch scores match, term and match_phrase on a keyword field
with BM25 in query context, so the score is the idf of the whole value.
#69051 (already on 4.2) made the engine score keyword postings but kept
MATCH out. This makes a keyword index eligible for MATCH scoring and
collects its whole value as one term, the one the reader looks up. It
also carries the follow-up fixes from #69790's review:
- A keyword MATCH_PHRASE drops its ` ~N` slop before collecting.
- A MATCH on a non-string leaf such as `CAST(n AS STRING)` collects
nothing.
- A VARIANT path that a segment indexed as BKD, or does not hold, adds
no statistics. This covers the BKD searcher coming from the searcher
cache, and SNII.
- A SNII index kind is looked up without copying the directory.
### Branch adaptations
- `can_score_match` keeps 4.2's analyzed rule (support_phrase only);
branch-4.2 has no lowercase-normalizer clause.
- `test_omit_norms` does not exist on 4.2, so its `.out` change is
dropped. The branch was cut before #69922 (the 4.2 backport of #69815)
and then merged with branch-4.2 to resolve its conflict in
`index_reader_helper_test.cpp`: both tests are kept, as on master.
- `test_match_keyword_score` inserts its VARIANT rows as JSON strings:
`parse_to_variant` in `VALUES` fails on 4.2 with a max-subcolumns
conversion error. The expected output is master's file, unchanged.
- **#69901 (BM25 statistics per index) is not on 4.2.** A MATCH on a
keyword index and a MATCH on an analyzed index of the same column in one
`score()` query (`name MATCH 'apple' USING ANALYZER english AND name
MATCH 'apple' USING ANALYZER none`) now fails with "Scoring predicates
for one field select different inverted indexes". Master behaved the
same before #69901. Before this change, that query ran with the keyword
clause scoring 0. Each clause alone scores fine. Two analyzed indexes on
one column already fail this way on 4.2. Backporting #69901 lifts this.
### Validation
On this branch merged with branch-4.2 c2aaed99db, ASAN build:
- BE UT: 1611 tests in `IndexReaderHelperTest`, `CollectionStatistics*`
and `*Snii*` pass.
- Regression: `test_match_keyword_score` passes against master's `.out`
on V2/V3/SNII. The following also pass:
`test_match_phrase_prefix_score`, `test_bm25_score`,
`test_score_filter_matrix`, `test_score_filter_usage`,
`test_search_score_topn_predicates`, `test_multi_analyzer_query`,
`test_match_using_analyzer`, `test_single_column_multi_index` (with
debug points on), `test_search_score_keyword_filters`,
`test_search_score_raw_index`, `test_search_score_pattern_constant`.
- Local note: the local thirdparty predates the lance-c upgrade, so
`lance_reader.cpp`'s `lance_scanner_set_scalar_index_auto_selection`
call was stubbed for the local build only. It is not part of this PR.
### Release note
score() of MATCH / MATCH_ALL / MATCH_PHRASE on a keyword (untokenized)
inverted index now returns the BM25 score of the value, the same as
SEARCH, instead of 0.
### Check List (For Author)
- Test: Regression test / Unit Test (see Validation)
- Behavior changed: Yes. Keyword MATCH contributes its BM25 score to
score() instead of 0.
- Does this need documentation: No
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <[email protected]>
---
be/src/storage/index/index_file_reader.cpp | 16 ++
be/src/storage/index/index_file_reader.h | 5 +
be/src/storage/index/index_reader_helper.cpp | 14 +-
.../inverted/reader/collection_statistics.cpp | 37 ++-
.../index/inverted/reader/predicate_collector.cpp | 36 ++-
.../inverted/snii/reader/snii_segment_reader.h | 6 +
.../inverted/reader/collection_statistics_test.cpp | 190 +++++++++++++++-
.../snii_segment_reader_logical_indexes_test.cpp | 9 +
.../storage/segment/index_reader_helper_test.cpp | 13 +-
.../inverted_index_p0/test_match_keyword_score.out | 253 +++++++++++++++++++++
.../search/test_search_score_keyword_filters.out | 97 ++++----
.../test_match_keyword_score.groovy | 146 ++++++++++++
.../test_search_score_keyword_filters.groovy | 3 +-
13 files changed, 750 insertions(+), 75 deletions(-)
diff --git a/be/src/storage/index/index_file_reader.cpp
b/be/src/storage/index/index_file_reader.cpp
index 12ddda2711e..5082151b510 100644
--- a/be/src/storage/index/index_file_reader.cpp
+++ b/be/src/storage/index/index_file_reader.cpp
@@ -402,6 +402,22 @@ IndexFileReader::snii_logical_indexes() const {
return _snii_segment_reader->logical_indexes();
}
+Result<std::optional<doris::snii::format::LogicalIndexKind>>
+IndexFileReader::snii_logical_index_kind(uint64_t index_id, std::string_view
suffix) const {
+ DCHECK(_storage_format == InvertedIndexStorageFormatPB::SNII);
+ std::shared_lock<std::shared_mutex> lock(_mutex);
+ if (_snii_segment_reader == nullptr) {
+ return
ResultError(Status::Error<ErrorCode::INVERTED_INDEX_FILE_NOT_FOUND>(
+ "SNII index file {} is not opened",
+
InvertedIndexDescriptor::get_index_file_path_v2(_index_path_prefix)));
+ }
+ const auto* entry = _snii_segment_reader->find_logical_index(index_id,
suffix);
+ if (entry == nullptr) {
+ return std::nullopt;
+ }
+ return entry->kind;
+}
+
Status IndexFileReader::snii_core_metadata(uint64_t index_id, std::string_view
suffix,
doris::snii::format::CoreMetadata*
out,
const io::IOContext* io_ctx) const {
diff --git a/be/src/storage/index/index_file_reader.h
b/be/src/storage/index/index_file_reader.h
index f08474fd8fe..62fdbe9fd97 100644
--- a/be/src/storage/index/index_file_reader.h
+++ b/be/src/storage/index/index_file_reader.h
@@ -32,6 +32,7 @@
#include <map>
#include <memory>
#include <mutex>
+#include <optional>
#include <shared_mutex>
#include <string>
#include <unordered_map>
@@ -97,6 +98,10 @@ public:
doris::snii::reader::LogicalIndexOpenMode open_mode) const;
// SNII only: every entry of the container directory, text and blob kinds
alike.
Result<std::vector<doris::snii::format::LogicalIndexMetadataRef>>
snii_logical_indexes() const;
+ // SNII only: the kind of one directory entry, nothing when the directory
has no such entry.
+ // A directory can hold thousands of VARIANT paths, so a lookup does not
copy it.
+ Result<std::optional<doris::snii::format::LogicalIndexKind>>
snii_logical_index_kind(
+ uint64_t index_id, std::string_view suffix) const;
// SNII only: reads the CoreMetadata of one text logical index with the
caller's IO context.
Status snii_core_metadata(uint64_t index_id, std::string_view suffix,
doris::snii::format::CoreMetadata* out,
diff --git a/be/src/storage/index/index_reader_helper.cpp
b/be/src/storage/index/index_reader_helper.cpp
index efd92d9b2d3..1a8f92476a2 100644
--- a/be/src/storage/index/index_reader_helper.cpp
+++ b/be/src/storage/index/index_reader_helper.cpp
@@ -24,11 +24,15 @@ namespace {
bool can_score_match(const TabletIndex* index_meta) {
const auto& properties = index_meta->properties();
- // parser=none can inherit support_phrase=true from FE defaults. MATCH
statistics skip
- // these keyword indexes, so their filters must not request a scoring
context either.
- return inverted_index::InvertedIndexAnalyzer::should_analyzer(properties)
&&
- get_parser_phrase_support_string_from_properties(properties) ==
- INVERTED_INDEX_PARSER_PHRASE_SUPPORT_YES;
+ // A keyword index keeps each value as one term, without positions or
norms, and its postings
+ // score with unit frequency and length whatever support_phrase says, as
an Elasticsearch
+ // keyword field does: over tags (x, x, y, z), tag MATCH 'x' scores idf(x)
= ln(2) for both
+ // matching rows, the same as SEARCH('tag:x'). MATCH statistics collect
these terms too.
+ if (!inverted_index::InvertedIndexAnalyzer::should_analyzer(properties)) {
+ return true;
+ }
+ return get_parser_phrase_support_string_from_properties(properties) ==
+ INVERTED_INDEX_PARSER_PHRASE_SUPPORT_YES;
}
} // namespace
diff --git a/be/src/storage/index/inverted/reader/collection_statistics.cpp
b/be/src/storage/index/inverted/reader/collection_statistics.cpp
index b73d80825af..daaefc8c6c1 100644
--- a/be/src/storage/index/inverted/reader/collection_statistics.cpp
+++ b/be/src/storage/index/inverted/reader/collection_statistics.cpp
@@ -73,6 +73,15 @@ Result<std::shared_ptr<lucene::search::IndexSearcher>>
open_optional_clucene_sta
}
return ResultError(status);
}
+ // A segment can index a field with BKD although the tablet schema calls
it text: a VARIANT
+ // path is typed per segment, so CAST(v['n'] AS STRING) MATCH '1' collects
statistics for v.n,
+ // which a segment holding numbers indexed as BKD. That index has no terms
and no segments file
+ // to open, so it adds no statistics, as a segment without the path does;
its rows match
+ // without the index and score nothing.
+ if (compound.value()->fileExists(
+
InvertedIndexDescriptor::get_temporary_bkd_index_data_file_name())) {
+ return std::shared_ptr<lucene::search::IndexSearcher> {};
+ }
lucene::index::IndexReader* reader = nullptr;
try {
reader = lucene::index::IndexReader::open(compound.value().get());
@@ -89,6 +98,20 @@ Result<std::shared_ptr<lucene::search::IndexSearcher>>
open_optional_clucene_sta
return std::make_shared<lucene::search::IndexSearcher>(reader, true);
}
+// Whether a SNII segment has no terms for this field: it indexed the field
with BKD, see
+// open_optional_clucene_stats_searcher, or the field is a VARIANT path this
segment does not hold,
+// as segments of {"a": 1} hold no path b. Any other missing index was not
built, and the caller
+// refuses to score the collection without it.
+Result<bool> snii_field_has_no_terms(const segment_v2::IndexFileReader&
file_reader,
+ const TabletIndex& index) {
+ const auto kind = DORIS_TRY(file_reader.snii_logical_index_kind(
+ static_cast<uint64_t>(index.index_id()),
index.get_index_suffix()));
+ if (!kind.has_value()) {
+ return !index.get_index_suffix().empty();
+ }
+ return *kind == ::doris::snii::format::LogicalIndexKind::kBkd;
+}
+
} // namespace
namespace collection_statistics_detail {
@@ -555,6 +578,11 @@ Status CollectionStatistics::process_segment(const
RowsetSharedPtr& rowset, int6
segment_v2::snii_doris::DorisSniiFileReader::ScopedIOContext
io_context_scope(io_ctx);
SniiScoringSegmentAccumulator segment_accumulator;
for (const auto& [ws_field_name, collect_info] : collect_infos) {
+ const bool no_terms =
+ DORIS_TRY(snii_field_has_no_terms(*idx_file_reader,
*collect_info.index_meta));
+ if (no_terms) {
+ continue;
+ }
auto logical_reader_result =
idx_file_reader->open_snii_index(collect_info.index_meta,
io_ctx);
if (!logical_reader_result.has_value()) {
@@ -665,8 +693,13 @@ Status CollectionStatistics::process_segment(const
RowsetSharedPtr& rowset, int6
auto searcher_variant =
inverted_index_cache_handle.get<CluceneSearcherCacheValue>()->searcher;
- auto index_searcher_ptr =
std::get<FulltextIndexSearcherPtr>(searcher_variant);
- index_searcher = index_searcher_ptr.get();
+ // CAST(v['n'] AS BIGINT) = 1 caches the BKD searcher of v.n under
this key, so a BKD index
+ // can also arrive from the cache: no terms, as in
open_optional_clucene_stats_searcher.
+ const auto* fulltext_searcher =
std::get_if<FulltextIndexSearcherPtr>(&searcher_variant);
+ if (fulltext_searcher == nullptr) {
+ continue;
+ }
+ index_searcher = fulltext_searcher->get();
index_reader = index_searcher->getReader();
#endif
auto source =
segment_v2::clucene_index_source(segment_v2::non_owning_reader(index_reader),
diff --git a/be/src/storage/index/inverted/reader/predicate_collector.cpp
b/be/src/storage/index/inverted/reader/predicate_collector.cpp
index 3d0b4fbbed6..3ab6c785ac6 100644
--- a/be/src/storage/index/inverted/reader/predicate_collector.cpp
+++ b/be/src/storage/index/inverted/reader/predicate_collector.cpp
@@ -34,6 +34,7 @@
#include "storage/index/inverted/analysis/analyzer/analyzer.h"
#include "storage/index/inverted/common/string_helper.h"
#include "storage/index/inverted/query/exec/term_pattern.h"
+#include "storage/index/inverted/query/logical/match_lowering.h"
#include "storage/index/inverted/reader/inverted_index_selector.h"
#include "storage/tablet/tablet_schema.h"
#include "storage/utils.h"
@@ -285,6 +286,32 @@ void preserve_selected_index_metadata(const
ScoringIndexCandidates& candidates,
using namespace scoring_collection_detail;
+namespace {
+
+// The terms a MATCH scores, the ones its reader looks up. A keyword index
stores the whole value
+// as one term, case and spaces included: tag MATCH 'Hello World' scores the
single term
+// "Hello World", as SEARCH('tag:"Hello World"') does. Any other index cuts
the value with the
+// query's analyzer.
+Result<std::vector<TermInfo>> match_scoring_terms(const std::string& value,
+ InvertedIndexQueryType
query_type,
+ const TabletIndex&
index_meta,
+ const
InvertedIndexAnalyzerCtx& analyzer_ctx) {
+ if
(!inverted_index::InvertedIndexAnalyzer::should_analyzer(index_meta.properties()))
{
+ // A keyword phrase reads a trailing " ~N" as its slop, as lower_match
does before the
+ // reader looks the term up: tag MATCH_PHRASE 'BP ~1' looks up BP.
Collecting "BP ~1"
+ // would leave the scorer without statistics for BP and fail the query.
+ std::string_view term = value;
+ if (query_type == InvertedIndexQueryType::MATCH_PHRASE_QUERY) {
+ index_query::logical::take_slop(&term);
+ }
+ return std::vector<TermInfo> {TermInfo(std::string(term))};
+ }
+ DORIS_CHECK(analyzer_ctx.analyzer_provider != nullptr);
+ return analyze_plain_query(value, analyzer_ctx);
+}
+
+} // namespace
+
VSlotRef* PredicateCollector::find_slot_ref(const VExprSPtr& expr) const {
if (!expr) {
return nullptr;
@@ -365,7 +392,10 @@ Status MatchPredicateCollector::collect(RuntimeState*
state, const TabletSchemaS
const auto* index_meta = DORIS_TRY(
select_index_meta(candidates.index_metas, candidates.field_type,
query_type,
analyzer_ctx->analyzer_key,
analyzer_ctx->legacy_analyzer_key));
- if
(!inverted_index::InvertedIndexAnalyzer::should_analyzer(index_meta->properties())
||
+ // Only a text index is read with a score. CAST(n AS STRING) MATCH '1' on
an INT column runs as
+ // a scalar filter, and the index selected for n is a BKD index that holds
no terms: opening it
+ // as a text index to count them would fail the query.
+ if (!is_string_type(candidates.field_type) ||
!IndexReaderHelper::is_need_similarity_score(expr->op(), index_meta)) {
return Status::OK();
}
@@ -375,10 +405,10 @@ Status MatchPredicateCollector::collect(RuntimeState*
state, const TabletSchemaS
return Status::OK();
}
- DORIS_CHECK(analyzer_ctx->analyzer_provider != nullptr);
auto options = DataTypeSerDe::get_default_format_options();
options.timezone = &state->timezone_obj();
- auto term_infos =
DORIS_TRY(analyze_plain_query(right_literal->value(options), *analyzer_ctx));
+ auto term_infos =
DORIS_TRY(match_scoring_terms(right_literal->value(options), query_type,
+ *index_meta,
*analyzer_ctx));
// The prefix of a phrase prefix is scored by the terms it expands to, so
the statistics are
// collected for them per segment; a prefix alone scores a constant and
needs none.
std::optional<TermInfo> scoring_prefix;
diff --git a/be/src/storage/index/inverted/snii/reader/snii_segment_reader.h
b/be/src/storage/index/inverted/snii/reader/snii_segment_reader.h
index c28329d66e3..4c640504489 100644
--- a/be/src/storage/index/inverted/snii/reader/snii_segment_reader.h
+++ b/be/src/storage/index/inverted/snii/reader/snii_segment_reader.h
@@ -115,6 +115,12 @@ public:
return directory_.entries();
}
+ // The directory entry of (index_id, suffix), owned by this reader;
nullptr when it has none.
+ const format::LogicalIndexMetadataRef* find_logical_index(uint64_t
index_id,
+ std::string_view
suffix) const {
+ return directory_.find(index_id, suffix);
+ }
+
Status index_exists(uint64_t index_id, std::string_view suffix, bool*
const exists) const;
// Loads the adjacent Core/STI/DBD group for (index_id, suffix) and builds
a
diff --git
a/be/test/storage/index/inverted/reader/collection_statistics_test.cpp
b/be/test/storage/index/inverted/reader/collection_statistics_test.cpp
index 8eef09fc51b..2f5fc8650c9 100644
--- a/be/test/storage/index/inverted/reader/collection_statistics_test.cpp
+++ b/be/test/storage/index/inverted/reader/collection_statistics_test.cpp
@@ -1581,6 +1581,69 @@ TEST_F(CollectionStatisticsTest,
SniiScoringRejectsMissingLogicalIndexForWholeCo
expect_no_collected_tokens(L"1");
}
+// A VARIANT path can be absent from a segment, as in {"a": 1} next to
{"host": "x"}. Unlike the
+// plain index above, whose absence means it was not built, the segment just
adds no statistics.
+TEST_F(CollectionStatisticsTest,
SniiScoringSkipsVariantPathMissingFromSegment) {
+ TabletSchemaPB schema_pb;
+ schema_pb.set_keys_type(DUP_KEYS);
+
schema_pb.set_inverted_index_storage_format(InvertedIndexStorageFormatPB::SNII);
+ auto tablet_schema = std::make_shared<TabletSchema>();
+ tablet_schema->init_from_pb(schema_pb);
+
+ constexpr int32_t kVariantUid = 9002;
+ TabletColumn variant_col;
+ variant_col.set_unique_id(kVariantUid);
+ variant_col.set_name("v");
+ variant_col.set_type(FieldType::OLAP_FIELD_TYPE_VARIANT);
+ tablet_schema->append_column(variant_col);
+
+ TabletColumn sub_col;
+ sub_col.set_unique_id(-1);
+ sub_col.set_name("v.host");
+ sub_col.set_type(FieldType::OLAP_FIELD_TYPE_STRING);
+ sub_col.set_parent_unique_id(kVariantUid);
+ sub_col.set_path_info(PathInData("v.host"));
+ tablet_schema->append_column(sub_col);
+
+ TabletIndexPB index_pb;
+ index_pb.set_index_id(2002);
+ index_pb.set_index_name("variant_subcolumn_idx");
+ index_pb.set_index_type(IndexType::INVERTED);
+ index_pb.add_col_unique_id(kVariantUid);
+ (*index_pb.mutable_properties())["parser"] = "standard";
+ (*index_pb.mutable_properties())["support_phrase"] = "true";
+ TabletIndex sub_index;
+ sub_index.init_from_pb(index_pb);
+ sub_index.set_escaped_escaped_index_suffix_path(sub_col.suffix_path());
+ tablet_schema->append_index(std::move(sub_index));
+
+ constexpr int kSlotId = 43;
+ runtime_state_->_mock_desc_tbl->add_slot_descriptor(SlotId(kSlotId),
kVariantUid);
+ auto match_expr =
std::make_shared<collection_statistics::MockVExpr>(TExprNodeType::MATCH_PRED);
+ match_expr->_children.push_back(
+ std::make_shared<collection_statistics::MockVSlotRef>("v.host",
SlotId(kSlotId)));
+
match_expr->_children.push_back(std::make_shared<collection_statistics::MockVLiteral>("alpha"));
+ VExprContextSPtrs expr_contexts
{std::make_shared<VExprContext>(match_expr)};
+
+ // The segment holds another logical index only.
+ const std::string segment_path = test_dir_ +
"/snii_missing_variant_path_0.dat";
+ auto write_status = write_snii_scoring_segment(segment_path);
+ ASSERT_TRUE(write_status.ok()) << write_status;
+ auto rowset_meta =
std::make_shared<collection_statistics::MockRowsetMeta>();
+ auto rowset =
std::make_shared<collection_statistics::MockRowset>(tablet_schema, rowset_meta);
+ rowset->set_num_segments(1);
+ rowset->set_segment_path(0, segment_path);
+ auto reader =
std::make_shared<collection_statistics::MockRowsetReader>(rowset);
+ std::vector<RowSetSplits> splits {RowSetSplits(reader)};
+
+ auto status =
+ stats_->collect(runtime_state_.get(), splits, tablet_schema,
expr_contexts, nullptr);
+
+ ASSERT_TRUE(status.ok()) << status.msg();
+
expect_no_collected_tokens(segment_v2::inverted_index::StringHelper::to_wstring(
+ std::to_string(kVariantUid) + ".v.host"));
+}
+
TEST_F(CollectionStatisticsTest, CollectWithMultipleRowsetSplits) {
auto tablet_schema = create_tablet_schema_with_inverted_index();
auto expr_contexts = create_match_expr_contexts();
@@ -2382,10 +2445,9 @@ TEST_F(CollectionStatisticsTest,
CollectNotExtractedColumnSkipsFallback) {
EXPECT_TRUE(collect_infos.empty());
}
-// L1: Index whose properties do not request an analyzer
-// (should_analyzer returns false). The matching index_meta is iterated
-// but skipped before insertion.
-TEST_F(CollectionStatisticsTest, CollectSkipsIndexWithoutAnalyzer) {
+// L1: Index whose properties do not request an analyzer (should_analyzer
returns false). A keyword
+// MATCH scores like SEARCH on the same field, so its whole value is collected
as one term.
+TEST_F(CollectionStatisticsTest, CollectsKeywordIndexValueAsOneTerm) {
auto tablet_schema = std::make_shared<TabletSchema>();
constexpr int32_t kColUid = 1300;
@@ -2412,7 +2474,7 @@ TEST_F(CollectionStatisticsTest,
CollectSkipsIndexWithoutAnalyzer) {
auto match_expr =
std::make_shared<collection_statistics::MockVExpr>(TExprNodeType::MATCH_PRED);
auto slot_ref =
std::make_shared<collection_statistics::MockVSlotRef>("kw", SlotId(kSlotId));
- auto literal = std::make_shared<collection_statistics::MockVLiteral>("v");
+ auto literal =
std::make_shared<collection_statistics::MockVLiteral>("Hello World");
match_expr->_children.push_back(slot_ref);
match_expr->_children.push_back(literal);
auto analyzer_ctx = std::make_shared<InvertedIndexAnalyzerCtx>();
@@ -2420,12 +2482,118 @@ TEST_F(CollectionStatisticsTest,
CollectSkipsIndexWithoutAnalyzer) {
analyzer_ctx->parser_type = InvertedIndexParserType::PARSER_NONE;
match_expr->set_analyzer_ctx(std::move(analyzer_ctx));
- MatchPredicateCollector collector;
- std::unordered_map<std::wstring, CollectInfo> collect_infos;
- auto status =
- collector.collect(runtime_state_.get(), tablet_schema, match_expr,
&collect_infos);
- ASSERT_TRUE(status.ok()) << status.msg();
- EXPECT_TRUE(collect_infos.empty());
+ for (auto op : {TExprOpcode::MATCH_ANY, TExprOpcode::MATCH_ALL,
TExprOpcode::MATCH_PHRASE}) {
+ SCOPED_TRACE(op);
+ match_expr->set_opcode(op);
+ MatchPredicateCollector collector;
+ std::unordered_map<std::wstring, CollectInfo> collect_infos;
+ auto status =
+ collector.collect(runtime_state_.get(), tablet_schema,
match_expr, &collect_infos);
+ ASSERT_TRUE(status.ok()) << status.msg();
+ ASSERT_EQ(collect_infos.size(), 1U);
+ const auto& collect_info = collect_infos.begin()->second;
+ ASSERT_NE(collect_info.index_meta, nullptr);
+ EXPECT_EQ(collect_info.index_meta->index_id(), 2300);
+ EXPECT_EQ(collect_info.unique_terms, std::vector<std::string>({"Hello
World"}));
+ ASSERT_EQ(collect_info.logical_scoring_leaves.size(), 1U);
+ EXPECT_EQ(collect_info.logical_scoring_leaves[0].clauses.size(), 1U);
+ }
+}
+
+// A keyword MATCH_PHRASE reads a trailing " ~N" or " ~N+" as its slop and
looks up the rest, so the
+// statistics must be collected for that term too; any other MATCH looks up
the whole value.
+TEST_F(CollectionStatisticsTest, KeywordPhraseCollectsTermWithoutSlop) {
+ auto tablet_schema = std::make_shared<TabletSchema>();
+
+ constexpr int32_t kColUid = 1301;
+ TabletColumn col;
+ col.set_unique_id(kColUid);
+ col.set_name("kw");
+ col.set_type(FieldType::OLAP_FIELD_TYPE_STRING);
+ tablet_schema->append_column(col);
+
+ TabletIndexPB index_pb;
+ index_pb.set_index_id(2301);
+ index_pb.set_index_name("kw_idx");
+ index_pb.set_index_type(IndexType::INVERTED);
+ index_pb.add_col_unique_id(kColUid);
+ TabletIndex index;
+ index.init_from_pb(index_pb);
+ tablet_schema->append_index(std::move(index));
+
+ constexpr int kSlotId = 81;
+ runtime_state_->_mock_desc_tbl->add_slot_descriptor(SlotId(kSlotId),
kColUid, "kw", {});
+
+ struct Case {
+ TExprOpcode::type op;
+ std::string value;
+ std::string term;
+ };
+ for (const auto& c : {Case {TExprOpcode::MATCH_PHRASE, "Hello World ~1",
"Hello World"},
+ Case {TExprOpcode::MATCH_PHRASE, "Hello World ~2+",
"Hello World"},
+ Case {TExprOpcode::MATCH_ANY, "Hello World ~1",
"Hello World ~1"}}) {
+ SCOPED_TRACE(c.value);
+ auto match_expr =
+
std::make_shared<collection_statistics::MockVExpr>(TExprNodeType::MATCH_PRED);
+ match_expr->_children.push_back(
+ std::make_shared<collection_statistics::MockVSlotRef>("kw",
SlotId(kSlotId)));
+ match_expr->_children.push_back(
+
std::make_shared<collection_statistics::MockVLiteral>(c.value));
+ auto analyzer_ctx = std::make_shared<InvertedIndexAnalyzerCtx>();
+ analyzer_ctx->parser_type = InvertedIndexParserType::PARSER_NONE;
+ match_expr->set_analyzer_ctx(std::move(analyzer_ctx));
+ match_expr->set_opcode(c.op);
+
+ MatchPredicateCollector collector;
+ std::unordered_map<std::wstring, CollectInfo> collect_infos;
+ auto status =
+ collector.collect(runtime_state_.get(), tablet_schema,
match_expr, &collect_infos);
+ ASSERT_TRUE(status.ok()) << status.msg();
+ ASSERT_EQ(collect_infos.size(), 1U);
+ EXPECT_EQ(collect_infos.begin()->second.unique_terms,
std::vector<std::string>({c.term}));
+ }
+}
+
+// CAST(n AS STRING) MATCH '1' on an INT column runs as a scalar filter. The
index selected for n is
+// a BKD index without terms, so the predicate collects nothing instead of
opening it as text.
+TEST_F(CollectionStatisticsTest, MatchOnNumericColumnCollectsNothing) {
+ auto tablet_schema = std::make_shared<TabletSchema>();
+
+ constexpr int32_t kColUid = 1302;
+ TabletColumn col;
+ col.set_unique_id(kColUid);
+ col.set_name("n");
+ col.set_type(FieldType::OLAP_FIELD_TYPE_INT);
+ tablet_schema->append_column(col);
+
+ TabletIndexPB index_pb;
+ index_pb.set_index_id(2302);
+ index_pb.set_index_name("n_idx");
+ index_pb.set_index_type(IndexType::INVERTED);
+ index_pb.add_col_unique_id(kColUid);
+ TabletIndex index;
+ index.init_from_pb(index_pb);
+ tablet_schema->append_index(std::move(index));
+
+ constexpr int kSlotId = 82;
+ runtime_state_->_mock_desc_tbl->add_slot_descriptor(SlotId(kSlotId),
kColUid, "n", {});
+
+ auto match_expr =
std::make_shared<collection_statistics::MockVExpr>(TExprNodeType::MATCH_PRED);
+ match_expr->_children.push_back(
+ std::make_shared<collection_statistics::MockVSlotRef>("n",
SlotId(kSlotId)));
+
match_expr->_children.push_back(std::make_shared<collection_statistics::MockVLiteral>("1"));
+ match_expr->set_analyzer_ctx(std::make_shared<InvertedIndexAnalyzerCtx>());
+
+ for (auto op : {TExprOpcode::MATCH_ANY, TExprOpcode::MATCH_ALL,
TExprOpcode::MATCH_PHRASE}) {
+ SCOPED_TRACE(op);
+ match_expr->set_opcode(op);
+ MatchPredicateCollector collector;
+ std::unordered_map<std::wstring, CollectInfo> collect_infos;
+ auto status =
+ collector.collect(runtime_state_.get(), tablet_schema,
match_expr, &collect_infos);
+ ASSERT_TRUE(status.ok()) << status.msg();
+ EXPECT_TRUE(collect_infos.empty());
+ }
}
TEST_F(CollectionStatisticsTest,
ExplicitNoneDoesNotSelectNormalizerIndexForScoring) {
diff --git
a/be/test/storage/index/inverted/snii/reader/snii_segment_reader_logical_indexes_test.cpp
b/be/test/storage/index/inverted/snii/reader/snii_segment_reader_logical_indexes_test.cpp
index c97e13c5478..7fd61ed5542 100644
---
a/be/test/storage/index/inverted/snii/reader/snii_segment_reader_logical_indexes_test.cpp
+++
b/be/test/storage/index/inverted/snii/reader/snii_segment_reader_logical_indexes_test.cpp
@@ -112,6 +112,15 @@ TEST_F(SniiSegmentReaderLogicalIndexesTest,
ListsEveryLogicalIndex) {
EXPECT_TRUE(saw_positions);
}
+TEST_F(SniiSegmentReaderLogicalIndexesTest,
FindsDirectoryEntryWithoutCopyingIt) {
+ const std::vector<LogicalIndexMetadataRef>& entries =
segment_.logical_indexes();
+ for (const LogicalIndexMetadataRef& entry : entries) {
+ EXPECT_EQ(&entry, segment_.find_logical_index(entry.index_id,
entry.index_suffix));
+ }
+ EXPECT_EQ(nullptr, segment_.find_logical_index(7, "body"));
+ EXPECT_EQ(nullptr, segment_.find_logical_index(8, ""));
+}
+
TEST_F(SniiSegmentReaderLogicalIndexesTest, ReadsCoreMetadataOfEachIndex) {
CoreMetadata docs_only;
ASSERT_TRUE(segment_.core_metadata_for_index(7, "", &docs_only).ok());
diff --git a/be/test/storage/segment/index_reader_helper_test.cpp
b/be/test/storage/segment/index_reader_helper_test.cpp
index 2fb699529ec..5cb3d0e604f 100644
--- a/be/test/storage/segment/index_reader_helper_test.cpp
+++ b/be/test/storage/segment/index_reader_helper_test.cpp
@@ -447,17 +447,20 @@ TEST_F(IndexReaderHelperTest,
RegexpScoresConstantOnEveryIndex) {
}
}
-TEST_F(IndexReaderHelperTest, MatchScoringRequiresAnalyzedPhraseIndex) {
+TEST_F(IndexReaderHelperTest,
MatchScoringRequiresKeywordOrAnalyzedPhraseIndex) {
struct Case {
std::map<std::string, std::string> properties;
bool expected;
};
+ // Keyword indexes score with unit frequency and length, so phrase support
does not matter;
+ // analyzed indexes still need it to score.
for (const auto& test : std::vector<Case> {
- {.properties = {{"support_phrase", "true"}}, .expected =
false},
- {.properties = {{"parser", "none"}, {"support_phrase",
"true"}},
- .expected = false},
+ {.properties = {}, .expected = true},
+ {.properties = {{"support_phrase", "true"}}, .expected =
true},
+ {.properties = {{"parser", "none"}}, .expected = true},
+ {.properties = {{"parser", "none"}, {"support_phrase",
"true"}}, .expected = true},
{.properties = {{"parser", "unknown"}, {"support_phrase",
"true"}},
- .expected = false},
+ .expected = true},
{.properties = {{"normalizer", "lowercase"},
{"support_phrase", "true"}},
.expected = true},
{.properties = {{"parser", "english"}, {"support_phrase",
"true"}},
diff --git
a/regression-test/data/inverted_index_p0/test_match_keyword_score.out
b/regression-test/data/inverted_index_p0/test_match_keyword_score.out
new file mode 100644
index 00000000000..5c3b79d48e2
--- /dev/null
+++ b/regression-test/data/inverted_index_p0/test_match_keyword_score.out
@@ -0,0 +1,253 @@
+-- This file is automatically generated. You should know what you did if you
want to edit this
+-- !V2_single_row_match --
+1 0.287682
+
+-- !V2_match --
+1 0.693147
+2 0.693147
+5 0.693147
+
+-- !V2_match_all --
+1 0.693147
+2 0.693147
+5 0.693147
+
+-- !V2_match_phrase --
+1 0.693147
+2 0.693147
+5 0.693147
+
+-- !V2_match_phrase_slop --
+1 0.693147
+2 0.693147
+5 0.693147
+
+-- !V2_match_phrase_ordered_slop --
+1 0.693147
+2 0.693147
+5 0.693147
+
+-- !V2_search --
+1 0.693147
+2 0.693147
+5 0.693147
+
+-- !V2_none_match --
+1 0.693147
+3 0.693147
+6 0.693147
+
+-- !V2_none_match_phrase --
+1 0.693147
+3 0.693147
+6 0.693147
+
+-- !V2_none_search --
+1 0.693147
+3 0.693147
+6 0.693147
+
+-- !V2_analyzed_and_keyword --
+1 0.983863
+2 0.893357
+5 0.961783
+
+-- !V2_search_and --
+1 0.983863
+2 0.893357
+5 0.961783
+
+-- !V2_analyzed_and_equality --
+1 0.290716
+2 0.20021
+5 0.268636
+
+-- !V2_analyzed_and_numeric_cast --
+1 0.290716
+4 0.268636
+
+-- !V2_variant_numeric_path --
+1 0
+3 0
+4 0.287682
+
+-- !V2_variant_text_path --
+1 0.693147
+4 0.693147
+
+-- !V2_variant_sparse_path --
+1 0.980829
+
+-- !V2_variant_numeric_path_cached_bkd --
+1 0
+3 0
+4 0.287682
+
+-- !V3_single_row_match --
+1 0.287682
+
+-- !V3_match --
+1 0.693147
+2 0.693147
+5 0.693147
+
+-- !V3_match_all --
+1 0.693147
+2 0.693147
+5 0.693147
+
+-- !V3_match_phrase --
+1 0.693147
+2 0.693147
+5 0.693147
+
+-- !V3_match_phrase_slop --
+1 0.693147
+2 0.693147
+5 0.693147
+
+-- !V3_match_phrase_ordered_slop --
+1 0.693147
+2 0.693147
+5 0.693147
+
+-- !V3_search --
+1 0.693147
+2 0.693147
+5 0.693147
+
+-- !V3_none_match --
+1 0.693147
+3 0.693147
+6 0.693147
+
+-- !V3_none_match_phrase --
+1 0.693147
+3 0.693147
+6 0.693147
+
+-- !V3_none_search --
+1 0.693147
+3 0.693147
+6 0.693147
+
+-- !V3_analyzed_and_keyword --
+1 0.983863
+2 0.893357
+5 0.961783
+
+-- !V3_search_and --
+1 0.983863
+2 0.893357
+5 0.961783
+
+-- !V3_analyzed_and_equality --
+1 0.290716
+2 0.20021
+5 0.268636
+
+-- !V3_analyzed_and_numeric_cast --
+1 0.290716
+4 0.268636
+
+-- !V3_variant_numeric_path --
+1 0
+3 0
+4 0.287682
+
+-- !V3_variant_text_path --
+1 0.693147
+4 0.693147
+
+-- !V3_variant_sparse_path --
+1 0.980829
+
+-- !V3_variant_numeric_path_cached_bkd --
+1 0
+3 0
+4 0.287682
+
+-- !SNII_single_row_match --
+1 0.287682
+
+-- !SNII_match --
+1 0.640724
+2 0.640724
+5 0.640724
+
+-- !SNII_match_all --
+1 0.640724
+2 0.640724
+5 0.640724
+
+-- !SNII_match_phrase --
+1 0.640724
+2 0.640724
+5 0.640724
+
+-- !SNII_match_phrase_slop --
+1 0.640724
+2 0.640724
+5 0.640724
+
+-- !SNII_match_phrase_ordered_slop --
+1 0.640724
+2 0.640724
+5 0.640724
+
+-- !SNII_search --
+1 0.640724
+2 0.640724
+5 0.640724
+
+-- !SNII_none_match --
+1 0.640724
+3 0.640724
+6 0.640724
+
+-- !SNII_none_match_phrase --
+1 0.640724
+3 0.640724
+6 0.640724
+
+-- !SNII_none_search --
+1 0.640724
+3 0.640724
+6 0.640724
+
+-- !SNII_analyzed_and_keyword --
+1 0.93144
+2 0.840934
+5 0.90936
+
+-- !SNII_search_and --
+1 0.93144
+2 0.840934
+5 0.90936
+
+-- !SNII_analyzed_and_equality --
+1 0.290716
+2 0.20021
+5 0.268636
+
+-- !SNII_analyzed_and_numeric_cast --
+1 0.290716
+4 0.268636
+
+-- !SNII_variant_numeric_path --
+1 0
+3 0
+4 0.287682
+
+-- !SNII_variant_text_path --
+1 0.693147
+4 0.693147
+
+-- !SNII_variant_sparse_path --
+1 0.980829
+
+-- !SNII_variant_numeric_path_cached_bkd --
+1 0
+3 0
+4 0.287682
+
diff --git a/regression-test/data/search/test_search_score_keyword_filters.out
b/regression-test/data/search/test_search_score_keyword_filters.out
index 5b1d6730b55..8175ca310c2 100644
--- a/regression-test/data/search/test_search_score_keyword_filters.out
+++ b/regression-test/data/search/test_search_score_keyword_filters.out
@@ -12,12 +12,12 @@
4 0.268636 true
-- !V2_false_keyword_match --
-1 0.290716 true
-4 0.268636 true
+1 0.983863 true
+4 0.961783 true
-- !V2_false_keyword_match_all --
-1 0.290716 true
-4 0.268636 true
+1 0.983863 true
+4 0.961783 true
-- !V2_true_internal_and --
1 0.983863 true
@@ -32,12 +32,12 @@
4 0.268636 true
-- !V2_true_keyword_match --
-1 0.290716 true
-4 0.268636 true
+1 0.983863 true
+4 0.961783 true
-- !V2_true_keyword_match_all --
-1 0.290716 true
-4 0.268636 true
+1 0.983863 true
+4 0.961783 true
-- !explicit_none_V2_false_internal_and --
1 0.983863 true
@@ -52,12 +52,12 @@
4 0.268636 true
-- !explicit_none_V2_false_keyword_match --
-1 0.290716 true
-4 0.268636 true
+1 0.983863 true
+4 0.961783 true
-- !explicit_none_V2_false_keyword_match_all --
-1 0.290716 true
-4 0.268636 true
+1 0.983863 true
+4 0.961783 true
-- !explicit_none_V2_true_internal_and --
1 0.983863 true
@@ -72,12 +72,12 @@
4 0.268636 true
-- !explicit_none_V2_true_keyword_match --
-1 0.290716 true
-4 0.268636 true
+1 0.983863 true
+4 0.961783 true
-- !explicit_none_V2_true_keyword_match_all --
-1 0.290716 true
-4 0.268636 true
+1 0.983863 true
+4 0.961783 true
-- !V3_false_internal_and --
1 0.983863 true
@@ -92,12 +92,12 @@
4 0.268636 true
-- !V3_false_keyword_match --
-1 0.290716 true
-4 0.268636 true
+1 0.983863 true
+4 0.961783 true
-- !V3_false_keyword_match_all --
-1 0.290716 true
-4 0.268636 true
+1 0.983863 true
+4 0.961783 true
-- !V3_true_internal_and --
1 0.983863 true
@@ -112,12 +112,12 @@
4 0.268636 true
-- !V3_true_keyword_match --
-1 0.290716 true
-4 0.268636 true
+1 0.983863 true
+4 0.961783 true
-- !V3_true_keyword_match_all --
-1 0.290716 true
-4 0.268636 true
+1 0.983863 true
+4 0.961783 true
-- !explicit_none_V3_false_internal_and --
1 0.983863 true
@@ -132,12 +132,12 @@
4 0.268636 true
-- !explicit_none_V3_false_keyword_match --
-1 0.290716 true
-4 0.268636 true
+1 0.983863 true
+4 0.961783 true
-- !explicit_none_V3_false_keyword_match_all --
-1 0.290716 true
-4 0.268636 true
+1 0.983863 true
+4 0.961783 true
-- !explicit_none_V3_true_internal_and --
1 0.983863 true
@@ -152,12 +152,12 @@
4 0.268636 true
-- !explicit_none_V3_true_keyword_match --
-1 0.290716 true
-4 0.268636 true
+1 0.983863 true
+4 0.961783 true
-- !explicit_none_V3_true_keyword_match_all --
-1 0.290716 true
-4 0.268636 true
+1 0.983863 true
+4 0.961783 true
-- !SNII_false_internal_and --
1 0.93144 true
@@ -172,12 +172,12 @@
4 0.268636 true
-- !SNII_false_keyword_match --
-1 0.290716 true
-4 0.268636 true
+1 0.93144 true
+4 0.90936 true
-- !SNII_false_keyword_match_all --
-1 0.290716 true
-4 0.268636 true
+1 0.93144 true
+4 0.90936 true
-- !SNII_true_internal_and --
1 0.93144 true
@@ -192,12 +192,12 @@
4 0.268636 true
-- !SNII_true_keyword_match --
-1 0.290716 true
-4 0.268636 true
+1 0.93144 true
+4 0.90936 true
-- !SNII_true_keyword_match_all --
-1 0.290716 true
-4 0.268636 true
+1 0.93144 true
+4 0.90936 true
-- !explicit_none_SNII_false_internal_and --
1 0.93144 true
@@ -212,12 +212,12 @@
4 0.268636 true
-- !explicit_none_SNII_false_keyword_match --
-1 0.290716 true
-4 0.268636 true
+1 0.93144 true
+4 0.90936 true
-- !explicit_none_SNII_false_keyword_match_all --
-1 0.290716 true
-4 0.268636 true
+1 0.93144 true
+4 0.90936 true
-- !explicit_none_SNII_true_internal_and --
1 0.93144 true
@@ -232,9 +232,10 @@
4 0.268636 true
-- !explicit_none_SNII_true_keyword_match --
-1 0.290716 true
-4 0.268636 true
+1 0.93144 true
+4 0.90936 true
-- !explicit_none_SNII_true_keyword_match_all --
-1 0.290716 true
-4 0.268636 true
+1 0.93144 true
+4 0.90936 true
+
diff --git
a/regression-test/suites/inverted_index_p0/test_match_keyword_score.groovy
b/regression-test/suites/inverted_index_p0/test_match_keyword_score.groovy
new file mode 100644
index 00000000000..32b5a58ef69
--- /dev/null
+++ b/regression-test/suites/inverted_index_p0/test_match_keyword_score.groovy
@@ -0,0 +1,146 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// score() over MATCH on a keyword (untokenized) inverted index. Like an
Elasticsearch match or term
+// query on a keyword field, it scores BM25 with unit frequency and length, so
every matching row
+// scores the idf of the whole value, the same as SEARCH on that field; it
used to score 0. An
+// equality predicate stays a pure filter.
+suite("test_match_keyword_score", "p0") {
+ // MATCH and SEARCH need index execution even when CI fuzzes these session
defaults.
+ sql "set enable_common_expr_pushdown = true"
+ sql "set enable_inverted_index_query = true"
+ sql "set enable_match_without_inverted_index = true"
+
+ for (def storageFormat : ["V2", "V3", "SNII"]) {
+ sql "DROP TABLE IF EXISTS test_match_keyword_score"
+ sql """
+ CREATE TABLE test_match_keyword_score (
+ id INT NOT NULL,
+ body STRING NULL,
+ tag STRING NULL,
+ tag_none STRING NULL,
+ n INT NULL,
+ INDEX idx_body(body) USING INVERTED
+ PROPERTIES("parser" = "english", "support_phrase" =
"true"),
+ INDEX idx_tag(tag) USING INVERTED PROPERTIES("ignore_above" =
"1024"),
+ INDEX idx_tag_none(tag_none) USING INVERTED
+ PROPERTIES("parser" = "none", "support_phrase" = "true"),
+ INDEX idx_n(n) USING INVERTED
+ ) ENGINE=OLAP
+ DUPLICATE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 1
+ PROPERTIES (
+ "replication_allocation" = "tag.location.default:1",
+ "inverted_index_storage_format" = "${storageFormat}"
+ )
+ """
+ // The first row alone is the reported case: one document scores idf =
ln(4/3) = 0.287682.
+ sql """INSERT INTO test_match_keyword_score VALUES (1, 'apple apple',
'BP', 'Hello World', 1)"""
+ sql "sync"
+ quickTest("${storageFormat}_single_row_match", """
+ SELECT id, ROUND(s, 6) FROM (
+ SELECT id, score() AS s FROM test_match_keyword_score
+ WHERE tag MATCH 'BP' ORDER BY s DESC LIMIT 10
+ ) hits ORDER BY id
+ """, true)
+
+ sql """INSERT INTO test_match_keyword_score VALUES
+ (2, 'apple pear', 'BP', 'hello world', 2), (3, 'pear', 'other',
'Hello World', 1),
+ (4, 'apple', 'x', 'Hello', 1), (5, 'apple', 'BP', NULL, NULL),
+ (6, 'apple', NULL, 'Hello World', 2)
+ """
+ sql "sync"
+
+ // The whole value is one term: MATCH, MATCH_ALL and MATCH_PHRASE
score as SEARCH does.
+ for (def query : [
+ ["match", "tag MATCH 'BP'"],
+ ["match_all", "tag MATCH_ALL 'BP'"],
+ ["match_phrase", "tag MATCH_PHRASE 'BP'"],
+ // A trailing " ~N" is the phrase slop, not part of the term:
it scores BP.
+ ["match_phrase_slop", "tag MATCH_PHRASE 'BP ~1'"],
+ ["match_phrase_ordered_slop", "tag MATCH_PHRASE 'BP ~1+'"],
+ ["search", "SEARCH('tag:BP')"],
+ // Case and spaces are kept, and phrase support inherited by
parser=none is ignored.
+ ["none_match", "tag_none MATCH 'Hello World'"],
+ ["none_match_phrase", "tag_none MATCH_PHRASE 'Hello World'"],
+ ["none_search", "SEARCH('tag_none:\"Hello World\"')"],
+ // Both leaves score: idf(apple) from body plus idf(BP) from
tag.
+ ["analyzed_and_keyword", "body MATCH 'apple' AND tag MATCH
'BP'"],
+ ["search_and", "SEARCH('body:apple AND tag:BP')"],
+ // An equality is a filter: only the body leaf scores.
+ ["analyzed_and_equality", "body MATCH 'apple' AND tag = 'BP'"],
+ // A MATCH on a number cast to text is a filter over the BKD
index of n: only the
+ // body leaf scores.
+ ["analyzed_and_numeric_cast", "body MATCH 'apple' AND CAST(n
AS STRING) MATCH '1'"]]) {
+ quickTest("${storageFormat}_${query[0]}", """
+ SELECT id, ROUND(s, 6) FROM (
+ SELECT id, score() AS s FROM test_match_keyword_score
+ WHERE ${query[1]} ORDER BY s DESC LIMIT 10
+ ) hits ORDER BY id
+ """, true)
+ }
+
+ // A VARIANT path is typed per segment: n holds numbers, indexed as
BKD, in the first
+ // insert and text in the second. The plain VARIANT index makes every
path a keyword
+ // index, so CAST(v['n'] AS STRING) MATCH collects statistics for n;
the BKD segment adds
+ // none instead of failing, and its rows match without the index with
score 0. p exists
+ // only in the first insert: the other segment has no index for it and
adds nothing.
+ sql "set default_variant_enable_typed_paths_to_sparse = false"
+ sql "set default_variant_enable_doc_mode = false"
+ sql "DROP TABLE IF EXISTS test_match_keyword_score_variant"
+ sql """
+ CREATE TABLE test_match_keyword_score_variant (
+ id INT NOT NULL,
+ v VARIANT NULL,
+ INDEX idx_v(v) USING INVERTED
+ ) ENGINE=OLAP
+ DUPLICATE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 1
+ PROPERTIES (
+ "replication_allocation" = "tag.location.default:1",
+ "inverted_index_storage_format" = "${storageFormat}",
+ "disable_auto_compaction" = "true"
+ )
+ """
+ sql """INSERT INTO test_match_keyword_score_variant VALUES
+ (1, '{"n": 1, "s": "abc", "p": "abc"}'), (2, '{"n": 2, "s": "abd",
"p": "abd"}'),
+ (3, '{"n": 1, "s": "xyz", "p": "xyz"}')
+ """
+ sql """INSERT INTO test_match_keyword_score_variant VALUES (4, '{"n":
"1", "s": "abc"}')"""
+ sql "sync"
+ for (def query : [
+ ["variant_numeric_path", "CAST(v['n'] AS STRING) MATCH '1'"],
+ ["variant_text_path", "CAST(v['s'] AS STRING) MATCH 'abc'"],
+ ["variant_sparse_path", "CAST(v['p'] AS STRING) MATCH
'abc'"]]) {
+ quickTest("${storageFormat}_${query[0]}", """
+ SELECT id, ROUND(s, 6) FROM (
+ SELECT id, score() AS s FROM
test_match_keyword_score_variant
+ WHERE ${query[1]} ORDER BY s DESC LIMIT 10
+ ) hits ORDER BY id
+ """, true)
+ }
+ // A numeric predicate leaves the BKD searcher of n in the searcher
cache, where collecting
+ // statistics finds it next: it used to take it for a text searcher
and abort the BE.
+ sql "SELECT COUNT(*) FROM test_match_keyword_score_variant WHERE
CAST(v['n'] AS BIGINT) = 1"
+ quickTest("${storageFormat}_variant_numeric_path_cached_bkd", """
+ SELECT id, ROUND(s, 6) FROM (
+ SELECT id, score() AS s FROM test_match_keyword_score_variant
+ WHERE CAST(v['n'] AS STRING) MATCH '1' ORDER BY s DESC LIMIT 10
+ ) hits ORDER BY id
+ """, true)
+ }
+}
diff --git
a/regression-test/suites/search/test_search_score_keyword_filters.groovy
b/regression-test/suites/search/test_search_score_keyword_filters.groovy
index a1d41ab4861..1e89bcb2bfb 100644
--- a/regression-test/suites/search/test_search_score_keyword_filters.groovy
+++ b/regression-test/suites/search/test_search_score_keyword_filters.groovy
@@ -66,7 +66,8 @@ suite("test_search_score_keyword_filters", "p0") {
["equality", "tag = 'x'"],
["keyword_match", "tag MATCH_ANY 'x'"],
["keyword_match_all", "tag MATCH_ALL 'x'"]]) {
- // The keyword predicate filters rows; only the SEARCH
leaf contributes a score.
+ // IN and equality only filter rows; a keyword MATCH also
adds the idf of its
+ // value, as an Elasticsearch match on a keyword field
does.
quickTest("${variant}${storageFormat}_${wand}_${filter[0]}", """
SELECT id, ROUND(bm25_score, 6), bm25_score > 0 AND
bm25_score < 1000000
FROM (
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
