This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-4.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-4.1 by this push:
new 5fc51d21256 branch-4.1: [fix](inverted index) Fix score() over
MATCH_PHRASE_PREFIX failing with "UnionTermIterator does not support scoring"
(#69699)
5fc51d21256 is described below
commit 5fc51d21256d598dc89afe5732550b62baeac4a9
Author: Jack <[email protected]>
AuthorDate: Wed Oct 28 18:34:27 2026 +0800
branch-4.1: [fix](inverted index) Fix score() over MATCH_PHRASE_PREFIX
failing with "UnionTermIterator does not support scoring" (#69699)
### What problem does this PR solve?
Issue Number: None
Related PR: #60331
Problem Summary:
SELECT score() FROM t WHERE c1 MATCH_PHRASE_PREFIX 'apple r' ORDER BY
score()
DESC LIMIT 10 fails with "[E3] UnionTermIterator does not support
scoring",
while the same filter without score() works.
The last term of a phrase prefix is a UnionTermIterator over the terms
the
prefix expands to. PhraseQuery sorts its iterators by document frequency
and
scores a row with the norm of the first one. When the terms the prefix
expands
to are rarer than another term of the phrase in a segment, the union
comes
first and UnionTermIterator::norm() threw. A union's document frequency
is the
sum of its terms', so whether a query fails depends on the data of each
segment. On a table holding "apple red", "apple green" and "banana
yellow",
`apple r` (red in 1 row, apple in 2) and `apple g` fail; on a table
where
`r*` matches more rows than apple the same query works. This regressed
when
the phrase-level BM25 scoring of #60331 (#61243 in 4.0) started reading
the
norm of the first iterator.
A norm belongs to the document, not to a term, so the sub-iterator
positioned
on the row, the top of the union's queue, reports it. The union now
returns
that norm.
Result on a branch-4.1 cluster built with this change: the reported
query
returns 0.4700036 and `apple g` returns a score; on a table where
branch-4.1 (4.1.5) threw for `apple gr`, the query now returns the row
that
the same filter without score() returns.
This is not a cherry-pick: master replaced the phrase query engine, and
the
same query there scores the prefix on the tablet idf of the terms it
expands to
(#69698).
### Release note
Fix score() over MATCH_PHRASE_PREFIX failing with "UnionTermIterator
does not
support scoring" when the prefix matches fewer rows than the other
terms.
### Check List (For Author)
- Test: Regression test, Unit Test
- Regression test test_match_phrase_prefix_score on a local branch-4.1
cluster.
- BE unit tests UnionTermIteratorTest.* and PhraseQueryTest.* pass,
including the new NormIsReadFromTheSubIteratorOnTheDocument.
- Behavior changed: Yes (the query no longer fails)
- Does this need documentation: No
### Notes for reviewers
The score this branch returns is still the phrase scored on the idf of
its
plain terms (`apple r` returns 0.4700036); it does not count the terms
the
prefix expands to. That is left as it was, because it needs the
statistics and
engine changes of master.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Sonnet 5.5 <[email protected]>
---
.../index/inverted/util/union_term_iterator.h | 7 +-
.../query/phrase_query/phrase_freq_test.cpp | 5 +-
.../inverted/util/union_term_iterator_test.cpp | 14 ++++
.../test_match_phrase_prefix_score.out | 29 ++++++++
.../test_match_phrase_prefix_score.groovy | 78 ++++++++++++++++++++++
5 files changed, 127 insertions(+), 6 deletions(-)
diff --git a/be/src/storage/index/inverted/util/union_term_iterator.h
b/be/src/storage/index/inverted/util/union_term_iterator.h
index aedb3fda185..1d3764874bc 100644
--- a/be/src/storage/index/inverted/util/union_term_iterator.h
+++ b/be/src/storage/index/inverted/util/union_term_iterator.h
@@ -142,10 +142,9 @@ public:
int32_t doc_freq() const { return _cost; }
- int32_t norm() const {
- throw Exception(ErrorCode::NOT_IMPLEMENTED_ERROR,
- "UnionTermIterator does not support scoring");
- }
+ // The norm belongs to the document, so the sub-iterator positioned on it
reports the same one
+ // as any other.
+ int32_t norm() const { return _docs_queue->top()->norm(); }
private:
int32_t _cost = 0;
diff --git
a/be/test/storage/index/inverted/query/phrase_query/phrase_freq_test.cpp
b/be/test/storage/index/inverted/query/phrase_query/phrase_freq_test.cpp
index e79188ab688..34273320070 100644
--- a/be/test/storage/index/inverted/query/phrase_query/phrase_freq_test.cpp
+++ b/be/test/storage/index/inverted/query/phrase_query/phrase_freq_test.cpp
@@ -237,7 +237,8 @@ TEST_F(PhraseFreqTest, NormVisitor_WithDISI) {
EXPECT_EQ(norm, 1);
}
-TEST_F(PhraseFreqTest, UnionTermIterator_NormThrowsException) {
+// A phrase prefix scores a row with the norm of its union, which the
sub-iterator on the row gives.
+TEST_F(PhraseFreqTest, UnionTermIterator_NormIsThatOfTheSubOnTheDocument) {
auto mock1 = std::make_shared<MockIterator>();
mock1->set_postings({{1, {0}}});
auto mock2 = std::make_shared<MockIterator>();
@@ -246,7 +247,7 @@ TEST_F(PhraseFreqTest,
UnionTermIterator_NormThrowsException) {
std::vector<std::shared_ptr<MockIterator>> subs = {mock1, mock2};
UnionTermIterator<MockIterator> union_iter(subs);
- EXPECT_THROW(union_iter.norm(), Exception);
+ EXPECT_EQ(union_iter.norm(), 1);
}
TEST_F(PhraseFreqTest, ExactPhraseMatcher_MultipleDocuments) {
diff --git a/be/test/storage/index/inverted/util/union_term_iterator_test.cpp
b/be/test/storage/index/inverted/util/union_term_iterator_test.cpp
index 8d210462206..d2c6c9c81fb 100644
--- a/be/test/storage/index/inverted/util/union_term_iterator_test.cpp
+++ b/be/test/storage/index/inverted/util/union_term_iterator_test.cpp
@@ -227,4 +227,18 @@ TEST(UnionTermIteratorTest, PositionExhaustionAndError) {
EXPECT_EQ(uti.freq(), 0);
}
+// A phrase prefix scores a document through the union of the terms its prefix
expands to.
+TEST(UnionTermIteratorTest, NormIsReadFromTheSubIteratorOnTheDocument) {
+ MockIterPtr it1 = std::make_shared<MockIterator>();
+ MockIterPtr it2 = std::make_shared<MockIterator>();
+ it1->set_postings({{10, {1}}, {30, {4}}});
+ it2->set_postings({{20, {2}}, {30, {3}}});
+
+ UnionTermIterator<MockIterator> uti({it1, it2});
+ for (int32_t doc : {10, 20, 30}) {
+ ASSERT_EQ(uti.advance(doc), doc);
+ EXPECT_EQ(uti.norm(), 1) << "doc=" << doc;
+ }
+}
+
} // namespace doris::segment_v2
\ No newline at end of file
diff --git
a/regression-test/data/inverted_index_p0/test_match_phrase_prefix_score.out
b/regression-test/data/inverted_index_p0/test_match_phrase_prefix_score.out
new file mode 100644
index 00000000000..a2d33a43922
--- /dev/null
+++ b/regression-test/data/inverted_index_p0/test_match_phrase_prefix_score.out
@@ -0,0 +1,29 @@
+-- This file is automatically generated. You should know what you did if you
want to edit this
+-- !v2_apple_r --
+1 0.3544
+4 0.3008
+5 0.3008
+
+-- !v3_apple_r --
+1 0.3544
+4 0.3008
+5 0.3008
+
+-- !v2_apple_gr --
+2 0.3544
+7 0.3544
+
+-- !v3_apple_gr --
+2 0.3544
+7 0.3544
+
+-- !v2_apple_red_g --
+4 0.9415
+
+-- !v3_apple_red_g --
+4 0.9415
+
+-- !v3_rows --
+2
+7
+
diff --git
a/regression-test/suites/inverted_index_p0/test_match_phrase_prefix_score.groovy
b/regression-test/suites/inverted_index_p0/test_match_phrase_prefix_score.groovy
new file mode 100644
index 00000000000..eb06d61d971
--- /dev/null
+++
b/regression-test/suites/inverted_index_p0/test_match_phrase_prefix_score.groovy
@@ -0,0 +1,78 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// score() over MATCH_PHRASE_PREFIX must not fail. When the terms the prefix
expands to were rarer
+// than the other terms of the phrase, the union of them led the phrase and
the scorer read its
+// norm, which threw "UnionTermIterator does not support scoring".
+suite("test_match_phrase_prefix_score", "p0") {
+ // Scoring requires expression pushdown, which fuzzy sessions can disable.
+ sql "set enable_common_expr_pushdown = true"
+
+ sql "DROP TABLE IF EXISTS test_match_phrase_prefix_score_v2"
+ sql "DROP TABLE IF EXISTS test_match_phrase_prefix_score_v3"
+
+ sql """
+ CREATE TABLE test_match_phrase_prefix_score_v2 (
+ id INT NULL,
+ c1 TEXT NULL,
+ INDEX idx_c1 (c1) USING INVERTED PROPERTIES("lower_case" = "true",
"parser" = "english", "support_phrase" = "true")
+ ) ENGINE=OLAP
+ DUPLICATE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 1
+ PROPERTIES ("replication_allocation" = "tag.location.default: 1",
"inverted_index_storage_format" = "V2");
+ """
+ sql """
+ CREATE TABLE test_match_phrase_prefix_score_v3 (
+ id INT NULL,
+ c1 TEXT NULL,
+ INDEX idx_c1 (c1) USING INVERTED PROPERTIES("lower_case" = "true",
"parser" = "english", "support_phrase" = "true")
+ ) ENGINE=OLAP
+ DUPLICATE KEY(id)
+ DISTRIBUTED BY HASH(id) BUCKETS 1
+ PROPERTIES ("replication_allocation" = "tag.location.default: 1",
"inverted_index_storage_format" = "V3");
+ """
+
+ // One insert per table keeps every table to a single segment.
+ def rows = """(1, 'apple red'), (2, 'apple green'), (3, 'banana yellow'),
(4, 'apple red green'),
+ (5, 'apple apple red'), (6, 'red red red'), (7, 'apple
gray'), (8, 'apple garden path')"""
+ sql "INSERT INTO test_match_phrase_prefix_score_v2 VALUES ${rows}"
+ sql "INSERT INTO test_match_phrase_prefix_score_v3 VALUES ${rows}"
+ sql "sync"
+
+ // Compare scores as strings so numeric rounding cannot accept zero for
scores below 0.5.
+ // 'apple r': the reported query, 'red' is rarer than 'apple'.
+ order_qt_v2_apple_r """ SELECT id, cast(round(score(), 4) as string) s
FROM test_match_phrase_prefix_score_v2
+ WHERE c1 MATCH_PHRASE_PREFIX 'apple r' ORDER BY
score() DESC LIMIT 10 """
+ order_qt_v3_apple_r """ SELECT id, cast(round(score(), 4) as string) s
FROM test_match_phrase_prefix_score_v3
+ WHERE c1 MATCH_PHRASE_PREFIX 'apple r' ORDER BY
score() DESC LIMIT 10 """
+
+ // 'apple gr' expands to green and gray, both rarer than apple.
+ order_qt_v2_apple_gr """ SELECT id, cast(round(score(), 4) as string) s
FROM test_match_phrase_prefix_score_v2
+ WHERE c1 MATCH_PHRASE_PREFIX 'apple gr' ORDER BY
score() DESC LIMIT 10 """
+ order_qt_v3_apple_gr """ SELECT id, cast(round(score(), 4) as string) s
FROM test_match_phrase_prefix_score_v3
+ WHERE c1 MATCH_PHRASE_PREFIX 'apple gr' ORDER BY
score() DESC LIMIT 10 """
+
+ // Two phrase terms before the prefix.
+ order_qt_v2_apple_red_g """ SELECT id, cast(round(score(), 4) as string) s
FROM test_match_phrase_prefix_score_v2
+ WHERE c1 MATCH_PHRASE_PREFIX 'apple red g'
ORDER BY score() DESC LIMIT 10 """
+ order_qt_v3_apple_red_g """ SELECT id, cast(round(score(), 4) as string) s
FROM test_match_phrase_prefix_score_v3
+ WHERE c1 MATCH_PHRASE_PREFIX 'apple red g'
ORDER BY score() DESC LIMIT 10 """
+
+ // The same rows without score().
+ order_qt_v3_rows """ SELECT id FROM test_match_phrase_prefix_score_v3
+ WHERE c1 MATCH_PHRASE_PREFIX 'apple gr' """
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
