This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-4.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-4.1 by this push:
new 5b87c1b9885 branch-4.1: [fix](nereids) Do not infer MATCH and SEARCH
predicates across column equalities #69419 (#69564)
5b87c1b9885 is described below
commit 5b87c1b98854128cc1a8b6af2b0ea6a75916ed73
Author: Jack <[email protected]>
AuthorDate: Mon Oct 26 10:02:10 2026 +0800
branch-4.1: [fix](nereids) Do not infer MATCH and SEARCH predicates across
column equalities #69419 (#69564)
### What problem does this PR solve?
Related PR: #69419
Problem Summary: Backport #69419 to `branch-4.1`.
Predicate inference copied a predicate on one column to every column
that is equal to it, also when the predicate contained MATCH,
multi_match or SEARCH. Their result depends on the inverted index of the
column, not only on its value: with `c1 = c2` and different parsers on
the two columns, the copy removed matching rows, and with the same index
it doubled `score()`, so a minimum score let low-scored rows through.
`InferPredicateByReplace.infer` now skips every input that contains
`Match`, `MultiMatch`, `Search` or `SearchExpression`.
#### Branch adaptations
The branch does not have the `NoneMovableFunction` check that master has
next to the new check, so only the new check is added. The test imports
`SearchDslParser` from its package on the branch.
`infer_match_predicate` sets `enable_common_expr_pushdown = true`, as
the score suites of the branch do: CI on branch-4.1 draws it at random
for each session, and without it MATCH is not evaluated with the index
in the scan and `score()` is 0.
### Release note
Fix wrong results and scores when predicate inference copied a MATCH,
multi_match or SEARCH predicate to an equal column.
### Check List (For Author)
- Test
- [x] Regression test
- [x] Unit Test
- [ ] Manual test
- Behavior changed:
- [x] Yes. Predicate inference no longer copies a predicate that
contains MATCH, multi_match or SEARCH to an equal column.
- Does this need documentation?
- [x] No.
### Validation
Validated together with the other two backports (#69399, #69419 and
#69292 on `branch-4.1` at 3f911d030b), in a local ASAN build:
- PASS: FE unit tests `LogicalPlanDeepCopierTest`, `CTEInlineTest`,
`InferPredicateByReplaceTest`, `InferPredicatesTest`,
`UnequalPredicateInferTest`, `BindScoreToOlapScanTest`,
`SearchExpressionTest`, `TopNRuntimeFilterTest` and
`MaterializedViewUtilsTest` (228 tests), and FE Checkstyle.
- PASS: `./build.sh --be` and `./build.sh --fe`.
- PASS: BE unit tests `SegmentIteratorScoreCommonExprTest`,
`SegmentIteratorApplyIndexExprTest` and `VirtualSlotRefTest` (31 tests);
clang-format 16 on the changed BE files.
- PASS: the 16 regression suites of the branch that use `score()`,
including `test_cte_inline_virtual_column`, `infer_match_predicate`,
`test_score_filter_usage` and `test_score_filter_matrix`. The framework
regenerated the `.out` of `test_score_filter_matrix` on this branch: it
drops the 46 SNII cases and keeps the 94 V2 and V3 cases of master
unchanged. The other `.out` files are unchanged.
- PASS: with the fuzzy session variables of CI
(`use_fuzzy_session_variable = true` and its pull request number), the
score suites failed in sessions that drew `enable_common_expr_pushdown =
false`; with the setting pinned, four runs of `test_score_filter_usage`,
`test_score_filter_matrix` and `test_cte_inline_virtual_column`, and
four of `infer_match_predicate`, passed.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <[email protected]>
---
.../rules/rewrite/InferPredicateByReplace.java | 8 ++-
.../rules/rewrite/InferPredicateByReplaceTest.java | 60 +++++++++++++++++
.../infer_predicate/infer_match_predicate.out | 19 ++++++
.../infer_predicate/infer_match_predicate.groovy | 76 ++++++++++++++++++++++
4 files changed, 162 insertions(+), 1 deletion(-)
diff --git
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/InferPredicateByReplace.java
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/InferPredicateByReplace.java
index c2ca99f0b61..f5cd0a18518 100644
---
a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/InferPredicateByReplace.java
+++
b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/InferPredicateByReplace.java
@@ -28,10 +28,14 @@ import org.apache.doris.nereids.trees.expressions.EqualTo;
import org.apache.doris.nereids.trees.expressions.Expression;
import org.apache.doris.nereids.trees.expressions.InPredicate;
import org.apache.doris.nereids.trees.expressions.Like;
+import org.apache.doris.nereids.trees.expressions.Match;
import org.apache.doris.nereids.trees.expressions.Not;
import org.apache.doris.nereids.trees.expressions.Or;
+import org.apache.doris.nereids.trees.expressions.SearchExpression;
import org.apache.doris.nereids.trees.expressions.Slot;
import org.apache.doris.nereids.trees.expressions.functions.ExpressionTrait;
+import org.apache.doris.nereids.trees.expressions.functions.scalar.MultiMatch;
+import org.apache.doris.nereids.trees.expressions.functions.scalar.Search;
import org.apache.doris.nereids.trees.expressions.literal.Literal;
import org.apache.doris.nereids.trees.expressions.visitor.ExpressionVisitor;
import org.apache.doris.nereids.trees.plans.Plan;
@@ -210,7 +214,9 @@ public class InferPredicateByReplace {
}
Map<Expression, Set<Expression>> exprPredicates = new HashMap<>();
for (Expression input : inputs) {
- if (input.anyMatch(expr -> !((ExpressionTrait)
expr).isDeterministic())
+ // MATCH, multi_match and SEARCH depend on column indexes, and
copies can change score().
+ if (input.containsType(Match.class, MultiMatch.class,
Search.class, SearchExpression.class)
+ || input.anyMatch(expr -> !((ExpressionTrait)
expr).isDeterministic())
|| input.getInputSlots().size() != 1) {
continue;
}
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/InferPredicateByReplaceTest.java
b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/InferPredicateByReplaceTest.java
index ad35028d7b9..da8a40645e7 100644
---
a/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/InferPredicateByReplaceTest.java
+++
b/fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/InferPredicateByReplaceTest.java
@@ -24,11 +24,18 @@ import
org.apache.doris.nereids.trees.expressions.GreaterThan;
import org.apache.doris.nereids.trees.expressions.InPredicate;
import org.apache.doris.nereids.trees.expressions.LessThan;
import org.apache.doris.nereids.trees.expressions.Like;
+import org.apache.doris.nereids.trees.expressions.MatchAny;
+import org.apache.doris.nereids.trees.expressions.MatchPhrase;
import org.apache.doris.nereids.trees.expressions.Not;
import org.apache.doris.nereids.trees.expressions.Or;
+import org.apache.doris.nereids.trees.expressions.SearchExpression;
import org.apache.doris.nereids.trees.expressions.SlotReference;
import org.apache.doris.nereids.trees.expressions.functions.scalar.Abs;
import org.apache.doris.nereids.trees.expressions.functions.scalar.DateTrunc;
+import org.apache.doris.nereids.trees.expressions.functions.scalar.MultiMatch;
+import org.apache.doris.nereids.trees.expressions.functions.scalar.Search;
+import
org.apache.doris.nereids.trees.expressions.functions.scalar.SearchDslParser;
+import org.apache.doris.nereids.trees.expressions.literal.BooleanLiteral;
import org.apache.doris.nereids.trees.expressions.literal.DateTimeV2Literal;
import org.apache.doris.nereids.trees.expressions.literal.IntegerLiteral;
import org.apache.doris.nereids.trees.expressions.literal.StringLiteral;
@@ -39,15 +46,68 @@ import org.apache.doris.nereids.types.DateType;
import org.apache.doris.nereids.types.IntegerType;
import org.apache.doris.nereids.types.StringType;
import org.apache.doris.nereids.types.TinyIntType;
+import org.apache.doris.nereids.util.PredicateInferUtils;
import com.google.common.collect.ImmutableList;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.Arguments;
+import org.junit.jupiter.params.provider.MethodSource;
import java.util.HashSet;
+import java.util.LinkedHashSet;
import java.util.Set;
+import java.util.stream.Stream;
public class InferPredicateByReplaceTest {
+ @ParameterizedTest
+ @MethodSource("matchAndSearchPredicates")
+ public void testDoNotInferMatchOrSearchPredicates(Expression predicate,
SlotReference a, SlotReference b) {
+ Set<Expression> inputs = new LinkedHashSet<>(ImmutableList.of(new
EqualTo(a, b), predicate));
+
+ Assertions.assertEquals(inputs, InferPredicateByReplace.infer(inputs));
+ Assertions.assertEquals(inputs,
PredicateInferUtils.inferPredicate(inputs));
+ Assertions.assertEquals(inputs,
PredicateInferUtils.inferAllPredicate(inputs));
+ }
+
+ private static Stream<Arguments> matchAndSearchPredicates() {
+ SlotReference a = new SlotReference("a", StringType.INSTANCE);
+ SlotReference b = new SlotReference("b", StringType.INSTANCE);
+ return Stream.of(
+ new MatchAny(a, new StringLiteral("apple")),
+ new MatchPhrase(a, new StringLiteral("apple red")),
+ new MultiMatch(a, new StringLiteral("any"), new
StringLiteral("apple")),
+ new Search(new StringLiteral("a:apple")),
+ new SearchExpression("a:apple",
SearchDslParser.parseDsl("a:apple"), ImmutableList.of(a)))
+ .flatMap(search -> {
+ Expression or = new Or(search, new EqualTo(a, new
StringLiteral("pear")));
+ return Stream.of(or,
+ new EqualTo(search, BooleanLiteral.TRUE),
+ new EqualTo(or, BooleanLiteral.TRUE),
+ new GreaterThan(new Cast(or,
IntegerType.INSTANCE), new IntegerLiteral(0)),
+ new InPredicate(or,
ImmutableList.of(BooleanLiteral.TRUE)),
+ new Not(new EqualTo(or, BooleanLiteral.TRUE)),
+ new Like(new Cast(or, StringType.INSTANCE), new
StringLiteral("%true%")));
+ })
+ .map(predicate -> Arguments.of(predicate, a, b));
+ }
+
+ @Test
+ public void testInferOrdinaryPredicatesAlongsideMatch() {
+ SlotReference a = new SlotReference("a", StringType.INSTANCE);
+ SlotReference b = new SlotReference("b", StringType.INSTANCE);
+ Expression match = new Or(new MatchAny(a, new StringLiteral("apple")),
+ new MatchAny(a, new StringLiteral("pear")));
+ Set<Expression> inputs = new LinkedHashSet<>(ImmutableList.of(
+ new EqualTo(a, b), match, new Like(a, new
StringLiteral("apple%"))));
+ Set<Expression> expected = new LinkedHashSet<>(inputs);
+ expected.add(new Like(b, new StringLiteral("apple%")));
+
+ Assertions.assertEquals(expected,
InferPredicateByReplace.infer(inputs));
+ Assertions.assertEquals(expected,
PredicateInferUtils.inferAllPredicate(inputs));
+ }
+
@Test
public void testInferWithEqualTo() {
SlotReference a = new SlotReference("a", IntegerType.INSTANCE);
diff --git
a/regression-test/data/nereids_rules_p0/infer_predicate/infer_match_predicate.out
b/regression-test/data/nereids_rules_p0/infer_predicate/infer_match_predicate.out
new file mode 100644
index 00000000000..cdc045bdd81
--- /dev/null
+++
b/regression-test/data/nereids_rules_p0/infer_predicate/infer_match_predicate.out
@@ -0,0 +1,19 @@
+-- This file is automatically generated. You should know what you did if you
want to edit this
+-- !different_index --
+1
+
+-- !wrapped_match --
+1
+
+-- !search --
+1
+
+-- !multi_match --
+1
+
+-- !score --
+1 0.4700036
+2 0.4700036
+
+-- !minimum_score --
+
diff --git
a/regression-test/suites/nereids_rules_p0/infer_predicate/infer_match_predicate.groovy
b/regression-test/suites/nereids_rules_p0/infer_predicate/infer_match_predicate.groovy
new file mode 100644
index 00000000000..2ad97577e89
--- /dev/null
+++
b/regression-test/suites/nereids_rules_p0/infer_predicate/infer_match_predicate.groovy
@@ -0,0 +1,76 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// An equality between two columns does not let a MATCH on one column be
copied to the other:
+// MATCH depends on the inverted index of its column, and a copy would add to
score().
+suite("infer_match_predicate") {
+ sql """ set enable_common_expr_pushdown = true; """
+ // c2 has an index without a parser, so MATCH on c2 only matches a whole
value.
+ sql "DROP TABLE IF EXISTS infer_match_predicate_parser"
+ sql """
+ CREATE TABLE infer_match_predicate_parser (
+ id INT,
+ c1 TEXT,
+ c2 TEXT,
+ INDEX idx_c1 (c1) USING INVERTED PROPERTIES ("parser" = "english"),
+ INDEX idx_c2 (c2) USING INVERTED
+ ) DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 1
+ PROPERTIES ("replication_num" = "1")
+ """
+ sql "INSERT INTO infer_match_predicate_parser VALUES (1, 'apple red',
'apple red'), (2, 'banana', 'banana')"
+ order_qt_different_index """
+ SELECT id FROM infer_match_predicate_parser
+ WHERE c1 = c2 AND (c1 MATCH_ANY 'apple' OR c1 MATCH_ANY 'pear')
+ """
+ order_qt_wrapped_match """
+ SELECT id FROM infer_match_predicate_parser
+ WHERE c1 = c2 AND CAST((c1 MATCH_ANY 'apple' OR c1 MATCH_ANY 'pear')
AS INT) > 0
+ """
+ order_qt_search """
+ SELECT id FROM infer_match_predicate_parser
+ WHERE c1 = c2 AND (search('c1:apple') OR c1 = 'pear')
+ """
+ order_qt_multi_match """
+ SELECT id FROM infer_match_predicate_parser
+ WHERE c1 = c2 AND (multi_match(c1, 'any', 'apple') OR multi_match(c1,
'any', 'pear'))
+ """
+
+ sql "DROP TABLE IF EXISTS infer_match_predicate_score"
+ sql """
+ CREATE TABLE infer_match_predicate_score (
+ id INT,
+ c1 TEXT,
+ c2 TEXT,
+ INDEX idx_c1 (c1) USING INVERTED PROPERTIES ("parser" = "english"),
+ INDEX idx_c2 (c2) USING INVERTED PROPERTIES ("parser" = "english")
+ ) DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 1
+ PROPERTIES ("replication_num" = "1")
+ """
+ sql """
+ INSERT INTO infer_match_predicate_score VALUES
+ (1, 'apple red', 'apple red'), (2, 'apple green', 'apple green'),
(3, 'banana yellow', 'banana yellow')
+ """
+ order_qt_score """
+ SELECT id, score() AS s FROM infer_match_predicate_score
+ WHERE c1 = c2 AND (c1 MATCH_ANY 'apple' OR c1 MATCH_ANY 'pear') ORDER
BY s DESC LIMIT 10
+ """
+ order_qt_minimum_score """
+ SELECT id, score() AS s FROM infer_match_predicate_score
+ WHERE c1 = c2 AND (c1 MATCH_ANY 'apple' OR c1 MATCH_ANY 'pear') AND
score() > 0.6
+ ORDER BY s DESC LIMIT 10
+ """
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
