This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 326ed5b15ff [test](fe) Cover CAST predicates kept out of Paimon and
table-scoped Hive file listings (#68960)
326ed5b15ff is described below
commit 326ed5b15ffae2baaa792e312406cd4bc9188187
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Tue Oct 13 14:53:25 2026 +0800
[test](fe) Cover CAST predicates kept out of Paimon and table-scoped Hive
file listings (#68960)
### What problem does this PR solve?
Issue Number: None
Related PR: #67933, #68577, #67479, #68568
Problem Summary:
**In short.** This ports the tests for two branch-4.x external table
fixes whose behavior master already has: Paimon must not receive
predicates that contain a CAST (#67933), and Hive file listings must be
cached per table (#68577). It changes no production code. The other two
fixes of the same batch (#67479, #68568) are on master with their tests
through #67687 and #68582.
**Background.**
- Paimon: Doris converts pushed conjuncts into connector expressions,
and that converter drops CASTs. #67933 found that `CAST(code AS INT) =
5` then reached Paimon as `code = 5`, and Paimon pruned the data files
holding `'05'` or `' 5'`, so the query silently lost rows. branch-4.x
rejected such predicates inside its Paimon converter. On master the
Paimon connector declares `supportsCastPredicatePushdown() = false`, and
`PluginDrivenScanNode.buildRemainingFilter` keeps every conjunct with a
CAST anywhere in its tree in Doris: only the plain conjuncts go to the
connector, and the source LIMIT is turned off when a conjunct is held
back.
- Hive: #68577 scoped the branch-4.2 file cache by table, so two tables
whose partitions point at the same directory no longer share a listing.
master's `HiveFileListingCache` keys entries by database, table,
location and partition values, and REFRESH TABLE drops only that table's
entries.
**The problem, and what it cost.** None of this was pinned on master:
| Behavior | Tested on master before this PR |
|---|---|
| `buildRemainingFilter` keeps conjuncts with a CAST away from a
connector that does not accept CASTs, leaves them for Doris and turns
the source LIMIT off | no; tests pin only the value of Paimon's switch |
| A plain conjunct next to a CAST conjunct is still pushed | no |
| The Paimon converter pushes string equality, INT greater-than, IN and
NOT IN | INT equality only |
| Two Hive tables with the same partition directory and values keep
separate listings across a refresh | only separately (different
locations, partitionless keys); a cache key without the table name
passes every existing `HiveFileListingCacheTest` case |
| CAST predicates on a Paimon table, end to end, through the JNI and the
native reader | no |
**How this PR fixes it.**
- `PluginDrivenScanNodeCastPredicateTest` (new, fe-core): plans the nine
CAST shapes of the branch-4.x test (`=`, reversed `=`, `>`, `IN`, `NOT
IN`, `IS NULL`, `IS NOT NULL`, `OR`, decimal scale) and calls
`buildRemainingFilter` on a node whose connector does not accept CASTs:
nothing is sent, the conjuncts stay as they were and the source LIMIT is
off. A plain conjunct next to a CAST conjunct is still pushed alone,
plain conjuncts are pushed whole, and the same CAST conjunct is pushed
once the connector accepts CASTs.
- `PaimonPredicateConverterTest`: string equality, INT greater-than, IN
and NOT IN are pushed, checked by evaluating the Paimon predicates on
rows.
- `HiveFileListingCacheTest`: two tables with the same partition
directory and values list it separately, and refreshing one keeps the
other's entry.
- `test_paimon_cast_predicate_pushdown` (new regression): writes one row
per data file so file pruning can happen, checks with EXPLAIN that the
CAST conjuncts show `predicatesFromPaimon: NONE` while plain conjuncts
are pushed, and reads the nine shapes, an AND with a plain conjunct and
a LIMIT through the JNI and the native reader.
**Results.** All the tests pass. Each new unit test fails when the logic
it guards is changed:
| Change to the code under test | Result |
|---|---|
| Remove the CAST check from `buildRemainingFilter` | 10 of the 14 cases
fail; the plain-conjunct case shows the filter sent to the connector as
`((code = 5) AND (id = 1))`, the stripped CAST that #67933 fixed |
| Drop IN support from the Paimon converter |
`plainStringRangeAndInListPredicatesArePushed` fails |
| Drop the table from the Hive cache key |
`refreshTableWithSharedPartitionLocationIsScopedPerTable` fails, the
other `HiveFileListingCacheTest` cases pass |
In the regression, the JNI and the native reader return the same rows,
and they match an internal Doris table holding the same data.
Co-authored-by: Gabriel <[email protected]>
Co-authored-by: 924060929 <[email protected]>
---
.../connector/hive/HiveFileListingCacheTest.java | 23 ++++
.../paimon/PaimonPredicateConverterTest.java | 45 ++++++
.../PluginDrivenScanNodeCastPredicateTest.java | 152 +++++++++++++++++++++
.../paimon/test_paimon_cast_predicate_pushdown.out | 95 +++++++++++++
.../test_paimon_cast_predicate_pushdown.groovy | 109 +++++++++++++++
5 files changed, 424 insertions(+)
diff --git
a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveFileListingCacheTest.java
b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveFileListingCacheTest.java
index 29bf96e6650..1a654fd327e 100644
---
a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveFileListingCacheTest.java
+++
b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveFileListingCacheTest.java
@@ -163,6 +163,29 @@ public class HiveFileListingCacheTest {
Assertions.assertEquals(1, (int) lister.callsPerLocation.get("locB"));
}
+ @Test
+ public void refreshTableWithSharedPartitionLocationIsScopedPerTable() {
+ CountingLister lister = new CountingLister();
+ HiveFileListingCache cache = new
HiveFileListingCache(HiveTestProperties.minimal(), lister);
+ String location = "/warehouse/shared/p=p1";
+ List<String> values = Collections.singletonList("p1");
+
+ // Two tables whose partitions point at the same directory with the
same values (branch-4.x #68577).
+ // WHY: b must list the shared directory itself instead of serving a's
listing, and refreshing b must
+ // not touch a. MUTATION: dropping (db, table) from the key -> b
reuses a's list -> red.
+ List<HiveFileStatus> filesA = cache.listDataFiles("db", "a", location,
values, FS);
+ List<HiveFileStatus> filesB = cache.listDataFiles("db", "b", location,
values, FS);
+ Assertions.assertEquals(2, lister.totalCalls);
+ Assertions.assertSame(filesA, cache.listDataFiles("db", "a", location,
values, FS));
+ Assertions.assertSame(filesB, cache.listDataFiles("db", "b", location,
values, FS));
+
+ cache.invalidateTable("db", "b");
+
+ Assertions.assertNotSame(filesB, cache.listDataFiles("db", "b",
location, values, FS));
+ Assertions.assertSame(filesA, cache.listDataFiles("db", "a", location,
values, FS));
+ Assertions.assertEquals(3, lister.totalCalls);
+ }
+
@Test
public void invalidateAllDropsEverything() {
CountingLister lister = new CountingLister();
diff --git
a/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonPredicateConverterTest.java
b/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonPredicateConverterTest.java
index 1b0763cfd2d..65a8af2be21 100644
---
a/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonPredicateConverterTest.java
+++
b/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonPredicateConverterTest.java
@@ -20,10 +20,13 @@ package org.apache.doris.connector.paimon;
import org.apache.doris.connector.spi.ConnectorType;
import org.apache.doris.connector.spi.pushdown.ConnectorColumnRef;
import org.apache.doris.connector.spi.pushdown.ConnectorComparison;
+import org.apache.doris.connector.spi.pushdown.ConnectorExpression;
+import org.apache.doris.connector.spi.pushdown.ConnectorIn;
import org.apache.doris.connector.spi.pushdown.ConnectorLike;
import org.apache.doris.connector.spi.pushdown.ConnectorLiteral;
import org.apache.paimon.data.BinaryString;
+import org.apache.paimon.data.GenericRow;
import org.apache.paimon.data.Timestamp;
import org.apache.paimon.predicate.Equal;
import org.apache.paimon.predicate.IsNull;
@@ -37,6 +40,7 @@ import org.junit.jupiter.api.Test;
import java.time.LocalDateTime;
import java.time.ZoneOffset;
+import java.util.Arrays;
import java.util.List;
/**
@@ -174,6 +178,47 @@ public class PaimonPredicateConverterTest {
"the INT literal must be carried through unchanged");
}
+ @Test
+ public void plainStringRangeAndInListPredicatesArePushed() {
+ RowType rowType = RowType.builder()
+ .field("id", DataTypes.INT())
+ .field("code", DataTypes.STRING())
+ .build();
+ PaimonPredicateConverter converter = new
PaimonPredicateConverter(rowType);
+ ConnectorColumnRef id = new ConnectorColumnRef("id", ANY);
+ ConnectorColumnRef code = new ConnectorColumnRef("code", ANY);
+ List<ConnectorExpression> oneAndTwo =
Arrays.asList(ConnectorLiteral.ofInt(1), ConnectorLiteral.ofInt(2));
+
+ // WHY: Paimon declares supportsCastPredicatePushdown() = false, so
the engine keeps every conjunct that
+ // contains a CAST in Doris and this converter never sees one
(PluginDrivenScanNodeCastPredicateTest).
+ // The plain shapes branch-4.x checked next to its CAST cases (#67933)
must still be pushed, otherwise
+ // the conservative CAST rule would have turned into "push nothing".
The predicates are checked by
+ // evaluating them on rows, since paimon may build IN as an OR of
equalities.
+ // MUTATION: dropping string equality, GT or IN support from the
converter -> empty result -> red.
+ Predicate stringEquality = single(converter.convert(new
ConnectorComparison(
+ ConnectorComparison.Operator.EQ, code,
ConnectorLiteral.ofString("5"))));
+ Assertions.assertTrue(stringEquality.test(GenericRow.of(1,
BinaryString.fromString("5"))));
+ Assertions.assertFalse(stringEquality.test(GenericRow.of(1,
BinaryString.fromString("05"))));
+
+ Predicate greaterThan = single(converter.convert(new
ConnectorComparison(
+ ConnectorComparison.Operator.GT, id,
ConnectorLiteral.ofInt(1))));
+ Assertions.assertTrue(greaterThan.test(GenericRow.of(2,
BinaryString.fromString("x"))));
+ Assertions.assertFalse(greaterThan.test(GenericRow.of(1,
BinaryString.fromString("x"))));
+
+ Predicate in = single(converter.convert(new ConnectorIn(id, oneAndTwo,
false)));
+ Assertions.assertTrue(in.test(GenericRow.of(2,
BinaryString.fromString("x"))));
+ Assertions.assertFalse(in.test(GenericRow.of(3,
BinaryString.fromString("x"))));
+
+ Predicate notIn = single(converter.convert(new ConnectorIn(id,
oneAndTwo, true)));
+ Assertions.assertTrue(notIn.test(GenericRow.of(3,
BinaryString.fromString("x"))));
+ Assertions.assertFalse(notIn.test(GenericRow.of(2,
BinaryString.fromString("x"))));
+ }
+
+ private static Predicate single(List<Predicate> predicates) {
+ Assertions.assertEquals(1, predicates.size(), "the predicate must be
pushed as one paimon predicate");
+ return predicates.get(0);
+ }
+
// ---------- null-safe equality (<=>) ----------
@Test
diff --git
a/fe/fe-core/src/test/java/org/apache/doris/datasource/scan/PluginDrivenScanNodeCastPredicateTest.java
b/fe/fe-core/src/test/java/org/apache/doris/datasource/scan/PluginDrivenScanNodeCastPredicateTest.java
new file mode 100644
index 00000000000..2f1ab81edff
--- /dev/null
+++
b/fe/fe-core/src/test/java/org/apache/doris/datasource/scan/PluginDrivenScanNodeCastPredicateTest.java
@@ -0,0 +1,152 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.datasource.scan;
+
+import org.apache.doris.analysis.Expr;
+import org.apache.doris.common.FeConstants;
+import org.apache.doris.common.jmockit.Deencapsulation;
+import org.apache.doris.connector.spi.ConnectorMetadata;
+import org.apache.doris.connector.spi.pushdown.ConnectorColumnRef;
+import org.apache.doris.connector.spi.pushdown.ConnectorComparison;
+import org.apache.doris.connector.spi.pushdown.ConnectorExpression;
+import org.apache.doris.qe.StmtExecutor;
+import org.apache.doris.utframe.TestWithFeService;
+
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.params.ParameterizedTest;
+import org.junit.jupiter.params.provider.ValueSource;
+import org.mockito.Mockito;
+
+import java.util.Collections;
+import java.util.List;
+import java.util.Optional;
+
+/**
+ * Guards how {@link PluginDrivenScanNode} keeps CAST predicates in Doris for
a connector that cannot
+ * evaluate them, such as Paimon.
+ *
+ * <p>The neutral expression converter drops CASTs, so a pushed {@code
CAST(code AS INT) = 5} would reach
+ * the connector as {@code code = 5} and could prune files that hold {@code
'05'} or {@code ' 5'}, losing
+ * rows. A connector that returns false from {@code
supportsCastPredicatePushdown} therefore never receives
+ * a conjunct with a CAST anywhere in its tree; the conjunct stays in Doris
and is evaluated after the
+ * scan. branch-4.x fixed the same row loss inside its Paimon predicate
converter (#67933); master enforces
+ * it in {@code buildRemainingFilter}, which had no direct test.</p>
+ *
+ * <p>The conjuncts come from planning real queries, so the shapes are the
ones Nereids produces. The
+ * filter is then built through the real private method on a {@code
CALLS_REAL_METHODS} node with only the
+ * connector metadata injected.</p>
+ */
+public class PluginDrivenScanNodeCastPredicateTest extends TestWithFeService {
+
+ @Override
+ protected void runBeforeAll() throws Exception {
+ FeConstants.runningUnitTest = true;
+ createDatabase("db1");
+ createTables("create table db1.cast_predicates (id int, code string,
amount decimal(10, 2))"
+ + " distributed by hash(id) properties('replication_num' =
'1')");
+ }
+
+ @ParameterizedTest
+ @ValueSource(strings = {"cast(code as int) = 5", "5 = cast(code as int)",
"cast(code as int) > 5",
+ "cast(code as int) in (5, 6)", "cast(code as int) not in (5, 6)",
+ "cast(code as int) is null", "cast(code as int) is not null",
+ "cast(code as int) = 5 or id = 1", "cast(amount as decimal(10, 1))
= 1.2"})
+ public void castConjunctsStayInDoris(String predicate) throws Exception {
+ List<Expr> conjuncts = planConjuncts(predicate);
+ // Precondition: the planner keeps the CAST as a CastExpr node, the
shape the rule looks for.
+
Assertions.assertTrue(conjuncts.stream().anyMatch(PluginDrivenScanNode::containsCastExpr),
predicate);
+ List<Expr> before = Expr.cloneList(conjuncts);
+
+ PluginDrivenScanNode node = nodeWith(conjuncts, false);
+ Optional<ConnectorExpression> filter = Deencapsulation.invoke(node,
"buildRemainingFilter");
+
+ Assertions.assertFalse(filter.isPresent(), predicate);
+ // Nothing was sent to the connector, so the source LIMIT must stay
off ...
+ Assertions.assertEquals(Collections.emptyList(),
+ Deencapsulation.getField(node, "filteredToOriginalIndex"),
predicate);
+ // ... and every conjunct must remain for Doris to evaluate.
+ Assertions.assertEquals(before, conjuncts, predicate);
+ }
+
+ @Test
+ public void independentPlainConjunctIsStillPushed() throws Exception {
+ List<Expr> conjuncts = planConjuncts("cast(code as int) = 5 and id =
1");
+ Assertions.assertEquals(2, conjuncts.size());
+ int plainIndex =
PluginDrivenScanNode.containsCastExpr(conjuncts.get(0)) ? 1 : 0;
+
Assertions.assertFalse(PluginDrivenScanNode.containsCastExpr(conjuncts.get(plainIndex)));
+ List<Expr> before = Expr.cloneList(conjuncts);
+
+ PluginDrivenScanNode node = nodeWith(conjuncts, false);
+ Optional<ConnectorExpression> filter = Deencapsulation.invoke(node,
"buildRemainingFilter");
+
+ Assertions.assertTrue(filter.isPresent());
+ Assertions.assertTrue(filter.get() instanceof ConnectorComparison,
filter.get().toString());
+ ConnectorComparison comparison = (ConnectorComparison) filter.get();
+ Assertions.assertEquals("id", ((ConnectorColumnRef)
comparison.getLeft()).getColumnName());
+ Assertions.assertEquals(Collections.singletonList(plainIndex),
+ Deencapsulation.getField(node, "filteredToOriginalIndex"));
+ Assertions.assertEquals(before, conjuncts);
+ }
+
+ @ParameterizedTest
+ @ValueSource(strings = {"code = '5'", "id > 1", "id in (1, 2)"})
+ public void plainConjunctsArePushedWhole(String predicate) throws
Exception {
+ List<Expr> conjuncts = planConjuncts(predicate);
+
Assertions.assertFalse(conjuncts.stream().anyMatch(PluginDrivenScanNode::containsCastExpr),
predicate);
+
+ PluginDrivenScanNode node = nodeWith(conjuncts, false);
+ Optional<ConnectorExpression> filter = Deencapsulation.invoke(node,
"buildRemainingFilter");
+
+ Assertions.assertTrue(filter.isPresent(), predicate);
+ // Every conjunct went to the connector, so no index remapping is
needed.
+ Assertions.assertNull(Deencapsulation.getField(node,
"filteredToOriginalIndex"), predicate);
+ }
+
+ @Test
+ public void castConjunctIsPushedWhenTheConnectorAcceptsCasts() throws
Exception {
+ List<Expr> conjuncts = planConjuncts("cast(code as int) = 5");
+
+ PluginDrivenScanNode node = nodeWith(conjuncts, true);
+ Optional<ConnectorExpression> filter = Deencapsulation.invoke(node,
"buildRemainingFilter");
+
+ // The same conjunct is pushed once the connector accepts CASTs, so
the rule above is the switch.
+ Assertions.assertTrue(filter.isPresent());
+ Assertions.assertNull(Deencapsulation.getField(node,
"filteredToOriginalIndex"));
+ }
+
+ private List<Expr> planConjuncts(String predicate) throws Exception {
+ StmtExecutor executor = new StmtExecutor(connectContext,
+ "select * from db1.cast_predicates where " + predicate);
+ executor.execute();
+ return executor.planner().getScanNodes().get(0).getConjuncts();
+ }
+
+ /**
+ * A node whose connector declares {@code supportsCast} and whose
conjuncts are {@code conjuncts}.
+ * {@code cachedMetadata} is injected directly so the per-statement
metadata funnel stays out of the test.
+ */
+ private static PluginDrivenScanNode nodeWith(List<Expr> conjuncts, boolean
supportsCast) {
+ PluginDrivenScanNode node = Mockito.mock(PluginDrivenScanNode.class,
Mockito.CALLS_REAL_METHODS);
+ ConnectorMetadata metadata = Mockito.mock(ConnectorMetadata.class);
+
Mockito.when(metadata.supportsCastPredicatePushdown(Mockito.any())).thenReturn(supportsCast);
+ Deencapsulation.setField(node, "cachedMetadata", metadata);
+ Deencapsulation.setField(node, "conjuncts", conjuncts);
+ return node;
+ }
+}
diff --git
a/regression-test/data/external_table_p0/paimon/test_paimon_cast_predicate_pushdown.out
b/regression-test/data/external_table_p0/paimon/test_paimon_cast_predicate_pushdown.out
new file mode 100644
index 00000000000..42744006644
--- /dev/null
+++
b/regression-test/data/external_table_p0/paimon/test_paimon_cast_predicate_pushdown.out
@@ -0,0 +1,95 @@
+-- This file is automatically generated. You should know what you did if you
want to edit this
+-- !jni_0 --
+1 05 1.24
+2 5 1.20
+3 5 1.30
+
+-- !jni_1 --
+1 05 1.24
+2 5 1.20
+3 5 1.30
+
+-- !jni_2 --
+1 05 1.24
+2 5 1.20
+3 5 1.30
+
+-- !jni_3 --
+1 05 1.24
+2 5 1.20
+3 5 1.30
+
+-- !jni_4 --
+
+-- !jni_5 --
+4 abc \N
+5 \N 1.25
+
+-- !jni_6 --
+1 05 1.24
+2 5 1.20
+3 5 1.30
+
+-- !jni_7 --
+1 05 1.24
+2 5 1.20
+3 5 1.30
+4 abc \N
+
+-- !jni_8 --
+1 05 1.24
+2 5 1.20
+
+-- !jni_9 --
+3 5 1.30
+
+-- !jni_limit --
+1
+
+-- !native_0 --
+1 05 1.24
+2 5 1.20
+3 5 1.30
+
+-- !native_1 --
+1 05 1.24
+2 5 1.20
+3 5 1.30
+
+-- !native_2 --
+1 05 1.24
+2 5 1.20
+3 5 1.30
+
+-- !native_3 --
+1 05 1.24
+2 5 1.20
+3 5 1.30
+
+-- !native_4 --
+
+-- !native_5 --
+4 abc \N
+5 \N 1.25
+
+-- !native_6 --
+1 05 1.24
+2 5 1.20
+3 5 1.30
+
+-- !native_7 --
+1 05 1.24
+2 5 1.20
+3 5 1.30
+4 abc \N
+
+-- !native_8 --
+1 05 1.24
+2 5 1.20
+
+-- !native_9 --
+3 5 1.30
+
+-- !native_limit --
+1
+
diff --git
a/regression-test/suites/external_table_p0/paimon/test_paimon_cast_predicate_pushdown.groovy
b/regression-test/suites/external_table_p0/paimon/test_paimon_cast_predicate_pushdown.groovy
new file mode 100644
index 00000000000..5b48699a813
--- /dev/null
+++
b/regression-test/suites/external_table_p0/paimon/test_paimon_cast_predicate_pushdown.groovy
@@ -0,0 +1,109 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// WHY: Paimon cannot evaluate a Doris CAST, and the connector expression
converter drops CASTs. A pushed
+// CAST(code AS INT) = 5 would reach Paimon as code = 5 and let it prune the
data files that hold '05' or ' 5',
+// so the query would silently return fewer rows. Paimon declares
supportsCastPredicatePushdown() = false: a
+// conjunct with a CAST anywhere in it stays in Doris, and only the plain
conjuncts reach Paimon. branch-4.x
+// fixed the same row loss in its Paimon predicate converter (#67933).
+//
+// Each row is written in its own INSERT so the rows land in separate data
files and file-level pruning is
+// reachable; with a single file the scan would read every row anyway and a
wrong pushdown would hide.
+suite("test_paimon_cast_predicate_pushdown", "p0,external") {
+ String enabled = context.config.otherConfigs.get("enablePaimonTest")
+ if (enabled == null || !enabled.equalsIgnoreCase("true")) {
+ logger.info("disable paimon test")
+ return
+ }
+
+ String minio_port = context.config.otherConfigs.get("iceberg_minio_port")
+ String externalEnvIp = context.config.otherConfigs.get("externalEnvIp")
+ String catalog_name = "test_paimon_cast_predicate_pushdown"
+ String dbName = "test_paimon_cast_predicate_pushdown_db"
+
+ spark_paimon_multi """
+ create database if not exists paimon.${dbName};
+ drop table if exists paimon.${dbName}.cast_pushdown_tbl;
+ create table paimon.${dbName}.cast_pushdown_tbl (id int, code string,
amount decimal(10, 2)) using paimon;
+ insert into paimon.${dbName}.cast_pushdown_tbl values (1, '05', 1.24);
+ insert into paimon.${dbName}.cast_pushdown_tbl values (2, ' 5', 1.20);
+ insert into paimon.${dbName}.cast_pushdown_tbl values (3, '5', 1.30);
+ insert into paimon.${dbName}.cast_pushdown_tbl values (4, 'abc', null);
+ insert into paimon.${dbName}.cast_pushdown_tbl values (5, null, 1.25);
+ """
+
+ sql """drop catalog if exists ${catalog_name}"""
+ sql """
+ CREATE CATALOG ${catalog_name} PROPERTIES (
+ 'type' = 'paimon',
+ 'warehouse' = 's3://warehouse/wh',
+ 's3.endpoint' = 'http://${externalEnvIp}:${minio_port}',
+ 's3.access_key' = 'admin',
+ 's3.secret_key' = 'password',
+ 's3.path.style.access' = 'true'
+ );
+ """
+ sql """use `${catalog_name}`.`${dbName}`"""
+
+ def castPredicates = [
+ "cast(code as int) = 5",
+ "5 = cast(code as int)",
+ "cast(code as int) > 4",
+ "cast(code as int) in (5, 6)",
+ "cast(code as int) not in (5, 6)",
+ "cast(code as int) is null",
+ "cast(code as int) is not null",
+ "cast(code as int) = 5 or id = 4",
+ "cast(amount as decimal(10, 1)) = 1.2",
+ ]
+
+ // A conjunct that contains a CAST never reaches Paimon.
+ castPredicates.each { predicate ->
+ explain {
+ sql "select id from cast_pushdown_tbl where ${predicate}"
+ contains "predicatesFromPaimon: NONE"
+ }
+ }
+ // Plain conjuncts are still pushed, also next to a CAST conjunct that
stays in Doris.
+ ["id = 3", "cast(code as int) = 5 and id = 3"].each { predicate ->
+ explain {
+ sql "select id from cast_pushdown_tbl where ${predicate}"
+ contains "predicatesFromPaimon:"
+ notContains "predicatesFromPaimon: NONE"
+ }
+ }
+
+ try {
+ ["jni": "true", "native": "false"].each { reader, forceJni ->
+ sql """set force_jni_scanner = ${forceJni}"""
+ (castPredicates + ["cast(code as int) = 5 and id =
3"]).eachWithIndex { predicate, index ->
+ "order_qt_${reader}_${index}"("""
+ select id, code, amount from cast_pushdown_tbl where
${predicate} order by id
+ """)
+ }
+ // The source LIMIT stays off while a CAST conjunct is evaluated
in Doris, so one matching row is
+ // returned instead of whatever row Paimon would have read first.
+ "order_qt_${reader}_limit"("""
+ select count(*) from (
+ select id from cast_pushdown_tbl where cast(code as int) =
5 limit 1
+ ) t
+ """)
+ }
+ } finally {
+ sql """set force_jni_scanner = false"""
+ }
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
