github-actions[bot] commented on code in PR #69493:
URL: https://github.com/apache/doris/pull/69493#discussion_r4373642130
##########
fe/pom.xml:
##########
@@ -418,7 +418,7 @@ under the License.
<quartz.version>2.3.2</quartz.version>
<aircompressor.version>2.0.3</aircompressor.version>
<!-- paimon -->
- <paimon.version>1.4.2</paimon.version>
+ <paimon.version>2.1.0</paimon.version>
Review Comment:
[P2] Keep old privilege tables decodable during BE-first upgrades
During the supported [BE-before-FE rolling
upgrade](https://doris.apache.org/docs/4.x/admin-manual/cluster-management/upgrade/),
the old 1.4.2 FE can still send a `PrivilegedFileStoreTable` for an existing
privilege-enabled warehouse. For example, `scan.manifest.parallelism=1` makes
the old `PaimonReaderOptions` return the wrapper unchanged, and
`serializeProcessedTable` Java-serializes it intact. The upgraded BE then calls
`PaimonUtils.deserialize` with the new connector classloader before any
normalization, but 2.1 removed that class and its privilege support classes, so
ordinary JNI queries fail with `ClassNotFoundException` while FE is still old.
Retain compatible deserialization or provide a migration of this transport
before replacing the BE dependency; removing new-FE wrapper references does not
make old-FE serialized tables decodable.
##########
fe/pom.xml:
##########
@@ -418,7 +418,7 @@ under the License.
<quartz.version>2.3.2</quartz.version>
<aircompressor.version>2.0.3</aircompressor.version>
<!-- paimon -->
- <paimon.version>1.4.2</paimon.version>
+ <paimon.version>2.1.0</paimon.version>
Review Comment:
[P1] Preserve legacy Paimon warehouse authorization on upgrade
An existing filesystem catalog can point at a warehouse with
`user.sys`/`privilege.sys` and set `paimon.user=reader`/`paimon.password`,
while its storage credentials can read the files. If `reader` lacks SELECT on
`db.secret`, 1.4.2 denies the ordinary query through
[PrivilegedFileStoreTable.newScan](https://github.com/apache/paimon/blob/ac223f47d916f4c8e9c07fd1745db448e4e7d0d8/paimon-core/src/main/java/org/apache/paimon/privilege/PrivilegedFileStoreTable.java).
This bump removes that layer: the [2.1
factory](https://github.com/apache/paimon/blob/3a5ffe7870697e2c2605e63413fb5ac5800aae7e/paimon-core/src/main/java/org/apache/paimon/catalog/CatalogFactory.java)
returns only the unwrapped/cached catalog, and independent query authorization
defaults off. Doris still accepts those user/password properties, so the
unchanged catalog can now read the previously denied table after the full
upgrade. Preserve the legacy checks, or detect legacy privilege metadata and
reject the catalog until
an explicit migration; silently dropping those ACLs changes an existing
authorization boundary.
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushDownVectorTopNIntoPaimonScan.java:
##########
@@ -0,0 +1,819 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite;
+
+import org.apache.doris.analysis.TableScanParams;
+import org.apache.doris.catalog.Env;
+import org.apache.doris.catalog.Type;
+import org.apache.doris.datasource.credentials.CredentialUtils;
+import org.apache.doris.datasource.credentials.VendedCredentialsFactory;
+import org.apache.doris.datasource.paimon.PaimonExternalCatalog;
+import org.apache.doris.datasource.paimon.PaimonExternalTable;
+import org.apache.doris.datasource.paimon.PaimonUtil;
+import org.apache.doris.datasource.paimon.source.PaimonScanNode;
+import org.apache.doris.datasource.paimon.source.PaimonVectorSearch;
+import org.apache.doris.datasource.property.storage.StorageProperties;
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.trees.expressions.Alias;
+import org.apache.doris.nereids.trees.expressions.ComparisonPredicate;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.InPredicate;
+import org.apache.doris.nereids.trees.expressions.IsNull;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Not;
+import org.apache.doris.nereids.trees.expressions.SlotReference;
+import org.apache.doris.nereids.trees.expressions.StatementScopeIdGenerator;
+import
org.apache.doris.nereids.trees.expressions.functions.NoneMovableFunction;
+import
org.apache.doris.nereids.trees.expressions.functions.scalar.InnerProductApproximate;
+import
org.apache.doris.nereids.trees.expressions.functions.scalar.L2DistanceApproximate;
+import org.apache.doris.nereids.trees.expressions.literal.ArrayLiteral;
+import org.apache.doris.nereids.trees.expressions.literal.Literal;
+import org.apache.doris.nereids.trees.expressions.literal.NullLiteral;
+import org.apache.doris.nereids.trees.plans.AnnTopNInfo;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalFileScan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalFilter;
+import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
+import org.apache.doris.nereids.trees.plans.logical.LogicalTopN;
+import org.apache.doris.nereids.util.ExpressionUtils;
+import org.apache.doris.qe.ConnectContext;
+import org.apache.doris.thrift.TVectorMetric;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+import com.google.common.collect.Maps;
+import org.apache.paimon.CoreOptions;
+import org.apache.paimon.schema.TableSchema;
+import org.apache.paimon.table.FileStoreTable;
+import org.apache.paimon.table.Table;
+import org.apache.paimon.types.ArrayType;
+import org.apache.paimon.types.DataType;
+import org.apache.paimon.types.DataTypeRoot;
+import org.apache.paimon.types.MapType;
+import org.apache.paimon.types.MultisetType;
+import org.apache.paimon.types.RowType;
+
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+
+/**
+ * Push a primary-key vector (ANN) top-N down into the Paimon scan
(index-only).
+ *
+ * <p>Recognizes {@code TopN[order by dis] -> Project[dis := dist_fn(vec, q)]
-> [Filter] -> PaimonScan}
+ * where {@code dist_fn} is {@code l2_distance_approximate} (ASC) or
+ * {@code inner_product_approximate} (DESC) and the Paimon table has a
PK-vector
+ * index on {@code vec} with a matching distance metric.
+ *
+ * <p>The rewrite mirrors the internal-table ANN path (see
+ * {@code PushDownVectorTopNIntoOlapScan}): the scan is told to produce a
+ * reader-generated {@code __paimon_search_score_to_dis} column via the
PK-vector index, and
+ * every occurrence of {@code dist_fn(vec, q)} is replaced by that column's
slot. The
+ * order key slot keeps its identity, so the surrounding TopN merges/limits
unchanged.
+ * The distance always comes from the index, never from the raw vector.
+ *
+ * <p>No arithmetic is emitted here even though paimon-rust reports a score
rather than
+ * a distance: the BE reader undoes the metric-dependent score transform in
place while
+ * filling the block, exactly as the internal table sqrt's faiss's squared
distance inside
+ * the index layer. So the column arrives already holding what {@code dist_fn}
would
+ * return, and the plan stays a bare slot reference -- which also keeps {@code
dis}
+ * non-nullable (an expression over the score would not be, since {@code
Divide} and
+ * {@code Sqrt} are always nullable) and makes predicates like
+ * {@code dist_fn(vec, q) < 10} correct without any direction flip.
+ *
+ * <p>Being index-only is just an optimization here, not a precondition: if
nothing
+ * else references the raw vector column it is left unreferenced and dropped
by column
+ * pruning (pure index-only read); if the query still needs it ({@code SELECT
vec} or a
+ * predicate on {@code vec}), it stays a normal projected column and the BE
vector
+ * reader materializes it alongside the score. Either way ANN is applied --
the rewrite
+ * never falls back merely because the vector is read.
+ */
+public class PushDownVectorTopNIntoPaimonScan implements RewriteRuleFactory {
+
+ /** Doris column types the BE paimon-rust predicate converter can push; see
+ * PaimonRustPredicateConverter._is_supported_slot_type on the BE side. */
+ private static final Set<Type> VECTOR_FILTER_SUPPORTED_COLUMN_TYPES =
ImmutableSet.of(
+ Type.BOOLEAN, Type.TINYINT, Type.SMALLINT, Type.INT, Type.BIGINT,
+ Type.STRING, Type.DATE, Type.DATEV2);
+
+ /** The BE rust predicate converter encodes the IN option list under a
16-bit
+ * child count; a bigger list is dropped whole into the post-fetch
residual. */
+ private static final int MAX_IN_PREDICATE_CHILDREN = 0xFFFF;
+
+ @Override
+ public List<Rule> buildRules() {
+ return ImmutableList.of(
+ logicalTopN(logicalProject(logicalFileScan())).when(t ->
t.getOrderKeys().size() == 1).then(topN -> {
+ LogicalProject<LogicalFileScan> project = topN.child();
+ LogicalFileScan scan = project.child();
+ return pushDown(topN, project, scan, Optional.empty());
+ }).toRule(RuleType.PUSH_DOWN_VECTOR_TOPN_INTO_PAIMON_SCAN),
+ logicalTopN(logicalProject(logicalFilter(logicalFileScan())))
+ .when(t -> t.getOrderKeys().size() == 1).then(topN -> {
+ LogicalProject<LogicalFilter<LogicalFileScan>>
project = topN.child();
+ LogicalFilter<LogicalFileScan> filter =
project.child();
+ LogicalFileScan scan = filter.child();
+ return pushDown(topN, project, scan,
Optional.of(filter));
+
}).toRule(RuleType.PUSH_DOWN_VECTOR_TOPN_INTO_PAIMON_SCAN)
+ );
+ }
+
+ private Plan pushDown(
+ LogicalTopN<?> topN,
+ LogicalProject<?> project,
+ LogicalFileScan scan,
+ Optional<LogicalFilter<?>> optionalFilter) {
+ // Only Paimon external tables carry PK-vector indexes.
+ if (!(scan.getTable() instanceof PaimonExternalTable)) {
+ return null;
+ }
+
+ // The index-only search path is executed exclusively by the BE
paimon-rust reader,
+ // which only FileScannerV2 can select (the V1 FileScanner explicitly
rejects
+ // PAIMON_RUST). If the rust reader or the V2 scanner is disabled, the
split would
+ // fall through to the JNI reader, which ignores the vector payload
and would
+ // return a non-ANN scan with an empty distance column. Fall back to a
normal
+ // scan then.
+ ConnectContext ctx = ConnectContext.get();
+ if (ctx == null || ctx.getSessionVariable() == null
+ || !ctx.getSessionVariable().isEnablePaimonRustReader()
+ || !ctx.getSessionVariable().enableFileScannerV2) {
+ return null;
+ }
+
+ // An incremental scan (t@incr / BETWEEN) must not be rewritten into a
+ // PK-vector search: the VectorScan builder has no incremental mode,
so a
+ // vector payload would silently return snapshot rows for a query that
asked
+ // for changes. Keeping the query un-rewritten runs it as the ordinary
+ // incremental scan the user requested, with the distance computed in
Doris.
+ // (The scan node's vector gate keeps !isIncremental as the
authoritative
+ // backstop for any path that produces a vector split regardless.)
+ //
+ // The same holds for a pinned file-creation-time read state: the
ordinary
+ // path enforces `scan.file-creation-time-millis` by dropping manifest
+ // entries older than the bound, but the vector search plans its own
splits
+ // and never consults that bound, so old files' nearest neighbors
could leak
+ // into the result. Snapshot/tag/version selectors are fine -- they are
+ // resolved into the table handle before the vector search -- so only
the
+ // file-level bound forces the decline.
+ //
+ // `scan.creation-time-millis` must decline as well: when a snapshot
exists
+ // past the bound it resolves into an ordinary snapshot pin (safe),
but when
+ // none does — the exact read state the option exists for —
PaimonScanParams
+ // resolves it into the SAME internal file-creation bound
(resolveOptions
+ // falls through to resolveFileCreationTime), which the vector path
again
+ // never consults. Which case applies depends on the snapshot manager's
+ // state, so the raw option key is declined unconditionally.
+ if (scan.getScanParams().isPresent()) {
+ TableScanParams scanParams = scan.getScanParams().get();
+ if (scanParams.incrementalRead()) {
+ return null;
+ }
+ Map<String, String> rawParams = scanParams.getMapParams();
+ if
(rawParams.containsKey(CoreOptions.SCAN_FILE_CREATION_TIME_MILLIS.key())
+ ||
rawParams.containsKey(CoreOptions.SCAN_CREATION_TIME_MILLIS.key())) {
+ return null;
+ }
+ }
+
+ // A bounded vector top-N is only correct when every filter conjunct is
+ // applied inside the search, before the limit is exhausted. The BE
+ // paimon-rust predicate converter skips conjuncts it cannot represent
(and
+ // the synthetic distance column is not in its field registry at all),
and
+ // the engine re-applies those residuals on the returned block --
after the
+ // top-K rows were already selected. A skipped residual can therefore
drop
+ // rows out of a bounded result: with candidate distances 0, 2 and 3, a
+ // `WHERE dis > 1 ORDER BY dis LIMIT 1` would fetch only the
distance-0 row,
+ // filter it out and return nothing instead of the distance-2 row. So
the
+ // rewrite only fires when each conjunct is provably of a form the
converter
+ // pushes: a simple comparison / IN / null-check on a table column
whose
+ // Doris type is in the converter's supported set (see
+ // PaimonRustPredicateConverter._is_supported_slot_type on BE).
Anything
+ // else -- an expression predicate, a predicate on the distance
itself, an
+ // unsupported column type -- declines the rewrite and the query runs
as an
+ // ordinary scan with the distance computed in Doris, which is always
+ // correct.
+ if (optionalFilter.isPresent() &&
!conjunctsRideIntoVectorSearch(optionalFilter.get())) {
+ return null;
+ }
+
+ // The order key must be a SlotReference produced by a Project alias.
+ Expression orderKey = topN.getOrderKeys().get(0).getExpr();
+ if (!(orderKey instanceof SlotReference)) {
+ return null;
+ }
+ SlotReference keySlot = (SlotReference) orderKey;
+ Alias orderKeyAlias = null;
+ Expression orderKeyExpr = null;
+ for (NamedExpression projection : project.getProjects()) {
+ if (projection.toSlot().equals(keySlot) && projection instanceof
Alias) {
+ orderKeyAlias = (Alias) projection;
+ orderKeyExpr = orderKeyAlias.child();
+ break;
+ }
+ }
+ if (orderKeyExpr == null) {
+ return null;
+ }
+
+ // l2 must sort ascending (smaller distance = closer); inner_product
descending.
+ boolean isAsc = topN.getOrderKeys().get(0).isAsc();
+ boolean l2Dist = orderKeyExpr instanceof L2DistanceApproximate;
+ boolean innerProduct = orderKeyExpr instanceof InnerProductApproximate;
+ if (!(l2Dist && isAsc) && !(innerProduct && !isAsc)) {
+ return null;
+ }
+
+ // The indexed column must be a direct slot reference. A Cast chain
over the
+ // column is NOT stripped: a legal cast can change vector values (for
example
+ // an ARRAY<FLOAT> -> ARRAY<INT> -> ARRAY<FLOAT> round trip loses
precision),
+ // and searching the original vectors while the query means the casted
ones
+ // changes both the distance and the row order. Doris's implicit
coercions do
+ // not produce a cast here for a matching ARRAY<FLOAT> column, so
anything
+ // wrapped in a Cast simply declines the rewrite.
+ Expression left;
+ Expression right;
+ if (l2Dist) {
+ left = ((L2DistanceApproximate) orderKeyExpr).left();
+ right = ((L2DistanceApproximate) orderKeyExpr).right();
+ } else {
+ left = ((InnerProductApproximate) orderKeyExpr).left();
+ right = ((InnerProductApproximate) orderKeyExpr).right();
+ }
+ if (!(left instanceof SlotReference && right.isConstant() && right
instanceof ArrayLiteral)) {
+ return null;
+ }
+ // A typed NULL inside the query vector ([CAST(NULL AS FLOAT), 0.0,
...]) is a
+ // NullLiteral whose getDouble() returns 0, so extracting the
coordinates
+ // would silently search against a zero vector. The ordinary path
raises
+ // instead (FunctionArrayDistance::_validate_no_nulls: "Second
argument ...
+ // cannot have null"), and the rewrite must not replace that error
with rows
+ // for a different input, so any null element declines the rewrite.
Both
+ // supported metrics share this.
+ if (((ArrayLiteral)
right).getValue().stream().anyMatch(NullLiteral.class::isInstance)) {
+ return null;
+ }
+ SlotReference vectorSlot = (SlotReference) left;
+ if (!vectorSlot.getOriginalColumn().isPresent() ||
!vectorSlot.getOriginalTable().isPresent()) {
+ return null;
+ }
+ String annColumnName = vectorSlot.getOriginalColumn().get().getName();
+
+ // Validate the PK-vector index and metric on the relation's bound
Paimon
+ // table: a branch table carries its own index options and can differ
from
+ // the main table (see boundPaimonTable).
+ PaimonExternalTable paimonTable = (PaimonExternalTable)
scan.getTable();
+ CoreOptions coreOptions;
+ try {
+ Table basePaimonTable = boundPaimonTable(paimonTable, scan);
+ coreOptions = CoreOptions.fromMap(basePaimonTable.options());
+ } catch (Throwable t) {
+ return null;
+ }
+ if (!coreOptions.primaryKeyVectorIndexEnabled()) {
+ return null;
+ }
+ boolean columnIndexed =
coreOptions.primaryKeyVectorIndexColumns().stream()
+ .anyMatch(c -> c.equalsIgnoreCase(annColumnName));
+ if (!columnIndexed) {
+ return null;
+ }
+ // v1 supports only l2 / inner_product; the function must match the
index metric.
+ String metric =
coreOptions.primaryKeyVectorDistanceMetric(annColumnName);
+ String expectedMetric = l2Dist ? "l2" : "inner_product";
+ if (!expectedMetric.equals(metric)) {
+ return null;
+ }
+
+ // A data predicate additionally requires the SDK's "physical row read"
+ // table mode: the predicate itself converts fine, but the pinned
+ // paimon-rust PkVectorSearchParams::resolve rejects every DATA
predicate
+ // unless deletion vectors are enabled and merge-on-read is disabled —
+ // so the failure lands AFTER this one-way rewrite replaced the
distance
+ // expression, turning a query that the ordinary path answers into a
+ // user-facing error. Partition-only conjuncts are enforced by the
search
+ // planning itself and stay fine, so decline only on a non-partition
one.
+ //
+ // The synthetic distance slot is named after a legal paimon user
column
+ // name (__paimon_search_score_to_dis, see the BE reader). A table that
+ // actually HAS a column of that name would collide with the synthetic
+ // slot's block-column matching; decline and run the ordinary path.
+ //
+ // A complex-typed column (STRUCT/MAP/nested ARRAY) must ride as a
whole
+ // column: nested pruning rewrites a slot whose fields are accessed —
+ // Project(struct_element(payload, 'b')) materializes the scan tuple as
+ // ROW<b> — but the vector reader projects only root names and returns
+ // the full Arrow value, and the struct SerDe then binds children by
+ // ORDINAL: the wrong child comes back. Filter conjuncts cannot reach
+ // this (the whitelist admits only scalar columns); the order-key alias
+ // is replaced by the synthetic slot and cannot reach it either; so
only
+ // a Project output expression over a complex column has to decline.
+ //
+ // A NoneMovableFunction in the projection (assert_true) must keep
+ // evaluating below the TopN — see projectContainsNonMovableExpression.
+ //
+ // A historically evolved schema must stay off the vector reader: its
+ // materialization decodes projected values through Arrow safe casts,
+ // which can silently NULL out-of-range values (a TIMESTAMP(3) widened
+ // to TIMESTAMP(9) reading a year-2300 value) where the ordinary path
+ // preserves them — see
PaimonRustReaderCapabilities.hasRustIncompatibleSchemaHistory.
+ try {
+ Table table = boundPaimonTable(paimonTable, scan);
+ if ((optionalFilter.isPresent()
+ && filterHasDataConjunct(optionalFilter.get(), table)
+ &&
!PaimonScanNode.vectorSearchSupportsDataPredicate(table))
+ || table.rowType().getFieldNames().stream().anyMatch(
+ name ->
name.equalsIgnoreCase(PaimonVectorSearch.SEARCH_DISTANCE_COLUMN)
+ ||
name.equalsIgnoreCase("__paimon_search_score")
+ ||
name.equalsIgnoreCase("_PKEY_VECTOR_POSITION"))
+ || !complexColumnsRideAsWholeColumns(project,
orderKeyAlias, orderKeyExpr)
+ || projectContainsNonMovableExpression(project,
orderKeyAlias)
+ || (table instanceof FileStoreTable
+ &&
PaimonVectorSearch.hasRustIncompatibleSchemaHistory(
+ (FileStoreTable) table))
+ || projectionNeedsNanosecondReader(project,
optionalFilter)) {
+ return null;
+ }
+ } catch (Throwable t) {
+ return null;
+ }
+
+ // Pre-flight the projection-dependent bounds the vector split would
hit at
+ // the scan node: a PK-vector split has no JNI fallback (it would
return an
+ // empty distance column), so a query whose projected columns would set
+ // projectedVariant -- or an ORC table with TIMESTAMP_LTZ in its
schema, the
+ // orc-ltz-shape bound -- must not be rewritten at all. Without the
rewrite
+ // those queries run through the ordinary paths, where they are
supported.
+ try {
+ if (projectedColumnsContainVariant(project, optionalFilter)
+ ||
orcSchemaContainsTimestampLtz(boundPaimonTable(paimonTable, scan))) {
+ return null;
+ }
+ } catch (Throwable t) {
+ return null;
+ }
+
+ // The thrift-level enum the AnnTopNInfo will carry. expectedMetric is
kept as a
+ // string for the comparison above (coreOptions returns a string), but
the
+ // AnnTopNInfo / wire contract uses TVectorMetric, so map once here.
DOT_PRODUCT
+ // is the thrift enum that paimon-rust's "inner_product" score
transform maps to
+ // (see the BE reader's parse_score_transform).
+ TVectorMetric expectedMetricEnum = l2Dist ? TVectorMetric.L2 :
TVectorMetric.DOT_PRODUCT;
+
+ // Pre-flight the rust eligibility the scan node will gate on. The
rewrite is
+ // one-way: a PK-vector split has no JNI fallback (the JNI reader
ignores the
+ // vector payload and would return a non-ANN scan with an empty
distance
+ // column), so the scan node can only fail loudly — a table whose gate
would
+ // fail (credential provider shape, REST-token IO, query-auth,
merge-option
+ // matrix, deletion-vector merge-on-read, scan-mode, storage scheme or
HDFS
+ // credentials, no rust-capable backend) must not be rewritten at all.
Without
+ // the rewrite the query still runs, computing the distance in Doris.
The
+ // projection-dependent bounds (ORC LTZ, projected variant) need the
final scan
+ // tuple and stay scan-node-side, and the scan node re-evaluates on the
+ // relation-option-processed table, so it stays authoritative for
t@options
+ // overrides this raw-table pre-flight cannot see.
+ try {
+ Table baseTable = boundPaimonTable(paimonTable, scan);
+ PaimonExternalCatalog catalog = (PaimonExternalCatalog)
paimonTable.getCatalog();
+ Map<StorageProperties.Type, StorageProperties>
storagePropertiesMap =
+
VendedCredentialsFactory.getStoragePropertiesMapWithVendedCredentials(
+
catalog.getCatalogProperty().getMetastoreProperties(),
+
catalog.getCatalogProperty().getStoragePropertiesMap(),
+ baseTable);
+ Map<String, String> backendStorageProperties =
+
CredentialUtils.getBackendPropertiesFromStorageMap(storagePropertiesMap);
+ String tableLocation = baseTable instanceof FileStoreTable
+ ? ((FileStoreTable) baseTable).location().toString()
+ : baseTable.options().get("path");
+ if (!PaimonVectorSearch.vectorScanEligible(baseTable,
tableLocation,
+ backendStorageProperties,
+ Env.getCurrentSystemInfo().getAllClusterBackends(true))) {
+ return null;
+ }
+ } catch (Throwable t) {
+ return null;
+ }
+ // NOTE: the FE-side data-predicate interception is intentionally
MINIMAL.
+ // The data predicate is handed in full to the BE via the vector search
+ // builder's residual filter, where the Rust candidate search stage
applies
+ // it together with deletion vectors. The two planner-side exceptions
above
+ // (unridable conjunct shapes, and the SDK's no-deletion-vector table
mode)
+ // exist because those failures would surface only AFTER the one-way
rewrite
+ // replaced the distance expression. Any other table shape Rust cannot
+ // handle correctly is Rust's responsibility to reject or answer
correctly,
+ // not the planner's.
+
+ // paimon-rust's vector index operates on f32, so carry the query as
floats
+ // rather than List<Double>: it avoids a double→float narrowing on the
wire
+ // and lets PaimonScanNode encode straight into a FLOAT32
TSearchVector.
+ List<Literal> vectorLiterals = ((ArrayLiteral) right).getValue();
+ float[] annQueryVector = new float[vectorLiterals.size()];
+ for (int i = 0; i < vectorLiterals.size(); i++) {
+ annQueryVector[i] = (float) vectorLiterals.get(i).getDouble();
Review Comment:
[P2] Decline non-finite query-vector literals
For a matching 2-D l2-indexed table containing one finite `vec=[0,0]`, this
reduced plan reaches the loop:
```text
TopN(dis ASC, 1)
Project(id, dis := l2_distance_approximate(
vec, [CAST('infinity' AS FLOAT), CAST(0 AS FLOAT)]))
PaimonScan
```
FE casts and array folding preserve the infinite coordinate, and only NULL
coordinates were excluded above. Ordinary `FunctionArrayDistance` returns that
row with an infinite distance, but the pinned Rust
`PkVectorSearchParams::resolve` calls
[validate_query](https://github.com/apache/paimon-rust/blob/0139f7a064c286bc6f03e3e33e4c91a0b003f62d/crates/paimon/src/vindex/pkvector/exact.rs),
which rejects non-finite coordinates with `query vector element ... must be
finite`. The rewrite has already replaced the original distance function and
cannot fall back. Check `Float.isFinite` on the extracted coordinates and keep
ordinary execution for these supported inputs, including NaN and inner-product
queries.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/source/PaimonScanNode.java:
##########
@@ -982,72 +1029,190 @@ && splitHasOrcFile((DataSplit) split)
// rust. Gate until the rust leaf has a Variant Arrow decoder.
boolean projectedVariant = desc.getSlots().stream()
.anyMatch(slot ->
PaimonUtil.containsVariant(slot.getType()));
- // Scheme capability gate: the pinned paimon-rust storage
- // dispatcher (io/storage.rs) selects the FileIO parser from the
- // table location's URI scheme, and libpaimon_c.a compiles in
- // separate COS, OBS, GCS and Azdls parsers besides the OSS and S3
- // ones. Doris normalizes every object store's credentials into
- // the AWS_* / use_path_style aliases (see the *Properties storage
- // classes), which the BE rust bridge translates only into the
- // fs.oss.* and s3.* key families — a cosn:// / obs:// / gs:// /
- // abfs:// warehouse would reach its scheme's parser without the
- // key family it reads (fs.cosn.userinfo.*, fs.obs.*, gcs.*,
- // azure.*) and fail the open instead of using JNI. Only the
- // schemes whose property translation is implemented and
- // open-tested (s3 / s3a / oss, via RUST_VERIFIED_LOCATION_SCHEMES)
- // plus the credential-free hdfs and local-filesystem parsers stay
- // rust-eligible; every other scheme falls back to JNI. A null
- // location also routes to JNI: the rust path needs the
- // paimon_table that only a real location can provide (BE rejects
- // a split without it).
- boolean schemeCapabilityVerified =
isRustVerifiedLocationScheme(source.getTableLocation());
- // An hdfs:// location is scheme-verified only together with the
credential-free
- // backend shape: the backend storage properties that ship to BE
also carry an
- // HDFS catalog's authentication (kerberos principal / keytab,
proxy user, HA
- // nameservice config), none of which the pinned rust HDFS parser
reads — the
- // scan would open as the BE process's ambient identity instead of
the
- // catalog's configured one and fail the access JNI honors. See
- // isRustVerifiedHdfsBackend.
- boolean hdfsBackendVerified =
!isHdfsLocationScheme(source.getTableLocation())
- || isRustVerifiedHdfsBackend(backendStorageProperties);
- // With merge-on-read=true the whole table already routes to JNI
(dvMergeOnRead);
- // for the remaining partial-update/aggregation DV tables, a split
that is
+ // With merge-on-read=true the whole table already routes to JNI
+ // (rustEligibility.dvMergeOnRead); for the remaining
+ // partial-update/aggregation DV tables, a split that is
// not fully materialized (uncompacted level-0 data, or
retractions not
// known to be applied — even a split with no deletion file
attached yet)
// fails the rust is_fully_materialized_pk_dv guard, so it falls
back per
// split instead of turning into a BE-open failure. nativeSplit and
// !fallbackRead only guard the cast — those splits already route
to JNI.
- boolean splitDvNotMaterialized = puAggDeletionVectors &&
!dvMergeOnRead
+ boolean splitDvNotMaterialized =
rustEligibility.puAggDeletionVectors
+ && !rustEligibility.dvMergeOnRead
&& nativeSplit && !fallbackRead
&& !isFullyMaterializedPkDvSplit((DataSplit) split);
+ boolean vectorSearchSplit = paimonSplit.getVectorPayloadBytes() !=
null;
+ // Session, shared table / storage / backend, and scan-shape
gates. The
+ // shared PaimonRustEligibility computation is also the
+ // PushDownVectorTopNIntoPaimonScan rewrite's pre-flight
+ // (PaimonVectorSearch.vectorScanEligible): the rewrite is
one-way, so both
+ // sites must agree on the same gates, and the scan node stays
authoritative
+ // — it evaluates the relation-option-processed table, so a
t@options
+ // override the raw-table pre-flight cannot see still rejects here
(loudly,
+ // as the vector path has no JNI fallback). !isIncremental and the
+ // split-half of the fallback-read gate are the same backstops: an
+ // incremental scan has no PK-vector semantics (the VectorScan
builder has
+ // no incremental mode), and a FallbackSplit shape cannot ride the
vector
+ // payload either — both fail loudly here rather than silently
returning
+ // snapshot rows for a query that asked for changes. Each gate's
value is
+ // also what the fallback logging below reports.
boolean canUseRust = sessionVariable.isEnablePaimonRustReader()
- && sessionVariable.enableFileScannerV2 && nativeSplit &&
!fallbackRead
- && !isIncremental && providerModeTranslatable &&
!queryAuthTable
- && !restTokenTable
- && !dvMergeOnRead && !splitDvNotMaterialized
- && !orcLtzSchema && !projectedVariant &&
!needsNanosecondReader() && !externalFileSplit
- && !deduplicateIgnoreDelete && !rustUnsupportedMergeOption
- && schemeCapabilityVerified && hdfsBackendVerified
- && paimonFileStoreTable != null;
- if (canUseRust) {
- // Branch tables can retain persisted modes without a
selector. Validate the
- // transported options too: the pinned Rust builder only
accepts default here.
- String scanMode =
PaimonScanParams.withoutTimeTravelSelectors(paimonFileStoreTable.schema())
- .options().get(CoreOptions.SCAN_MODE.key());
- canUseRust = scanMode == null ||
"default".equalsIgnoreCase(scanMode);
- }
- if (canUseRust) {
- // Every candidate can receive this range; older BEs cannot
decode reader type 3.
- canUseRust = !backendPolicy.getBackends().isEmpty()
- &&
backendPolicy.getBackends().stream().allMatch(Backend::isPaimonRustReaderSupported);
+ && sessionVariable.enableFileScannerV2
+ && rustEligibility.eligible()
+ && !fallbackRead && !isIncremental
+ && !orcLtzSchema && !projectedVariant &&
!needsNanosecondReader();
+ // The remaining gates judge the DataSplit the BATCH reader opens:
the
+ // native DataSplit shape, unmaterialized partial-update /
aggregation
+ // deletion vectors, external-path files, and the
+ // PaimonRustReaderCapabilities whitelist. A PK-vector search
split is
+ // never read as a DataSplit (it rides serialized in
vector_payload and
+ // executes paimon-rust's eager Top-K chain, which applies deletion
+ // vectors and the merge inside the search itself), so it does not
+ // consult them.
+ //
+ // The nested-projection bound is NOT DataSplit-specific: the ABI
+ // projects only root names and the Arrow struct SerDe binds
children by
+ // ordinal, so a pruned complex slot (nested pruning, on by
default)
+ // would return the wrong child on the vector path exactly as it
would
+ // on the batch path. The rewrite pre-flights this and declines;
this
+ // is the loud backstop for shapes the raw-table pre-flight cannot
see.
+ boolean vectorSchemaIncompatible = false;
+ if (vectorSearchSplit) {
+ canUseRust = canUseRust && rustEligibility.eligible()
+ &&
PaimonRustReaderCapabilities.hasFullNestedProjection(desc);
+ if (canUseRust) {
+ if (rustReaderCapabilities == null) {
+ rustReaderCapabilities = new
PaimonRustReaderCapabilities(paimonFileStoreTable, desc);
+ }
+ // Per-file schema-compat backstop mirroring the rewrite's
+ // hasRustIncompatibleSchemaHistory pre-flight: the vector
+ // reader materializes projected rows through the same
Arrow
+ // decoders, so a historically incompatible file must stay
off
+ // the vector path. Can only fail for shapes the pre-flight
+ // could not see (options-selected schemas, mid-statement
+ // ALTER races) — loud, like every vector backstop.
+ vectorSchemaIncompatible =
+ !rustReaderCapabilities.canVectorRead((DataSplit)
split);
+ canUseRust = canUseRust && !vectorSchemaIncompatible;
+ }
}
- if (canUseRust) {
- if (rustReaderCapabilities == null) {
- rustReaderCapabilities = new
PaimonRustReaderCapabilities(paimonFileStoreTable, desc);
+ if (!vectorSearchSplit && canUseRust) {
Review Comment:
[P2] Preserve external-file eligibility for vector scans
This applies `!externalFileSplit` only to ordinary scans, although vector
materialization also uses the warehouse-selected Rust FileIO. A supported
catalog with `paimon.resolving-file-io.enabled=true`, both credential sets and
an S3 warehouse can store PK-vector data under `oss://...` using
`data-file.external-paths` and its round-robin strategy. Java
[ResolvingFileIO](https://github.com/apache/paimon/blob/3a5ffe7870697e2c2605e63413fb5ac5800aae7e/paimon-common/src/main/java/org/apache/paimon/fs/ResolvingFileIO.java)
dispatches each URI to its matching backend, so the ordinary JNI query works.
`TopN(dis ASC,1) -> Project(id,dis:=l2_distance_approximate(vec,q)) ->
PaimonScan` is now rewritten, but the Rust reader opens that external data path
through
[Storage::S3](https://github.com/apache/paimon-rust/blob/0139f7a064c286bc6f03e3e33e4c91a0b003f62d/crates/paimon/src/io/storage.rs),
which rejects the OSS scheme. Projected user columns require this
materialization even with complete ind
ex coverage. Decline unsupported external storage before replacing the
distance expression, or provide per-file storage routing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
