pzhdfy commented on code in PR #69493:
URL: https://github.com/apache/doris/pull/69493#discussion_r4374809049
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/source/PaimonScanNode.java:
##########
@@ -982,72 +1029,190 @@ && splitHasOrcFile((DataSplit) split)
// rust. Gate until the rust leaf has a Variant Arrow decoder.
boolean projectedVariant = desc.getSlots().stream()
.anyMatch(slot ->
PaimonUtil.containsVariant(slot.getType()));
- // Scheme capability gate: the pinned paimon-rust storage
- // dispatcher (io/storage.rs) selects the FileIO parser from the
- // table location's URI scheme, and libpaimon_c.a compiles in
- // separate COS, OBS, GCS and Azdls parsers besides the OSS and S3
- // ones. Doris normalizes every object store's credentials into
- // the AWS_* / use_path_style aliases (see the *Properties storage
- // classes), which the BE rust bridge translates only into the
- // fs.oss.* and s3.* key families — a cosn:// / obs:// / gs:// /
- // abfs:// warehouse would reach its scheme's parser without the
- // key family it reads (fs.cosn.userinfo.*, fs.obs.*, gcs.*,
- // azure.*) and fail the open instead of using JNI. Only the
- // schemes whose property translation is implemented and
- // open-tested (s3 / s3a / oss, via RUST_VERIFIED_LOCATION_SCHEMES)
- // plus the credential-free hdfs and local-filesystem parsers stay
- // rust-eligible; every other scheme falls back to JNI. A null
- // location also routes to JNI: the rust path needs the
- // paimon_table that only a real location can provide (BE rejects
- // a split without it).
- boolean schemeCapabilityVerified =
isRustVerifiedLocationScheme(source.getTableLocation());
- // An hdfs:// location is scheme-verified only together with the
credential-free
- // backend shape: the backend storage properties that ship to BE
also carry an
- // HDFS catalog's authentication (kerberos principal / keytab,
proxy user, HA
- // nameservice config), none of which the pinned rust HDFS parser
reads — the
- // scan would open as the BE process's ambient identity instead of
the
- // catalog's configured one and fail the access JNI honors. See
- // isRustVerifiedHdfsBackend.
- boolean hdfsBackendVerified =
!isHdfsLocationScheme(source.getTableLocation())
- || isRustVerifiedHdfsBackend(backendStorageProperties);
- // With merge-on-read=true the whole table already routes to JNI
(dvMergeOnRead);
- // for the remaining partial-update/aggregation DV tables, a split
that is
+ // With merge-on-read=true the whole table already routes to JNI
+ // (rustEligibility.dvMergeOnRead); for the remaining
+ // partial-update/aggregation DV tables, a split that is
// not fully materialized (uncompacted level-0 data, or
retractions not
// known to be applied — even a split with no deletion file
attached yet)
// fails the rust is_fully_materialized_pk_dv guard, so it falls
back per
// split instead of turning into a BE-open failure. nativeSplit and
// !fallbackRead only guard the cast — those splits already route
to JNI.
- boolean splitDvNotMaterialized = puAggDeletionVectors &&
!dvMergeOnRead
+ boolean splitDvNotMaterialized =
rustEligibility.puAggDeletionVectors
+ && !rustEligibility.dvMergeOnRead
&& nativeSplit && !fallbackRead
&& !isFullyMaterializedPkDvSplit((DataSplit) split);
+ boolean vectorSearchSplit = paimonSplit.getVectorPayloadBytes() !=
null;
+ // Session, shared table / storage / backend, and scan-shape
gates. The
+ // shared PaimonRustEligibility computation is also the
+ // PushDownVectorTopNIntoPaimonScan rewrite's pre-flight
+ // (PaimonVectorSearch.vectorScanEligible): the rewrite is
one-way, so both
+ // sites must agree on the same gates, and the scan node stays
authoritative
+ // — it evaluates the relation-option-processed table, so a
t@options
+ // override the raw-table pre-flight cannot see still rejects here
(loudly,
+ // as the vector path has no JNI fallback). !isIncremental and the
+ // split-half of the fallback-read gate are the same backstops: an
+ // incremental scan has no PK-vector semantics (the VectorScan
builder has
+ // no incremental mode), and a FallbackSplit shape cannot ride the
vector
+ // payload either — both fail loudly here rather than silently
returning
+ // snapshot rows for a query that asked for changes. Each gate's
value is
+ // also what the fallback logging below reports.
boolean canUseRust = sessionVariable.isEnablePaimonRustReader()
- && sessionVariable.enableFileScannerV2 && nativeSplit &&
!fallbackRead
- && !isIncremental && providerModeTranslatable &&
!queryAuthTable
- && !restTokenTable
- && !dvMergeOnRead && !splitDvNotMaterialized
- && !orcLtzSchema && !projectedVariant &&
!needsNanosecondReader() && !externalFileSplit
- && !deduplicateIgnoreDelete && !rustUnsupportedMergeOption
- && schemeCapabilityVerified && hdfsBackendVerified
- && paimonFileStoreTable != null;
- if (canUseRust) {
- // Branch tables can retain persisted modes without a
selector. Validate the
- // transported options too: the pinned Rust builder only
accepts default here.
- String scanMode =
PaimonScanParams.withoutTimeTravelSelectors(paimonFileStoreTable.schema())
- .options().get(CoreOptions.SCAN_MODE.key());
- canUseRust = scanMode == null ||
"default".equalsIgnoreCase(scanMode);
- }
- if (canUseRust) {
- // Every candidate can receive this range; older BEs cannot
decode reader type 3.
- canUseRust = !backendPolicy.getBackends().isEmpty()
- &&
backendPolicy.getBackends().stream().allMatch(Backend::isPaimonRustReaderSupported);
+ && sessionVariable.enableFileScannerV2
+ && rustEligibility.eligible()
+ && !fallbackRead && !isIncremental
+ && !orcLtzSchema && !projectedVariant &&
!needsNanosecondReader();
+ // The remaining gates judge the DataSplit the BATCH reader opens:
the
+ // native DataSplit shape, unmaterialized partial-update /
aggregation
+ // deletion vectors, external-path files, and the
+ // PaimonRustReaderCapabilities whitelist. A PK-vector search
split is
+ // never read as a DataSplit (it rides serialized in
vector_payload and
+ // executes paimon-rust's eager Top-K chain, which applies deletion
+ // vectors and the merge inside the search itself), so it does not
+ // consult them.
+ //
+ // The nested-projection bound is NOT DataSplit-specific: the ABI
+ // projects only root names and the Arrow struct SerDe binds
children by
+ // ordinal, so a pruned complex slot (nested pruning, on by
default)
+ // would return the wrong child on the vector path exactly as it
would
+ // on the batch path. The rewrite pre-flights this and declines;
this
+ // is the loud backstop for shapes the raw-table pre-flight cannot
see.
+ boolean vectorSchemaIncompatible = false;
+ if (vectorSearchSplit) {
+ canUseRust = canUseRust && rustEligibility.eligible()
+ &&
PaimonRustReaderCapabilities.hasFullNestedProjection(desc);
+ if (canUseRust) {
+ if (rustReaderCapabilities == null) {
+ rustReaderCapabilities = new
PaimonRustReaderCapabilities(paimonFileStoreTable, desc);
+ }
+ // Per-file schema-compat backstop mirroring the rewrite's
+ // hasRustIncompatibleSchemaHistory pre-flight: the vector
+ // reader materializes projected rows through the same
Arrow
+ // decoders, so a historically incompatible file must stay
off
+ // the vector path. Can only fail for shapes the pre-flight
+ // could not see (options-selected schemas, mid-statement
+ // ALTER races) — loud, like every vector backstop.
+ vectorSchemaIncompatible =
+ !rustReaderCapabilities.canVectorRead((DataSplit)
split);
+ canUseRust = canUseRust && !vectorSchemaIncompatible;
+ }
}
- if (canUseRust) {
- if (rustReaderCapabilities == null) {
- rustReaderCapabilities = new
PaimonRustReaderCapabilities(paimonFileStoreTable, desc);
+ if (!vectorSearchSplit && canUseRust) {
Review Comment:
Fixed in 59042bb: the external-file gate now applies to the vector path —
the scan node checks the split's external files in the base rust gate (the
`external_file_split` fallback reason is now also reported for vector
splits), and the rewrite pre-flight declines when the catalog (or the
table's options) opts into the resolving-file-io layout, since the rust
table's single FileIO rejects foreign schemes (an oss:// external path on an
S3 table). Covered by testResolvingFileIoCatalogNotPushedDown.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
