sunchao commented on code in PR #6059:
URL: https://github.com/apache/datafusion-comet/pull/6059#discussion_r4199407148
##########
spark/src/main/scala/org/apache/comet/objectstore/NativeConfig.scala:
##########
@@ -214,6 +216,13 @@ object NativeConfig {
}
val s3CompliantSchemes = parseSchemeSet(s3CompliantRaw)
+ // ABFS: Hadoop resolves the mechanism and values for this URI's account
on the driver, and
+ // only those travel to native (never another account's keys, never the
whole namespace).
+ if (scheme == "abfs" || scheme == "abfss") {
+ return options.toMap ++ AbfsAuthResolver.toOptions(
Review Comment:
[P2] Preserve credentials for every account used by the scan.
`CometNativeScan` calls this method only for `firstFileUri` and stores the
result in scan-wide options. Previously those options retained all
account-scoped keys, allowing each task’s native store to select its own
account’s key. This return now retains only the first account’s resolved values
under global names. A `spark.read.parquet(pathA, pathB)` scan with distinct
account keys therefore supplies A’s key to B’s store, even when the files
occupy separate `FilePartition`s. Spark and the base select B’s key, while this
head causes authentication failures for B. Could resolution preserve
account/container identity through task planning, or make such scans fall back
instead of reusing the first credential?
Evidence: Reproduced with exact base/head NativeConfig and Azure
construction code. Configure
`fs.azure.account.key.accta.dfs.core.windows.net=YWNjb3VudC1h` and
`fs.azure.account.key.acctb.dfs.core.windows.net=YWNjb3VudC1i`, using
`abfss://[email protected]/a.parquet` and the corresponding acctb
URL. Both Hadoop filesystems initialize offline on 3.3.4, 3.4.1, 3.4.2 and
3.5.0. For B’s URL, actual object_store credential inspection selects B with
base options extracted from A, but A with head options extracted from A.
Resolving B separately restores B. Source tracing confirms scan-wide extraction
at CometNativeScan.scala:365 and reuse with each partition’s URL at
native/core/src/execution/planner.rs:1955. Rust reproduction:
`CARGO_TARGET_DIR=/tmp/comet6059-target cargo test --offline --locked
--manifest-path /tmp/comet6059-42772-verify-as3o1c5h/Cargo.toml --lib
review::second_account_reuses_first_accounts_key -- --nocapture`.
Instrumentation only exposes the concre
te store for credential inspection. No Azure requests were made.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
