dwsmith1983 opened a new pull request, #6059: URL: https://github.com/apache/datafusion-comet/pull/6059
## Which issue does this PR close? Closes #5542. ## Rationale for this change The native Azure store seeded its builder from the `AZURE_*` environment and then layered the translated Hadoop `fs.azure.*` keys on top. Same-key collisions resolved to Hadoop, but object_store's `build()` chooses the credential by a fixed order across keys, bearer token, then account key, then workload identity, then client secret, then SAS. An ambient `AZURE_STORAGE_TOKEN` or `AZURE_STORAGE_ACCOUNT_KEY` therefore outranked a configured Hadoop account key or service principal, and the `AZURE_FEDERATED_TOKEN_FILE` the AKS webhook injects into every annotated pod turned a configured client-secret principal into workload identity, dropping the secret. Hadoop's ABFS driver reads no environment variables, so the two readers could resolve different identities for the same table, surfacing as a 403 on tables the stock reader handles, or a read under a credential the job never configured. ## What changes are included in this PR? - `create_store` builds through `build_builder`, which takes the environment as an explicit iterator. When the translated Hadoop configuration carries an auth mechanism, an account key, a SAS token, a federated token file, a client secret or an MSI endpoint, or `fs.azure.account.oauth.provider.type` names Hadoop's `MsiTokenProvider`, no `AZURE_*` environment variable is consulted at all. That covers credentials and transport settings alike: an environment endpoint, proxy, invalid-certificate switch or emulator flag could otherwise redirect or intercept the configured identity, since object_store reads `AZURITE_BLOB_STORAGE_URL` on its own once the emulator flag is set. - When Hadoop configures no mechanism, or only names the identity through the client id and tenant, the full environment applies as before, so workload identity from the AKS webhook keeps working. `apply_env` reads the environment the way `MicrosoftAzureBuilder::from_env` does, including `IDENTITY_ENDPOINT` applied after the `AZURE_*` variables so it wins over `AZURE_MSI_ENDPOINT`. - A partial Hadoop mechanism, such as a token file without a client id and tenant, is not completed from the environment. Hadoop requires the client id and tenant as well, and object_store then falls through its credential chain to the node's managed identity, which the documentation now says. - The module documentation and the Azure section of the data sources guide describe the new precedence. User-facing change: with a Hadoop auth mechanism configured, `AZURE_ALLOW_HTTP`, `AZURE_PROXY_URL`, `AZURE_STORAGE_ENDPOINT` and the emulator flag no longer apply, and no `fs.azure.*` key is translated to them. Transport settings from the environment apply only alongside environment credentials. A Hadoop-keyed way to set a proxy or emulator endpoint can be a follow-up if anyone needs it. ## How are these changes tested? Unit tests in `azure.rs` drive `build_builder` with an explicit environment and inspect the builder with `get_config_value` before any store is built, so nothing reads or mutates the process environment and nothing touches the network. They map to the issue's three scenarios (`env_bearer_token_is_ignored_when_hadoop_sets_account_key`, `env_account_key_is_ignored_when_hadoop_sets_client_secret_principal`, `env_federated_token_file_is_ignored_when_hadoop_sets_client_secret_principal`), keep workload identity from the environment working with and without Hadoop naming the identity (`env_workload_identity_is_used_when_hadoop_has_no_auth`, `hadoop_client_id_and_tenant_still_accept_env_federated_token_file`, `hadoop_workload_identity_provider_type_still_accepts_env_token_file`), and pin the boundaries: transport settings ignored with Hadoop auth and applied without it, `IDENTITY_ENDPOINT` applied only without Hadoop auth and winning over `AZURE_MSI_ENDPOINT` in either order, the MSI pro vider type counting as a mechanism, and a token file alone not borrowing the client id and tenant. `create_store` is also built end to end with only a Hadoop account key. The "ignored" tests fail against unconditional environment seeding, and the ordering and provider-type tests fail against the previous behaviour. No Scala test is added: CI has no Azure account and the decision is entirely native; the Delta contrib declines `abfss` scans, so only the plain Parquet path is affected. No CI label is needed: no serde, planner, shim or diff-file change. Run locally: `cargo test -p datafusion-comet --lib azure` (25 tests), `cargo clippy --all-targets --workspace -- -D warnings`, `cargo fmt`, prettier on the docs page. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
