karuppayya opened a new pull request, #4246: URL: https://github.com/apache/datafusion-comet/pull/4246
## Which issue does this PR close? Closes #4245 ## Rationale for this change Follow-up to #4213. The Profile credential provider added in #4213 ignored the two Hadoop S3A configs that select which profile to use: - `fs.s3a.auth.profile.name` - `fs.s3a.auth.profile.file` Per [discussion on #4213](https://github.com/apache/datafusion-comet/pull/4213#issuecomment-4381552760) with @parthchandra, the native side should resolve these itself (as `AssumedRoleCredentialProvider` and `SimpleAWSCredentialsProvider` already do) so the behavior is independent of the user's bundled Hadoop version. The Hadoop `ProfileAWSCredentialsProvider` class itself is only available in Hadoop 3.4.2+, but the AWS SDK profile loader is available regardless. ## What changes are included in this PR? - Read `fs.s3a.auth.profile.name` / `fs.s3a.auth.profile.file` (with bucket-scoped overrides) into the `Profile` credential provider metadata. - Wire them into `ProfileFileCredentialsProvider::builder()` via `profile_name(...)` and `profile_files(ProfileFiles::builder().with_file(Credentials, ...))`. - Add unit tests for the no-config, name+file, and bucket-override cases. - Document the supported options in the S3 datasources guide. ## How are these changes tested? - New unit tests in `native/core/src/parquet/objectstore/s3.rs`: - `test_profile_credential_provider` (existing, updated for the new variant shape) - `test_profile_credential_provider_with_name_and_file` (new) - `test_profile_credential_provider_bucket_overrides` (new) - All 40 tests in `parquet::objectstore::s3` pass locally. - `cargo clippy -p datafusion-comet --lib --tests -- -D warnings` clean. - `cargo fmt --check` clean. Made with [Cursor](https://cursor.com) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
