HonahX commented on issue #570: URL: https://github.com/apache/iceberg-python/issues/570#issuecomment-2033588669
@geruh, thanks for highlighting this issue. The confusion largely stems from the naming convention used when the `profile_name`, `region_name`, `aws_access_key_id`, etc., were introduced in [#7781](https://github.com/apache/iceberg/pull/7781). Initially, these configurations were intended solely for GlueCatalog, but their generic names suggest they might influence both Glue and S3 operations. To address this, we can consider renaming these configurations with a `glue.` prefix (e.g., `glue.profile_name`) to clarify their scope. However, to maintain API compatibility, we may need to support both the new and old naming conventions temporarily. > But on the other hand it seems reasonable that the AWS profile config should work uniformly across both the catalog and filesystem levels. +1 for unified configurations. I think it may be convenient to introduce other unified configurations, with generic names like `aws-access-key-id`. So the overall order of config will be: 1. Client-specific configs: glue.access-key-id, s3.access-key-id, etc. 3. Unified AWS configurations like aws-access-key-id 5. Environment variables and the default AWS config > However, we're currently utilizing PyArrow's [S3FileSystem](https://arrow.apache.org/docs/python/generated/pyarrow.fs.S3FileSystem.html#pyarrow.fs.S3FileSystem), which doesn't inherently support AWS profiles. This means we'd need to bridge that gap manually. Regarding the `profile_name` support for PyArrow's S3FileSystem, it seems there might not be a direct solution from the pyiceberg side. This functionality appears to be more suitably addressed through enhancements to the PyArrow library itself. WDYT? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
