nastra commented on PR #12827: URL: https://github.com/apache/iceberg/pull/12827#issuecomment-2821362593
> @nastra I'm a little concerned about this approach because we're doing a lot of hand-crafting/manipulation of the request as opposed to using features of the SDK. There are two alternatives I can think of: > > 1. Update `S3FileIO::client()` to take a path: `S3FileIO::client(<path>)` then fetch or create a new S3Client with the best matched StorageCredential. This does create more clients, but they're light-weight and we can even share the underlying http client. > 2. Use the request-level credentials provider override. Each request (e.g. `s3.getObject(...)`) supports a request-level override that includes setting a credential provider for that specific request (`r -> r.overrideConfiguration(o -> o.credentialsProvider(storageCredentialProvider))`) > > While the `ExecutionInterceptor` may work, it's manipulating the request at a much lower level that makes me concerned that we'll run into issues. > > I think I prefer #1 above because it is an approach we can replicate for the GCSFileIO and ADLSFileIO. The more similar the implementations, the less likely we'll have inconsistent handling. The idea of this PR was to just explore how something with an `ExecutionInterceptor` would look like. While the approach requires less code and seems simpler, I do agree that we're modifying things at a lower level, which could introduce unwanted side-effects/bugs. There's also https://github.com/apache/iceberg/pull/12799 which effectively does what you described in Option 1. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
