singhpk234 commented on PR #12827: URL: https://github.com/apache/iceberg/pull/12827#issuecomment-2814270644
Thanks @danielcweeks, my understanding for the approaches are the following : For 1, My only concern with the following was each client maintaining its own connection pool, with large number of prefixes, which would not be scalable IMHO, if we can share the pool across all the s3 client it would be handle this concern. I am not sure if that something we do today. For 2, Prefix level credProvider, are we thinking of creating a cred provider per path, and each cred provider will do a refresh logic and get the cred for the the path it corresponds to ? this would trigger a lot of request to the creds endpoint ? I understand the ExecutionInterceptor is very specific to AWS, and touches the request at a very low level, but it helps in not having to manage a lot of connections to s3 (per prefix) or making a lot of calls to creds refresh endpoint. It is never the less something S3A supports, though not for cred vending for audit logging : https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/auditing.html Sorry for hopping in this discussion, I wanted to contribute here based on what i have seen for solving these cred vending issues in past ! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
