singhpk234 opened a new pull request, #13716:
URL: https://github.com/apache/iceberg/pull/13716
### About the change
Credentials refresh is a big pain when using long running jobs, what this
means is credentials vended by table during the load table can expire and in
this situation the credentials needs to be refreshed otherwise the whole job
fails.
A lot of work has been done in past to enable refresh by introducing a
credentials endpoint and attaching a credential provider which can talk to this
endpoint and fetch new credentials when this is closer to expiration, these can
be configured presently by :
client.refresh-credentials-endpoint and client.refresh-credentials-enabled
catalog configuration
One important issue here is that the credentials endpoint is that its table
scoped
```
public static final String V1_TABLE_CREDENTIALS =
"/v1/{prefix}/namespaces/{namespace}/tables/{table}/credentials";
```
what this means is client side even though it can override / configure but
it can only configure it one table at a time so if a job references more than
one table, then this would mean other table would fail.
Now an argument can be made can this property be returned by catalog as part
of loadTable call, well it can but the prefix that the client would be using
its not something a service would be aware of never the less this is more or
less a fixed url, rather than a dynamically generated url
So based on my understanding, if the catalog supports /creds endpoint which
it indicates in the /config call and the client side configuration says
refresh the creds via catalog prop then before making the table file io one can
just create the endpoint for refresh and pass it to fileIO to make sure a
credential provider is initialized to make e2e refresh work.
### Testing
Pending
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
