bryanck commented on PR #8555: URL: https://github.com/apache/iceberg/pull/8555#issuecomment-1719726330
@pvary > My concern here is that we tie ourselves to a "random" place of time to refresh the metadata. In my opinion we will have very specific events where we need refresh the metadata - namely, when we have a record with unexpected schema. The proposed solution does not point to this direction. I wouldn't say it is random. In the committer it happens after a commit. In the writer, it happens when a writer is initialized. > > I think @gaborgsomogyi and my self would be happy to discuss this problems, since the token delegation framework was designed to solve the exact same issues (Kerberos/AWS token refresh) in Flink, I think it would be good to solve the token renewal with it, if it is possible. > I have a working prototype of refreshing the table using the delegation manager. The main complication is that the manager is iniailized as part of the JM/TM initialization, not application initialization. This requires configuring the catalog for the delegation manager independently and introducing a separate configuration. It also required the application run in the same classloader as the JM or Tm. If those can be addressed then it should be easy to plug in a table supplier that reads from that instead of the catalog. > My problem is that having a solution which does not point to the right direction would make future work even harder. > As mentioned above, I think it does not point to the right direction. We need the refresh capability, but we specifically need it with a way to trigger manually. Also refreshing the whole table to get the new credentials seems problematic as well. Which part of this do you feed is not going in the right direction? The main change is introducing an abstraction that allows a table to be refreshed, which is the first step to any solution. The actual reloading table supplier is a very small part if this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
