pvary commented on PR #8555: URL: https://github.com/apache/iceberg/pull/8555#issuecomment-1716989137
I am very concerned about this simplified solution. We should consider: - Concurrency issues, when the table metadata has been changed between reloads. Maybe we should only refresh the table metadata during checpoints, so we can localize, and handle changes in one place? - Load issues, as every TaskManager with a writer task connects to the catalog, and does so periodically. This will become costly very soon. I would prioritize a centralized solution. - The current default implementation does not solve the schema evolution problem, since that would mean on-demand table reload instead of periodic reload If the main goal here is to refresh the credentials, then maybe a proxy solution above the accessor could be a better solution, like the https://github.com/apache/hive/blob/master/standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/RetryingMetaStoreClient.java where we get a new Kerberos token when the previous one has expired. What do you think? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
