stevenzwu commented on PR #8555: URL: https://github.com/apache/iceberg/pull/8555#issuecomment-1722263748
> Implemenation-wise, a centralized solution for refreshing the table metadata is ideal. The implementation here, where the subtasks perform the refresh, still is works for many cases. If it is a point of contention, we can remove the implementation (ReloadingTableSupplier) but keep the ability to set one. > Implementation-wise, a FileIO-specific API for credential refresh will still add load to an Iceberg REST implementation. Table load with the REST API and refs-only snapshot mode can be fairly efficient. Normally, I would think table load (dozens or hundreds of milliseconds?) is probably a magnitude or two more expensive than credentials refresh (a few milliseconds?). This is where I share the same concern that @pvary raised about every task manager refreshing/loading tables periodically. Can you elaborate a little how REST catalog makes table load efficient? I also echo @pvary 's comment on the confusion btw `TableLoader` and `TableSupplier`. There are actually some questions on the design `TableLoader` (e.g. [not handling well close well](https://github.com/apache/iceberg/pull/6614)). I would be in favor a refactor/redesign of the table loader/catalog lifecycle in Flink. But that will probably be a big discussion by itself. I also thought you use pre-signed urls from REST catalog. with that, what is the credentials expiration problem here? S3 session credentials also have auto-refresh capability. Hence, I am trying to understand more about the credentials expiration and why is it tied to table object and needs refresh. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
