gaborkaszab opened a new issue, #11766: URL: https://github.com/apache/iceberg/issues/11766
### Proposed Change There are clients of the Iceberg table format (e.g. query engines) that cache table metadata. In order to keep the cache up-to-date they implement different mechanisms like event processing (HMS with Impala) or simply do a full table load on each request. This proposal introduces a way to perform an actual table load only if there is a change on the table metadata since the last request. There is a new Iceberg Catalog level API proposed, and this proposal also describes the implementation details for the REST catalog including the changes required for the REST spec. Typical use case this would solve: - Engine receives a query for a particular table - Engine doesn't have this table in the cache so loads it from an Iceberg Catalog (REST in this proposal) - Engine gets another request for the same table - Engine does a freshness aware loading for this table. Full table loading is only performed if the table has changed since. If the table hasn't changed the engine can use the one in it's cache. ### Proposal document https://docs.google.com/document/d/1rnVSP_iv2I47giwfAe-Z3DYhKkKwWCVvCkC9rEvtaLA ### Specifications - [X] Table - [ ] View - [X] REST - [ ] Puffin - [ ] Encryption - [X] Other -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
