okumin commented on PR #14137: URL: https://github.com/apache/iceberg/pull/14137#issuecomment-3322309443
@gaborkaszab As you say, the original motivation is exactly for REST on HMS, but the sentence "the original approach didn't work" could be revised a bit. Any options, such as where we add the caching layer (I'd say the HMS-specific approach is still valid), or any detailed configurations, such as the recommended cache size, are open and flat. I'd say we can potentially have the following three possibilities. (A) Let's say caching table metadata is legal and practical for REST or other use cases. In this case, reusing the same logic as [the manifest caching](https://github.com/apache/iceberg/pull/4518) might make sense; users would have a consistent experience across table metadata and manifests(and potentially manifest list files in the future), and it can be reused by other systems, e.g., REST Catalog over JDBC Catalog. I created this PR to demonstrate that direction. (B) Let's say caching table metadata is both legal and practical, but the caching layer should not be part of iceberg-core. In this case, I would close this PR, and we will implement caching only in REST over HMS. (C) Let's say caching table metadata is illegal, e.g., it violates the ACID semantics, or impractical, e.g., table metadata is typically too extensive to cache. In this case, we should reconsider whether we really need HIVE-29035. I currently assume caching table metadata is practical and (A) can outperform (B) because the logic is not so specific to Hive Metastore. If we can agree with this point, I will send an email to talk about if iceberg-core should be able to cache table metadata and the manifest list. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
