Anurag-S1ngh commented on PR #4335: URL: https://github.com/apache/iceberg-python/pull/4335#issuecomment-6813687083
Thanks Fokko. No, I have not run into it myself. I picked this up from #4300, where the repeated reads of the manifest list and of every manifest on each plan are described. On memory, it is off by default and bounded: 100 MiB total, 8 MiB per entry, LRU eviction when full, and entries expire after 60s. Anything over the per-entry limit is not cached at all, so the cache cannot go over the limit, it just stops caching. Only manifest lists and manifests go through it, not data files or metadata.json, and all four limits are configurable. If the defaults are still too generous I can lower them, or drop the in-memory cache and keep only the override point on FileIO. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
