chinmay-bhat commented on PR #787:
URL: https://github.com/apache/iceberg-python/pull/787#issuecomment-2161194170
> Being able to configure (and also disable) the cachine would be a very
nice touch
Hi @Fokko, I'm curious to know why we would want to allow custom sized
manifest caches.
I agree user should be able to disable the cache.
Here's my initial implementation for configuring and disabling cache:
```python
<top of file>
MANIFEST_CACHE_SIZE = "manifest-cache-size"
config = Config()
manifest_cache_size = config.get_int(MANIFEST_CACHE_SIZE) if
config.get_int(MANIFEST_CACHE_SIZE) else 128
....
@lru_cache(maxsize=manifest_cache_size)
def _manifests(io: FileIO, manifest_list: str) -> List[ManifestFile]:
"""Return the manifests from the manifest list."""
file = io.new_input(manifest_list)
return list(read_manifest_list(file))
class Snapshot:
....
def manifests(self, io: FileIO, use_cache: bool = True) ->
List[ManifestFile]:
"""Return the manifests for the given snapshot."""
if self.manifest_list:
if use_cache:
return _manifests(io, self.manifest_list)
else:
_manifests.cache_clear()
return _manifests.__wrapped__(io, self.manifest_list)
return []
```
Is this in the right direction?
referencing: [how to disable
lru_cache](https://stackoverflow.com/a/56544479) and [lru_cache
doc](https://docs.python.org/3/library/functools.html#functools.lru_cache)
> Looks like the CI is still a bit sad :(
@Fokko @HonahX I need help resolving this error. Error seems unrelated to
the changes in this PR. I've added more info in this
[comment](https://github.com/apache/iceberg-python/pull/787#issuecomment-2157626790).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org
