sungwy commented on issue #1162: URL: https://github.com/apache/iceberg-python/issues/1162#issuecomment-2361785949
That's a very interesting observation 👀 I'm so curious to understand what's happening. If that's in fact what's happening, I think your proposed solution sounds promising... I'm still trying to understand this issue thoroughly In the lru cached `_manifests` we are caching a list of manifest files, but in fact we are only reading a single manifest list, and deserializing each of the records as `ManifestFile` Records. So the only PyArrowFile that I'd expect to be left 'open' according to this theory would be the manifest_list https://github.com/apache/iceberg-python/blob/de47590c6ac4f507cb2337c20504a62c484339f9/pyiceberg/manifest.py#L623-L639 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
