Declow commented on issue #2325: URL: https://github.com/apache/iceberg-python/issues/2325#issuecomment-3195414286
@kevinjqliu It makes sense that the list of manifest files keeps increasing as the write continues but it is not root cause. As long as the function is cached the AvroFileHeader is never GC'ed. These objects are created each time a AvroFile is read inside of read_manifest_list. If you run the script with any cache size it will continue to increase the memory usage. /avro/reader.py:330: size=116 KiB, count=1190, average=100 B If you run the script for more than 50 iterations. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
