andormarkus commented on issue #2325: URL: https://github.com/apache/iceberg-python/issues/2325#issuecomment-3217285108
**Update: Cache Clearing Shows Significant Improvement** Following up on my previous comment - I have more comprehensive results after running 75+ invocations. **Key Finding:** The manual cache clearing **does work significantly**, but not completely. **Updated Results:** - **Major success**: Memory now **stabilizes** instead of continuously growing to OOM - **Prevents crashes**: No more Lambda timeouts due to memory limits - **Partial recovery**: Stabilizes at 1600-1800MB range instead of expected 1500MB baseline <img width="1568" height="969" alt="Image" src="https://github.com/user-attachments/assets/43c376df-e76c-4d66-a727-b7d8b0e90d9e"; /> **Interpretation:** The `_manifests` cache clearing prevents the primary leak, but there's still ~200-300MB of persistent allocations that don't get cleared back to baseline. This aligns with @Declow's observation that "even if I remove the cache completely it still increases." **Production Impact:** - Before: Continuous growth → OOM crashes within 5 invocations - After: Stable at higher plateau → sustainable for production The cache clearing is definitely part of the solution - it transforms a continuous leak into a bounded increase. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
