manuzhang opened a new pull request, #3382: URL: https://github.com/apache/iceberg-rust/pull/3382
## Which issue does this PR close? Closes #1744. ## What changes are included in this PR? Fanout writers keep every partition writer open, so high partition cardinality can consume excessive memory. Add an opt-in nonzero limit: close the least recently used writer before opening its replacement, retain its output, and allow later writes to reopen that partition. The existing constructor remains unlimited. The limit bounds active writers, while completed file metadata remains buffered. Eviction can produce more, smaller files. ## Are these changes tested? All 10 partitioning tests pass, covering unlimited behavior, limits of one and two, LRU eviction, reopened partitions, output preservation, empty input, and close/build failures. Real Parquet coverage checks file counts and per-partition record totals with and without a limit. Formatting, all-target/all-feature Clippy for the iceberg crate, Cargo Machete, locked workspace dependency validation, and license packaging checks pass. The public API snapshot adds only the new constructor. ## AI Disclosure Codex assisted with the implementation and tests. Review the assumptions that the limit should be opt-in and that LRU eviction is appropriate; the issue does not specify a default limit or eviction policy. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
