fvaleye opened a new pull request, #1571: URL: https://github.com/apache/iceberg-rust/pull/1571
## Which issue does this PR close? - Closes #1551 ## What changes are included in this PR? My general approach: - Non-blocking rollover: new files start immediately while old ones close concurrently, if we reach the limit - Controlled concurrency: uses `max_concurrent_closes` to prevent memory exhaustion. By default, the default behavior is kept by having only one background task running. - Resource management: Automatic cleanup and proper error propagation when calling `close()` The main idea suggested by the issue is to improve the `RollingFileWriter` to close files in the background, improving performance by allowing writes to continue without waiting for the previous file to be fully closed. Key changes: - A JoinSet of all closing futures is now used to manage background closing tasks. - When rolling over, the writer spawns a task to close the previous file and immediately starts writing to a new one. - The close method now waits for all background tasks to complete before returning. - Concurrency can be configured via a parameter to control how many files are closed in parallel. ## Are these changes tested? Yes, with unit-testing and local benchmark tests -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
