swanandx opened a new pull request, #3277: URL: https://github.com/apache/iceberg-rust/pull/3277
## Which issue does this PR close? - Closes #3276 ## What changes are included in this PR? <!-- Provide a summary of the modifications in this PR. List the main changes such as new features, bug fixes, refactoring, or any other updates. --> Open each file inside the stream that reads it, and drop the `try_buffer_unordered`. Every file is now opened by the stream that reads its rows, so each open is followed straight away by the read. This halves how many files can be open at once, from twice `concurrency_limit_data_files` down to `concurrency_limit_data_files`. The scan no longer opens the next file while reading the current one. I expect that to cost little, since the other files in the batch keep reading while one file's footer is fetched, but I have not benchmarked it against a remote store. The other option is to bring back the `spawn` and bounded channel that #806 replaced, which keeps the prefetch. That is a bigger change and needs care to abort the tasks on drop. Happy to do that one instead if you prefer it. ## Are these changes tested? <!-- Specify what test covers (unit test, integration test, etc.). If tests are not included in your PR, please explain why (for example, are they covered by existing tests)? --> Yes. `flow_control_tests.rs` runs a scan over one HTTP/2 connection modelled as a `Storage`. No network. `test_concurrent_scan_does_not_starve_shared_connection` stalls on main and passes with this change. It uses concurrency 4; against this harness anything from 3 to 10 stalls. `test_sequential_scan_over_shared_connection` is a control. Concurrency 1 takes the reader's other branch, so it passes either way. It is there to show the harness itself does not stall. ## AI Disclosure <!-- https://iceberg.apache.org/contribute/#guidelines-for-ai-assisted-contributions --> Claude Code (Claude Opus 5) helped analyse the stall and draft the test harness. I reviewed every line, and confirmed the test fails without the fix -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
