sdd commented on PR #806: URL: https://github.com/apache/iceberg-rust/pull/806#issuecomment-2556556568
I'm not sure this is what we want. `try_buffer_unordered` executes futures **concurrently** according to the docs: https://docs.rs/futures/latest/futures/stream/trait.TryStreamExt.html#method.try_buffer_unordered This does not necessarily mean **in parallel**, which is what we want. I encountered a similar thing when we were originally using [try_for_each_concurrent](https://docs.rs/futures/latest/futures/stream/trait.TryStreamExt.html#method.try_for_each_concurrent) here. As per the note in the second paragraph of the docs [here](https://docs.rs/futures/0.3.25/futures/stream/trait.StreamExt.html#method.for_each_concurrent), no threads are introduced. The executor switches between each future on a single thread. We're CPU bound in the reader rather than IO-bound like in the planning phase, and so we want true parallelism. The double-spawn approach is not as neat but it ensures parallelism. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
