Xuanwo commented on PR #806: URL: https://github.com/apache/iceberg-rust/pull/806#issuecomment-2556576097
> I'm not sure this is what we want. `try_buffer_unordered` executes futures **concurrently** according to the docs: [docs.rs/futures/latest/futures/stream/trait.TryStreamExt.html#method.try_buffer_unordered](https://docs.rs/futures/latest/futures/stream/trait.TryStreamExt.html#method.try_buffer_unordered) Hi, `try_buffer_unordered` does execute concurrently that multiple futures are been polled at the same time by multiple async runtime. My ultimate goal is to eliminate `concurrency_limit_data_files` from our reading process and handle only `FileScanTasks` to create a `Stream<Item=RecordBatch>`. This approach will allow users to utilize our APIs to implement various reading strategies. Our current implementation prevents the query engine from adopting Push-Based Execution, which should have control over the underlying I/O and CPU operations. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
