alexanderbianchi opened a new pull request, #3588: URL: https://github.com/apache/iceberg-rust/pull/3588
## Which issue does this PR close? Related to #3276 and #3277. No issue is closed by this draft. Depends on #3587. This branch currently includes that prerequisite commit. The data-file change is the second commit; [review the incremental diff here](https://github.com/alexanderbianchi/iceberg-rust/compare/codex/delete-decoding-cpu...codex/data-decoding-cpu). Rebase onto main after #3587 merges. ## What changes are included in this PR? Data-file footer and page-index decoding currently run on I/O workers, while page decoding runs on whichever runtime polls the scan. Move the complete file-open/planning phase and Parquet stream consumption onto the configured CPU runtime. Storage opens and byte-range reads continue to use the I/O adapter from #3587. Keep opening/planning in spawned, abort-on-drop tasks so buffered opens and bloom-filter fetches continue progressing when the consumer polls another file. Run bloom-filter evaluation directly inside that CPU task. Deliver decoded batches through a bounded channel per file. Reserve capacity before polling Parquet, allowing at most one output batch ahead of the consumer. Retain an abort-on-drop producer handle with the returned stream so dropping a scan cancels outstanding work. Await the producer at EOF to propagate decoding failures, panics, and runtime shutdown instead of silently truncating results. Existing file-concurrency controls are retained. The channel bounds output batches, not Parquet's internal row-group buffers. This adds CPU task/channel overhead; no throughput benchmark has been run. ## Are these changes tested? - `cargo test -p iceberg --lib arrow::`: 291 passed. - `cargo test -p iceberg --lib scan::`: 104 passed. - New tests cover CPU stream polling with an I/O-runtime consumer, bounded decoding ahead of the consumer, cancellation while waiting for capacity or decoding, propagation of decoding errors and panics, and a real Parquet scan with separate runtimes that asserts storage opens and reads execute on I/O workers. - Existing open-progress and cancellation regressions from #3277 pass. - `cargo clippy -p iceberg --lib --tests -- -D warnings`: passed. - `cargo fmt --all -- --check`: passed. ## AI Disclosure Implementation, tests, and this description were prepared with OpenAI Codex assistance. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
