alexanderbianchi opened a new pull request, #67: URL: https://github.com/apache/datafusion-iceberg/pull/67
## What changes Port the file-planning/execution split from apache/iceberg-rust#2671 to the current `iceberg-datafusion` `DataSourceExec` read path, using **only existing public iceberg-rust APIs**. This is the local counterpart of the separation already used by datafusion-contrib/datafusion-distributed#773; it does not replace DFD's work-unit feed or worker reader. With `iceberg_local.plan_files=true`, both table providers call `TableScan::plan_files()` during `TableProvider::scan()`, retain the selected `FileScanTask`s under a query memory reservation, and expose up to `target_partitions` round-robin output partitions. During execution, the same tasks are read through a reader built from `Table::reader_builder()`, which preserves the table's `FileIO` and runtime. A single built reader is cloned across local partitions, sharing its delete-loader state. Neither the `TableScan` nor its manifest/evaluator caches are retained after file discovery. The ungrouped `PlannedFileScan` is exposed to downstream consumers that need to assign the selected tasks differently. The option is **off by default**. The existing lazy `TableScan::to_arrow()` path remains unchanged. `partition_statistics()` still reports unknown statistics; this PR makes no selected-file statistics, worker routing, dynamic-filtering, or performance claims. The pinned iceberg-rust revision and the work-unit wire format are unchanged. ## Trade-offs and follow-ups - **Reader settings propagation:** `Table::reader_builder()` inherits table-local IO/runtime and the current reader defaults, but not non-default reader options configured on a `TableScan`. Today these providers configure only snapshot, projection, and predicate, which are carried by the tasks. If scan-specific batch size, data-file concurrency, row-group, page-selection, or bloom settings are introduced, lazy and planned reads could diverge. Multi-partition planned scans deliberately use one in-flight data file per partition rather than the lazy reader's default concurrency. Follow up in iceberg-rust with lightweight, portable scan-level reader options and **one core settings-to-builder mapping** shared by `TableScan::to_arrow()` and engines applying those options to worker-/session-local builders. Do not serialize `FileIO`, a runtime, or a built reader. - **Planning overhead and lifecycle:** planning all tasks earlier performs manifest I/O and retains task memory even for early-stopping queries. Repeated `TableProvider::scan()` calls still re-plan. `iceberg_local.planning_max_files` (100,000 by default) and the DataFusion memory pool bound retained work, returning an error rather than a truncated plan. Abandoned-planning cancellation and planning from a host runtime with IO disabled are not covered here; validate both before enabling eager planning by default. - **Reconstruction:** `IcebergDataSource::new_with_predicate()` continues to rebuild a lazy single-partition source; a distributed caller must explicitly transport/assign `PlannedFileScan` tasks when replacing the source rather than assume that constructor carries them over. - **Delete behavior:** the retained tasks carry delete references and clones of the built reader share delete-loader state, but the checked-in multi-file fixture has no delete files. Add a delete-bearing eager/lazy parity test and verify cross-partition delete fetches/cache lifetime before merging. Add an abandoned-planning cancellation test when a suitable fixture/hook exists. The option is documented in `crates/datafusion/README.md`. Its `iceberg_local` prefix is intentionally distinct from DFD's `iceberg` config extension for sessions that register both. ## Validation - `cargo test -p iceberg-datafusion --locked --test eager_file_planning` (8 tests): pass, including default lazy mode, pruned/empty files, partition counts, filtered results, global and ordered LIMIT, custom table runtime, file limit, and query-memory rejection. - `cargo fmt --all -- --check`: pass. - `cargo clippy --workspace --locked --all-targets -- -D warnings`: pass. - `cargo test --workspace --locked`: pass. - `git diff --check`: pass. ## AI disclosure Developed with AI assistance; reviewed and validated locally before opening as a draft. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
