qzyu999 opened a new issue, #2716: URL: https://github.com/apache/iceberg-rust/issues/2716
### What's the feature are you trying to implement? ### What's the feature you're trying to implement? Expose bounded-memory DataFusion execution operations through `pyiceberg-core`'s Python bindings, enabling PyIceberg to perform compute-heavy table operations (compaction, delete resolution, CoW rewrites) entirely in Rust with spill-to-disk guarantees. ### Motivation PyIceberg ([iceberg-python#3554](https://github.com/apache/iceberg-python/issues/3554)) needs to perform operations that are infeasible under bounded memory with PyArrow alone. `pyiceberg-core` already provides Rust-backed transforms and a read-only DataFusion TableProvider. This epic extends it with execution operations — each exposed as a single Python function that runs an entire DataFusion plan in Rust (GIL released, memory bounded by FairSpillPool, spill-to-disk via DiskManager). Design principle: **operation-level FFI**. Python passes file paths + config, Rust does all I/O + compute internally, returns only metadata (DataFile JSON). Data never crosses the FFI boundary. --- ### Part 1: Primitives (iceberg-rust core + iceberg-datafusion) These are foundational pieces needed before the execution operations can be built. Some already have issues/PRs in progress by other contributors. #### Transaction actions (others' work — we depend on these) | Status | Issue/PR | Description | |--------|----------|-------------| | Under review | [#2185](https://github.com/apache/iceberg-rust/pull/2185) | `OverwriteAction` — atomic add + delete files (CoW primitive) | | Draft | [#2620](https://github.com/apache/iceberg-rust/pull/2620) | `MergingSnapshotProducer` — foundation for RewriteFiles | | Open | [#1607](https://github.com/apache/iceberg-rust/issues/1607) | `RewriteFiles` support (umbrella) | | Open | [#2244](https://github.com/apache/iceberg-rust/issues/2244) | Implement `RewriteFilesAction` | | Open | [#2242](https://github.com/apache/iceberg-rust/issues/2242) | Process delete files when writing snapshots | | Open | [#2243](https://github.com/apache/iceberg-rust/issues/2243) | Implement `SnapshotValidator` | | Open | [#2711](https://github.com/apache/iceberg-rust/issues/2711) | DataFusion `InsertOp::Overwrite` silently ignored | #### DataFusion integration primitives (our work — no blockers) | Status | Description | |--------|-------------| | To do | **Bounded-memory session helper** — `create_bounded_session()` in `iceberg-datafusion` with `FairSpillPool` + `DiskManager` | | To do | **`IcebergOverwriteCommitExec`** — DataFusion `ExecutionPlan` node that uses `OverwriteAction`/`RewriteFiles` for atomic file replacement (blocked on #2185 or #2244 landing) | --- ### Part 2: Python execution module (bindings/python) The `pyiceberg_core.execution` module — new submodule in `bindings/python/src/execution.rs` exposing operations to Python via PyO3. #### Infrastructure (no blockers) - [ ] Module structure + `register_module` in `lib.rs` - [ ] Result types (`CowRewriteResult`, `CompactionResult`) - [ ] Function signatures with `todo!()` bodies #### Read-path operations (needs: bounded session helper only) - [ ] `execute_equality_resolution` — anti-join of data × equality delete files - [ ] `execute_antijoin_paths` — path anti-join for orphan file detection #### Write-path operations (needs: bounded session + OverwriteAction/RewriteFiles) - [ ] `execute_cow_rewrite` — streaming filter → write for CoW deletes/overwrites - [ ] `execute_compaction` — sort + rewrite with target file size splitting #### Future operations - [ ] `execute_upsert` — hash join for matched rows - [ ] `execute_eq_to_positional` — convert equality deletes to positional - [ ] `execute_position_delete_compaction` — rewrite data files with pos deletes applied --- ### Related - [iceberg-python#3554](https://github.com/apache/iceberg-python/issues/3554) — PyIceberg DataFusion integration epic (consumer of this work) - #2269 — [EPIC] Missing Write Actions - #2186 — CoW and MoR support epic - #1797 — Community discussion on making iceberg-rust more usable ### Willingness to contribute I can contribute these features independently, delivered incrementally per the checklists above. ### Willingness to contribute I would be willing to contribute to this feature with guidance from the Iceberg Rust community -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
