kevinjqliu opened a new issue, #3036: URL: https://github.com/apache/iceberg-rust/issues/3036
## Summary Remove `pyiceberg_core.datafusion.IcebergDataFusionTable` from the `pyiceberg-core` Python bindings. This proposal only removes the Python export. It does **not** remove any DataFusion table provider from iceberg-rust. ## Rationale This experimental feature was intended to integrate PyIceberg with DataFusion Python. **However, the resulting query does not use PyIceberg's scan implementation.** PyIceberg passes the table identifier, metadata location, and file I/O properties to `pyiceberg-core`, which loads a separate Rust `StaticTable`. iceberg-rust then handles metadata and manifest parsing, pruning, scan planning, delete handling, and data reads. **This creates two independent Iceberg implementations behind a PyIceberg `Table`: Python for `Table.scan()` and Rust when the table is registered with DataFusion.** Their features and behavior can differ, and PyIceberg fixes do not apply to the Rust path. The API therefore appears to query through PyIceberg while actually relying on iceberg-rust's feature set and semantics. The binding also tightly couples `pyiceberg-core` to DataFusion's Rust FFI and its major-version compatibility requirements. ## Proposal Remove the following from the `pyiceberg-core` Python bindings: - The `pyiceberg_core.datafusion` module - The `IcebergDataFusionTable` Python class - Related module registration, dependencies, and tests The corresponding PyIceberg change should remove `Table.__datafusion_table_provider__` and update its documentation and optional dependencies. ## Non-goals This proposal does **not** remove or deprecate the Rust `iceberg-datafusion` crate, `IcebergTableProvider`, or `IcebergStaticTableProvider`. The Rust integration remains available to applications that choose iceberg-rust as their Iceberg implementation. Because the Python API has been released, the removal should be coordinated with PyIceberg and included in an appropriate breaking release. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
