andygrove opened a new pull request, #2718: URL: https://github.com/apache/datafusion-ballista/pull/2718
# Which issue does this PR close? No issue. This is preparation for the DataFusion 56 upgrade, following the same approach used for DataFusion 55 (#2241). # Rationale for this change Tracking DataFusion's main branch ahead of the 56 release lets us absorb upstream breaking changes in small steps instead of all at once when 56 ships. Pinning to an exact commit keeps builds reproducible, and the pin can be bumped periodically until the release, when we switch back to crates.io versions. # What changes are included in this PR? Dependencies: - Pin `datafusion`, `datafusion-cli`, `datafusion-functions-aggregate-common`, `datafusion-proto`, `datafusion-proto-common` and `datafusion-substrait` to apache/datafusion@611311cd7ef50b40dd33f88ec527e2913de8e3d1. - Bump the dependencies DataFusion main now requires: arrow / arrow-flight 60 (the `flight-sql-experimental` feature is renamed `flight-sql`), object_store 0.14.2, and rustyline 19 in `ballista-cli`. - Raise the MSRV from 1.94 to 1.95 to match DataFusion. - Re-sync the vendored `datafusion.proto` and `datafusion_common.proto` with `dev/update_datafusion_proto.py`. Ports to upstream API changes: - `LogicalPlan::DescribeTable` is replaced by `LogicalPlan::Show`, which now covers `DESCRIBE`, `SHOW TABLES`, `SHOW COLUMNS`, `SHOW CREATE TABLE`, `SHOW ALL` and `SHOW FUNCTIONS`. These all read the client's catalog and cannot be serialized, so `BallistaQueryPlanner` plans every `Show` locally. Previously `SHOW TABLES` and friends were rewritten into `information_schema` queries, which the planner already ran locally. - `OptimizerRule::supports_rewrite` was removed upstream. - `ExecutionPlan::with_preserve_order` is deprecated and no longer called, so `ChaosExec` stops overriding it. - `TableProvider::scan` takes `Option<&[usize]>` for the projection. Test updates: - DataFusion now rejects a null-aware `HashJoinExec` in `Partitioned` mode. Ballista's planner and AQE already lower null-aware joins to `CollectLeft`, so only two join-selection tests that built the `Partitioned` form needed to change. - DataFusion now rejects `RANGE` offsets on a `Utf8` order key, so the parallel-window "no encoding" test uses a `CURRENT ROW` frame instead. It still fails if the key is switched to an encodable type. - `DiskManager` now creates its temp directories on the first spill file, so the spill-dir test creates one before checking the paths. - Rebuilding an `InterleaveExec` with non-interleavable children no longer errors upstream, so a precondition check in the `NormalizeInterleaveRule` test is dropped. The rule itself may no longer be needed, which can be a follow-up. - Refresh the TPC-H plan-stability golden files and insta snapshots. DataFusion now embeds projections into `SortMergeJoinExec` instead of placing a `ProjectionExec` above it, and the byte-size statistics estimates changed. The `EXPLAIN ANALYZE` expectation picks up new `AggregateExec` metrics. The Python bindings (`python/`) are not moved to the pin because they build against a datafusion-python release. # Are there any user-facing changes? Downstream crates now need arrow 60, object_store 0.14 and Rust 1.95, and see the DataFusion main API (for example `LogicalPlan::Show` and the new `TableProvider::scan` signature). Labelled `api-change` for that reason. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
