andygrove commented on PR #1954: URL: https://github.com/apache/datafusion-ballista/pull/1954#issuecomment-5860627798
I merged the latest `main` into this branch and did some cleanup before review. Here's what changed. **Merge conflicts** - `extension.rs`: #2387 moved the config accessors to the `ballista_config_option!` macro, so the reuse flag goes through it now. That adds a `with_ballista_reuse_exchange_enabled` setter next to the getter. - `configs.md`: regenerated with `dev/update_config_docs.sh`. **Design** - Reuse now lives in `DefaultDistributedPlanner`. `with_exchange_reuse(codec)` turns it on, and `plan_query_stages` merges the stages at the end. The pass depends on how that planner numbers and builds stages, so it felt like the right home. - `StaticExecutionGraph::new_with_reuse` is gone, and `execution_graph.rs` is unchanged from `main`. `TaskManager` builds the planner with reuse when the session flag is on. - The planner takes the `PhysicalExtensionCodec` that ships stages to executors instead of a key closure. Reuse is only safe if the key is faithful, and the codec's encoding is. `BallistaCodec` gets a small `physical_extension_codec_arc()` accessor for this. - Distributed `EXPLAIN` takes the planner too. On the static path it now shows the stages that actually run. **Behavior** - Stages with a volatile expression like `random()` are never merged. Two identical `SELECT random() ...` subtrees should still draw their own values. - The config key is now `ballista.optimizer.reuse_exchange`, to match `not_in_subquery_rewrite` and `stage_build_side`. - AQE is on by default now, so reuse only applies to the static planner. That means `ballista.planner.adaptive.enabled=false` or a physical plan submission. The config description says so. **Smaller cleanups** - Stages whose inputs were not remapped are kept as they are instead of being rebuilt. - The key comes from DataFusion's `physical_plan_to_bytes_with_extension_codec`. - The tests are a bit leaner, and there's a new test for volatile expressions. The TPC-H plan-stability goldens did not change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
