andygrove opened a new issue, #2606: URL: https://github.com/apache/datafusion-ballista/issues/2606
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.** A review of complexity and duplication across Ballista's Rust code at commit ea1126c0649b turned up a set of hotspots: functions that are very large and change often, and code copied in several places where fixes have already landed in one copy but not the other. This epic tracks the follow-up issues so they can be picked up independently. All sub-issues are labelled `tech debt` and `help wanted`; the smaller ones are also `good first issue`. **Describe the solution you'd like** **Bugs found along the way** - [ ] #2593 Executor never honours `scheduler_connect_timeout_seconds` and retries the scheduler connection forever - [ ] #2594 AQE: `resolve_stage` and `rollback_*_stage` drop the stage from the graph when the conversion fails - AQE `update_task_status` gaps (dead `reset_running_stages` handling, missing late fetch-failure path, less useful max-failures message) were added as a comment on the existing #1986 rather than a new issue. **Duplicated scheduler state machine** - [ ] #2595 [EPIC] Share execution graph logic between `StaticExecutionGraph` and `AdaptiveExecutionGraph` **Large, frequently changed functions** - [ ] #2596 Split `QueryStageScheduler::on_receive` into per-event handlers - [ ] #2597 Break `start_executor_process_inner` into startup phases - [ ] #2601 AQE: split `SelectJoinRule::optimize` into per-action methods and flatten its nesting **Duplicated logic** - [ ] #2598 Deduplicate `execute_query_pull` and `execute_query_push` in `DistributedQueryExec` - [ ] #2599 Extract shared execute/handoff logic from the three shuffle writer execs - [ ] #2600 Share the binding driver between `bind_task_bias` and `bind_task_round_robin` - [ ] #2602 TUI: unify `App::on_key` and `App::on_key_sync` so key handling lives in one place **Good first issues** - [ ] #2603 Deduplicate the TPC-H table schema across scheduler tests and benchmarks - [ ] #2604 `BallistaClient`: share retry logic and don't panic when `io_retries_times` is 0 - [ ] #2605 TUI: collapse repeated sort-order blocks in `JobsData::sort_jobs` and share table scrolling helpers **Describe alternatives you've considered** Some findings were deliberately left out: - `maybe_promote_to_broadcast` and `DynamicJoinSelectionExec::to_actual_join` are complex, but broadcast and AQE join work is actively changing them, so a refactor now would conflict. - `BallistaPhysicalExtensionCodec::try_encode` / `try_decode` dispatch over every exec node, which is inherent; the repeated partition-location conversion and encode boilerplate are better folded into the next serde change. - Test-fixture duplication (execution-graph failure tests, gRPC registration tests, shuffle reader fixtures) is real but low risk. **Additional context** Line references in the sub-issues are pinned to commit ea1126c0649b. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
