andygrove opened a new issue, #2598: URL: https://github.com/apache/datafusion-ballista/issues/2598
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.** In [`ballista/core/src/execution_plans/distributed_query.rs`](https://github.com/apache/datafusion-ballista/blob/ea1126c0649b2bc08e1e233b1972b64318821c16/ballista/core/src/execution_plans/distributed_query.rs) (26 commits over the last 12 months), `execute_query_pull` (L507-L686) and `execute_query_push` (L691-L870) are about 180 lines each and share: - a near-identical prologue: config extraction, endpoint creation and customization, connect, client construction; - an identical job-status handling tail: the None/Queued/Running/Failed arms and the whole `Successful` arm, i.e. timing, three gauge metrics, and the per-partition `fetch_partition` stream construction ([pull L593-L686](https://github.com/apache/datafusion-ballista/blob/ea1126c0649b2bc08e1e233b1972b64318821c16/ballista/core/src/execution_plans/distributed_query.rs#L593-L686), [push L805-L870](https://github.com/apache/datafusion-ballista/blob/ea1126c0649b2bc08e1e233b1972b64318821c16/ballista/core/src/execution_plans/distributed_query.rs#L805-L870)). Both carry `#[allow(clippy::too_many_arguments)]`. `SubstraitSchedulerClient::poll_job` in [examples/standalone-substrait.rs#L305-L342](https://github.com/apache/datafusion-ballista/blob/ea1126c0649b2bc08e1e233b1972b64318821c16/examples/examples/standalone-substrait.rs#L305-L342) is a third copy of the status handling. Any change to metrics or fetch parameters has to be made in each. **Describe the solution you'd like** - Extract `connect_scheduler(...)` for the prologue. - Extract a shared status handler, plus `finish_successful_job(...)` that sets the metrics and builds the partition stream. Pull and push then differ only in how they get the next status (poll plus sleep vs. streamed items). - Bundle the repeated fetch parameters (`max_message_size`, `use_tls`, retry settings, `customize_endpoint`) into a small struct, removing the `too_many_arguments` allows. - Have the Substrait example call a public polling helper instead of keeping its own copy. **Describe alternatives you've considered** Share only the `Successful` arm, which is the largest identical block. **Additional context** Line references are pinned to commit ea1126c0649b. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
