andygrove opened a new issue, #2003: URL: https://github.com/apache/datafusion-ballista/issues/2003
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.** Ballista currently implements a batch execution model directly modeled after Apache Spark. Query stages are executed to completion, with output written to shuffle files. The plan is optionally re-optimized based on statistics from completed query stages (AQE). This is a well proven design that scales massively for ETL and analytical workloads. This issue is to discuss whether we should consider extending Ballista to also support a model that is closer to DataFusion's in-process execution model, where the whole plan is executed eagerly and results are streamed between stages, bypassing the shuffle files. This can lead to faster time to first results in some cases, and lower latency queries. **Describe the solution you'd like** **Describe alternatives you've considered** **Additional context** -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
