andygrove opened a new issue, #1602: URL: https://github.com/apache/datafusion-ballista/issues/1602
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.** The benchmarks in https://github.com/apache/datafusion-ballista/pull/1600 show that Comet achieves ~100x greater throughput than Ballista. These are apples-to-apples benchmarks that read real Parquet files and pass them through ShuffleWriterExec. Since the Comet shuffle implementation has been stable across many Comet releases and has been proven at scale, I think we should just replace the Ballista implementation. **Describe the solution you'd like** This will happen over a series of PRs **Describe alternatives you've considered** **Additional context** -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
