andygrove opened a new issue, #7368: URL: https://github.com/apache/datafusion-comet/issues/7368
### What is the problem the feature request solves? On a Comet Native Executor cluster (#7289), a hash exchange becomes a `RepartitionExec(Hash)`, which Ballista shuffles with DataFusion's hash function rather than Spark's Murmur3. That is consistent within a query, which is all joins, aggregates and windows need. A plan that relies on Spark's own placement of rows, such as a join with a bucketed table read without a shuffle, or a storage-partitioned join, would return wrong results. ### Describe the potential solution apache/datafusion-ballista#2723, on Ballista's `comet-native` branch, lets a hash shuffle place rows by a partition id the client computes into a column. The producing stage could compute Spark's partition id with Comet's own `CometPartitioning` hash, and the exchange would route by it. Ballista's adaptive planner currently refuses that partition-id marker, so this also needs that path supported there. ### Additional context The hash exchange is built in `exchange` in `native/native-executors/src/assembly.rs`, whose comment notes the limitation. Part of #7288. ### Willingness to contribute Yes. I can contribute this feature independently. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
