andygrove opened a new issue, #7073: URL: https://github.com/apache/datafusion-comet/issues/7073
### What is the problem the feature request solves? A shuffle whose child is a native Comet plan falls to the JVM columnar (row-based) shuffle when native shuffle rejects the partitioning. The batches then go Arrow to `UnsafeRow` to Arrow again, an avoidable round trip. `CometShuffleExchangeExec.shuffleSupported` picks native shuffle only if `nativeShuffleFailureReasons` is empty, and these partitionings are rejected by default: - round-robin (`spark.comet.shuffle.native.partitioning.roundrobin.enabled`, default false); - hash on struct, array or map keys (`spark.comet.shuffle.native.partitioning.hash.nested.enabled`, default false); - hash with an expression native cannot evaluate; - range partitioning on an unsupported type. `repartition(n)` over a native plan is the common case, and it is round-robin. Partly addressed since #5905 was filed: #6095 added an opt-in positional round robin (`...roundrobin.positional.enabled`) that skips hashing, but it only applies when Comet can prove the task replays its rows in the same order, so most plans still keep content-hash placement or the JVM path. #6607 converts Spark row input to Arrow for native shuffle, which is the other direction (Spark child, not native child). ### Describe the potential solution The native writer already accepts JVM Arrow batches through the synthetic `ShuffleWriterInput` scan. Add a `CometPartitioning::PrecomputedIds(column)` variant and append an Int32 partition-id column computed on the JVM side by a codegen projection, so Spark computes the id and its semantics are unchanged. Every partitioning native cannot evaluate then runs through native shuffle instead of the row path. Separately, decide the default for nested hash keys. The stated blocker is that the nested murmur3 kernel's performance was unmeasured. `native/spark-expr/benches/hash.rs` now has struct, list<struct> and map shapes, so someone can run it against a flat-key baseline and either flip the default or point at the map path (`hash_funcs/utils.rs`, which slices one entry at a time) as the one to vectorize. ### Additional context Finding J1 in #5905. Related: #5397 and #5449 (round-robin on nested schemas), #6855 (nested-type conversion epic). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
