andygrove opened a new issue, #7366: URL: https://github.com/apache/datafusion-comet/issues/7366
### What is the problem the feature request solves? On a Comet Native Executor cluster (#7289), the scheduler range-partitions a range exchange by split points it samples, so a global sort runs in parallel. It does so only when every sort key is a column whose type Arrow orders as Spark does: integers, decimals, dates, timestamps, strings, binary and booleans. A sort on a float or double, on a nested type, or on an expression still runs in one task. The range shuffle compares raw values. Comet's sort normalizes float keys first (`NormalizeNaNAndZero`, `NormalizeNestedFloats`), so a NaN with its sign bit set, which x86 produces for `0.0 / 0.0`, would land in the lowest range but sort last. ### Describe the potential solution Route rows by the same normalized expressions Comet's sort uses, built with `PhysicalPlanner::create_sort_expr` as Comet's own range shuffle does, instead of by raw columns. That needs `try_encode_expr` and `try_decode_expr` for those expressions in the cluster's `CometPhysicalCodec`, and the sample has to be taken of the same expressions. It would also cover sort keys that are expressions rather than columns. ### Additional context The check is `orders_like_spark` in `native/native-executors/src/assembly.rs`. It needs confirming that Ballista's range shuffle writer accepts sort expressions that are not columns. Part of #7288. ### Willingness to contribute Yes. I can contribute this feature independently. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
