peterxcli opened a new issue, #5687: URL: https://github.com/apache/datafusion-comet/issues/5687
### What is the problem the feature request solves? PR #5679 fixed `concat_ws` correctness for `array<string>` arguments by routing those expressions through Spark codegen dispatch, with Spark fallback when the dispatcher is disabled. This avoids the native execution failure from #5675, but these valid Spark expressions still have no native path. Spark accepts any ordered mix of string and `array<string>` arguments after the separator. It flattens array elements in argument order, skips null string arguments and null array elements, and returns null for a null separator. DataFusion's built-in `concat_ws` accepts only string arguments. This follows the maintainer suggestion in https://github.com/apache/datafusion-comet/pull/5679#issuecomment-5543800985. ### Describe the potential solution Add a Spark-compatible native `concat_ws` implementation for mixed string and `array<string>` inputs, then allow the array shape through `CometConcatWs` instead of codegen dispatch. Coverage should include: - mixed scalar and array arguments, preserving argument and element order; - multiple arrays; - null scalar arguments, null arrays, and null array elements; - empty arrays and a null separator; - non-foldable column inputs through the native execution path. Keep the existing dispatcher/fallback guard until the native implementation covers these semantics. ### Additional context - Correctness fallback: #5675 / #5679 - DataFusion's generic `concat_ws` currently rejects list inputs, so this likely needs a Spark-compatible Comet implementation rather than directly exposing the existing DataFusion function. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
