michaelmitchell-bit opened a new pull request, #4592: URL: https://github.com/apache/datafusion-comet/pull/4592
## Which issue does this PR close? Closes #4561. ## Rationale for this change `split_part` lowers to `element_at(StringSplitSQL(str, delimiter), partNum)`. Comet already supports `element_at`, but the inner `StringSplitSQL` expression was unsupported, so `split_part` fell back to Spark. ## What changes are included in this PR? Adds native support for Spark 4.x `StringSplitSQL` via a literal-delimiter `split_sql` function. Unlike `split` / `StringSplit`, this splits on the delimiter literally rather than as a regex and keeps trailing empty parts. This also fixes `ListExtract` default-value typing so defaults are cast to the array element type rather than the list type. That is needed for `split_part`, which uses an empty-string default through `element_at`. Updates expression support docs and adds SQL-file coverage for `split_part`, including literal delimiters, dynamic delimiters, nulls, out-of-range defaults, negative positions, and invalid zero index behavior. ## How are these changes tested? Ran: - `cargo fmt -p datafusion-comet-spark-expr` - `cargo test -p datafusion-comet-spark-expr string_funcs::split` - `cargo test -p datafusion-comet-spark-expr list_extract` - `cargo build` - `./mvnw test -Dtest=none -Dsuites="org.apache.comet.CometSqlFileTestSuite split_part" -Dscalastyle.skip=true` - `./mvnw compile -Pspark-4.0 -DskipTests -Dscalastyle.skip=true` - `./mvnw compile -Pspark-4.2 -DskipTests -Dscalastyle.skip=true` - `git diff --check` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
