andygrove opened a new issue, #23890: URL: https://github.com/apache/datafusion/issues/23890
### Describe the bug `next_day` behaves differently across the Spark versions `datafusion-spark` targets, and DataFusion's `.slt` files have no way to express a version-specific expectation. This issue records the divergence so it can be scheduled once that mechanism exists. Companion to https://github.com/apache/datafusion/issues/23887. ### The divergence Given a non-NULL start date and an unparseable `day_of_week`: | Spark version | Default `spark.sql.ansi.enabled` | Default result | Error class when ANSI is on | | ------------- | -------------------------------- | -------------- | --------------------------- | | 3.5.8 | `false` | `NULL` | `_LEGACY_ERROR_TEMP_2000`, wrapped by `QueryExecutionErrors.ansiIllegalArgumentError` | | 4.0.4 | `true` | error | `ILLEGAL_DAY_OF_WEEK`, SQLSTATE 22009 | | 4.1.3 | `true` | error | `ILLEGAL_DAY_OF_WEEK`, SQLSTATE 22009 | | 4.2.0 | `true` | error | `ILLEGAL_DAY_OF_WEEK`, SQLSTATE 22009 | Two independent changes landed in Spark 4.0: 1. ANSI mode became the default, so the default result for the same query flipped from `NULL` to an error. 2. `DateTimeUtils.getDayOfWeekFromString` changed from `throw new IllegalArgumentException("Illegal input for day of week: ...")` to `throw new SparkIllegalArgumentException(errorClass = "ILLEGAL_DAY_OF_WEEK", ...)`, and `NextDay.nullSafeEval` changed from catching `IllegalArgumentException` and re-wrapping to catching `SparkIllegalArgumentException` and rethrowing unchanged. A third 4.0 change is not currently observable: `inputTypes` moved from `Seq(DateType, StringType)` to `Seq(DateType, StringTypeWithCollation(supportsTrimCollation = true))`. `datafusion-spark` has no collation support, so there is nothing to diverge from yet, but a non-default collation on the `day_of_week` argument is a divergence axis that exists only in Spark 4.x. ### Current state in DataFusion Both modes are now implemented and tested behind `datafusion.execution.enable_ansi_mode`, whose default is `false`. Anyone comparing against a stock Spark 4.x will see DataFusion's default disagree with Spark's default. The `.slt` file asserts Spark 4.2.0 behavior in each mode and carries a comment naming the versions. DataFusion does not model Spark error classes, so `_LEGACY_ERROR_TEMP_2000` versus `ILLEGAL_DAY_OF_WEEK` is not represented at all. `datafusion-spark` raises `Execution error: Illegal input for day of week: <input>`, matching Spark's message text but not its error class or SQLSTATE. ### Additional context Relevant files: - `datafusion/spark/src/function/datetime/next_day.rs` - `datafusion/sqllogictest/test_files/spark/datetime/next_day.slt` - Spark: `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala`, `case class NextDay` - Spark: `sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala`, `getDayOfWeekFromString` Surfaced by the audit-datafusion-spark-expression skill. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
