andygrove opened a new issue, #23889:
URL: https://github.com/apache/datafusion/issues/23889
### Describe the bug
`datafusion-spark`'s `next_day` declares
`Signature::exact(vec![DataType::Date32, DataType::Utf8],
Volatility::Immutable)`
in `datafusion/spark/src/function/datetime/next_day.rs`.
Spark's `NextDay` extends `ImplicitCastInputTypes` with
`inputTypes = Seq(DateType, StringTypeWithCollation(...))`, so Spark casts
the
first argument to `DATE` before evaluating. A `STRING` or a `TIMESTAMP` start
date is accepted. DataFusion raises a planning error instead.
This affects Spark 3.5.8 through 4.2.0 identically. The behavior has not
changed across those versions.
### To Reproduce
Spark SQL, verified against a live `pyspark==4.2.0` under both
`spark.sql.ansi.enabled=true` and `=false`:
```sql
SELECT next_day('2015-07-23', 'Mon');
-- 2015-07-27
SELECT next_day(TIMESTAMP '2015-07-23 12:12:12', 'Mon');
-- 2015-07-27
SELECT next_day('2015-07-23 12:12:12', 'Mon');
-- 2015-07-27
```
Spark's own golden file
`sql/core/src/test/resources/sql-tests/results/date.sql.out` asserts all
three.
DataFusion SQL:
```sql
SELECT next_day('2015-07-23'::string, 'Mon'::string);
-- Error during planning: Failed to coerce arguments to satisfy a call to
-- 'next_day' function: coercion from Utf8View, Utf8View to the signature
-- Exact(Date32, Utf8) failed. No function matches the given name and
argument
-- types 'next_day(Utf8View, Utf8View)'.
SELECT next_day('2015-07-23 12:12:12'::timestamp, 'Mon'::string);
-- Error during planning: Failed to coerce arguments to satisfy a call to
-- 'next_day' function: coercion from Timestamp(ns), Utf8View to the
signature
-- Exact(Date32, Utf8) failed.
```
### Expected behavior
`next_day` should accept the argument types Spark accepts and cast them the
way
Spark does.
### Additional context
Not fixed in the audit that surfaced this because it is a semantics decision
rather than a mechanical correction. Widening the signature raises questions
the maintainers should answer first:
1. Should `datafusion-spark` model Spark's `ImplicitCastInputTypes`
generally,
or does it expect the caller to insert the casts? The answer applies to
every function in the crate, not just `next_day`, so a one-off widening
here
would set an undeclared precedent.
2. Spark's STRING-to-DATE cast is itself ANSI dependent. `next_day('xx',
'Mon')`
raises `CAST_INVALID_INPUT` under ANSI and returns NULL otherwise. Any
widening has to route through a Spark-compatible cast, not
`arrow::compute::cast`.
3. `datafusion/sqllogictest/test_files/spark/datetime/next_day.slt` currently
asserts the coercion error as expected behavior. That assertion has to be
replaced, not merely extended.
The blocked cases are checked in as commented-out queries in
`datafusion/sqllogictest/test_files/spark/datetime/next_day.slt`, carrying
the
Spark 4.2.0 results above. Uncommenting them is the contract for verifying a
fix.
Relevant file and line:
`datafusion/spark/src/function/datetime/next_day.rs`, `SparkNextDay::new`.
Surfaced by the audit-datafusion-spark-expression skill.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
