adriangb opened a new pull request, #24636:
URL: https://github.com/apache/datafusion/pull/24636
## Which issue does this PR close?
- Closes #24632.
## Rationale for this change
With `datafusion.execution.time_zone` set, `to_timestamp` on a string
**column** declared a
return type of `Timestamp(ns, <tz>)` but produced an array of `Timestamp(ns,
None)`. The
declared type and the produced array disagreed, so any plan that
materialized the column
failed at execution:
```sql
SET datafusion.execution.time_zone = 'UTC';
SELECT to_timestamp(s) AS v FROM (VALUES ('2026-01-01T00:00:00')) AS t(s);
-- Internal error: Function 'to_timestamp' returned value of type
'Timestamp(ns)' while
-- the following type was promised at planning time and expected:
'Timestamp(ns, "UTC")'
```
A second symptom: because the planner trusts the declared type it inserts no
coercion, so a
comparison against another timezone-aware value reached the Arrow kernel
with mismatched
types (`Invalid comparison operation: Timestamp(ns) <= Timestamp(ns,
"UTC")`).
Scalar arguments were unaffected — they are constant-folded through a
different code path
that did pick up the timezone — which is why every existing test passed.
## What changes are included in this PR?
`handle`/`handle_multiple` in `datafusion/functions/src/datetime/common.rs`
built the result
array straight from `O::DATA_TYPE`, which for timestamps carries no
timezone, and ignored
the `dt` return type they were passed. The scalar branches of the same
functions went
through `scalar_value(dt, ..)` and did carry the timezone — hence the
mismatch.
The array branches now re-annotate the result with `dt` before returning it.
`to_date`, the
other caller, returns `Date32` and is unaffected.
Two existing unit tests in `to_timestamp.rs` asserted `Timestamp(_, None)`
for array results
even though their helpers configure `"UTC"` as the execution timezone; those
expectations
encoded the bug and are updated.
## Are these changes tested?
Yes.
- `to_timestamp_timezone.slt` gains coverage for non-constant-folded (array)
input: the
`arrow_typeof` and value for `to_timestamp` under `UTC` and
`America/New_York`, the
explicit-format overload, all four precision variants, the comparison from
the issue, and
a `to_date` case confirming it is unaffected. Every one of these fails on
`main`.
- A new `to_timestamp_array_respects_execution_timezone` unit test asserts,
for each of the
five `to_timestamp*` UDFs and both overloads, that the returned array's
type equals what
`return_type` advertises.
The full sqllogictest suite and the extended workspace test suite pass, as
does
`./dev/rust_lint.sh`.
## Are there any user-facing changes?
Yes, a bug fix: `to_timestamp*` on a string column now returns an array
annotated with
`datafusion.execution.time_zone`, matching its declared return type, instead
of failing at
execution. No API changes.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
