This is an automated email from the ASF dual-hosted git repository.
wenchen pushed a commit to branch branch-3.5
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.5 by this push:
new bdfa6efaa5f9 [SPARK-50870][SQL] Add the timezone when casting to
timestamp in V2ScanRelationPushDown
bdfa6efaa5f9 is described below
commit bdfa6efaa5f9f5a7abb7adfcada2414be182ed63
Author: changgyoopark-db <[email protected]>
AuthorDate: Tue Jan 21 12:02:04 2025 +0800
[SPARK-50870][SQL] Add the timezone when casting to timestamp in
V2ScanRelationPushDown
### What changes were proposed in this pull request?
Add the timezone information to a cast expression when the destination type
requires it.
### Why are the changes needed?
When current_timestamp() is materialized as a string, the timezone
information is gone (e.g., 2024-12-27 10:26:27.684158) which prohibits further
optimization rules from being applied to the affected data source.
For example,
```
Project [1735900357973433#10 AS current_timestamp()#6]
+- 'Project [cast(2025-01-03 10:32:37.973433#11 as timestamp) AS
1735900357973433#10]
+- RelationV2[2025-01-03 10:32:37.973433#11] xxx
```
-> This query fails to execute because the injected cast expression lacks
the timezone information.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Existing tests.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #49549 from changgyoopark-db/SPARK-50870.
Authored-by: changgyoopark-db <[email protected]>
Signed-off-by: Wenchen Fan <[email protected]>
(cherry picked from commit 24abb0fbb1dbb18b513bceabe22c6183f5e32034)
Signed-off-by: Wenchen Fan <[email protected]>
---
.../sql/execution/datasources/v2/V2ScanRelationPushDown.scala | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/V2ScanRelationPushDown.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/V2ScanRelationPushDown.scala
index ef3982ff9087..46a06c5f5eaf 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/V2ScanRelationPushDown.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/V2ScanRelationPushDown.scala
@@ -322,7 +322,12 @@ object V2ScanRelationPushDown extends Rule[LogicalPlan]
with PredicateHelper {
if (expression.dataType == expectedDataType) {
expression
} else {
- Cast(expression, expectedDataType)
+ val cast = Cast(expression, expectedDataType)
+ if (cast.timeZoneId.isEmpty && cast.needsTimeZone) {
+ cast.withTimeZone(conf.sessionLocalTimeZone)
+ } else {
+ cast
+ }
}
def buildScanWithPushedAggregate(plan: LogicalPlan): LogicalPlan =
plan.transform {
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
