andygrove opened a new issue, #6816:
URL: https://github.com/apache/datafusion-comet/issues/6816
## What / Why
Comet converts between instants and local time with chrono-tz, which stores
each zone's precomputed transitions up to its last one, around 2099 for zones
that observe DST. After that it applies the zone's standard offset all year.
Spark uses `java.time`, whose `ZoneRules` keep applying the zone's last
transition rules forever. So after about 2100, a native expression that works
in local time in a DST zone can be off by the DST offset for the summer months.
For example, in an `America/Los_Angeles` session with
`spark.comet.expression.TruncTimestamp.allowIncompatible=true`,
`date_trunc('DAY', TIMESTAMP'2300-07-01 12:00:00')` is `2300-07-01T07:00:00Z`
(PDT) in Spark and `2300-07-01T08:00:00Z` (PST) in Comet. This happens both for
literal formats and for formats that come from a column. The same offset
applies to `hour` and to casts from timestamps to strings and dates, which run
natively by default.
The compatibility guide already documents the difference ("approximately
beyond year 2100"). This issue tracks fixing it rather than documenting it.
## Possible fixes
#4754 evaluates moving from chrono to Jiff, which applies a zone's POSIX TZ
rule after its last transition, as `java.time` does.
Found during review of #6354.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
