Shekharrajak opened a new issue, #12751: URL: https://github.com/apache/iceberg/issues/12751
* There is no definition added in spark for tiemstamp datatype variants : https://spark.apache.org/docs/3.5.0/sql-data-sources-avro.html#supported-types-for-spark-sql---avro-conversion * Avro spec : https://avro.apache.org/docs/1.12.0/specification/ ``` Timestamps The timestamp-{millis,micros,nanos} logical type represents an instant on the global timeline, independent of a particular time zone or calendar. Upon reading a value back, we can only reconstruct the instant, but not the original representation. In practice, such timestamps are typically displayed to users in their local time zones, therefore they may be displayed differently depending on the execution environment. timestamp-millis: logical type annotates an Avro long, where the long stores the number of milliseconds from the unix epoch, 1 January 1970 00:00:00.000. timestamp-micros: logical type annotates an Avro long, where the long stores the number of microseconds from the unix epoch, 1 January 1970 00:00:00.000000. timestamp-nanos: logical type annotates an Avro long, where the long stores the number of nanoseconds from the unix epoch, 1 January 1970 00:00:00.000000000. Example: Given an event at noon local time (12:00) on January 1, 2000, in Helsinki where the local time was two hours east of UTC (UTC+2). The timestamp is first shifted to UTC 2000-01-01T10:00:00 and that is then converted to Avro long 946720800000 (milliseconds) and written. Local Timestamps The local-timestamp-{millis,micros,nanos} logical type represents a timestamp in a local timezone, regardless of what specific time zone is considered local. local-timestamp-millis: logical type annotates an Avro long, where the long stores the number of milliseconds, from 1 January 1970 00:00:00.000. local-timestamp-micros: logical type annotates an Avro long, where the long stores the number of microseconds, from 1 January 1970 00:00:00.000000. local-timestamp-nanos: logical type annotates an Avro long, where the long stores the number of nanoseconds, from 1 January 1970 00:00:00.000000000. Example: Given an event at noon local time (12:00) on January 1, 2000, in Helsinki where the local time was two hours east of UTC (UTC+2). The timestamp is converted to Avro long 946728000000 (milliseconds) and then written. ``` * Iceberg Spec for all the timestamp types : https://iceberg.apache.org/spec/#avro ``` timestamp | Timestamp, microsecond precision, without timezone timestamptz | Timestamp, microsecond precision, with timezone ``` But spark 3 treat non timezone as TimestampNTZType but the avro serialiser will treat TimestampTZType as logicaltype timestamp-millis and TimestampNTZType as local-timestamp-millis -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
