andygrove opened a new issue, #5011:
URL: https://github.com/apache/datafusion-comet/issues/5011
### Describe the bug
The INT96 counterpart of #5010. Comet's native Parquet scan does not apply
the hybrid-Julian → proleptic-Gregorian rebase to INT96 timestamp columns, so
pre-1582 timestamps read from a legacy-calendar file come back shifted.
`spark.sql.parquet.int96RebaseModeInRead` (and the
`spark.sql.legacy.parquet.int96RebaseModeInRead` alias) is one of the configs
listed under #4180; Comet reads neither, and also ignores the
`org.apache.spark.legacyINT96` file metadata that takes precedence over it.
Filed separately from #5010 because the INT96 read path, the metadata key
and the governing config are all distinct from the INT32/INT64 date-time path,
even though the root cause is the same missing rebase in the native scan.
### Steps to reproduce
```scala
val path = "/tmp/legacy_int96"
spark.conf.set("spark.sql.parquet.outputTimestampType", "INT96")
spark.conf.set("spark.sql.parquet.int96RebaseModeInWrite", "LEGACY")
spark.sql("""SELECT cast(s as timestamp) as ts
FROM VALUES ('1000-01-01 12:34:56'),('1990-01-01 00:00:00') AS
v(s)""")
.write.mode("overwrite").parquet(path)
spark.conf.unset("spark.sql.parquet.outputTimestampType")
spark.conf.unset("spark.sql.parquet.int96RebaseModeInWrite")
spark.conf.set("spark.comet.enabled", "false")
spark.read.parquet(path).show(false)
// +---------------------+
// |ts |
// +---------------------+
// |1000-01-01 12:34:56 |
// |1990-01-01 00:00:00 |
// +---------------------+
spark.conf.set("spark.comet.enabled", "true")
spark.read.parquet(path).show(false)
// +---------------------+
// |ts |
// +---------------------+
// |1000-01-06 12:41:58 | <-- wrong
// |1990-01-01 00:00:00 |
// +---------------------+
```
No non-default config is needed on the read side: Spark rebases because the
file carries `org.apache.spark.legacyINT96`. Setting
`spark.sql.parquet.int96RebaseModeInRead` to `LEGACY`, `CORRECTED` or
`EXCEPTION` makes no difference to Comet's output either.
### Expected behavior
Same options as #5010: implement the INT96 rebase honouring the file
metadata and `spark.sql.parquet.int96RebaseModeInRead`, or fall back to Spark
for affected files and non-default config values.
### Additional context
* Reproduced on `apache/main` @ `0761e549a`, default Maven profile (Spark
4.1.2), macOS aarch64, debug build.
* `spark.comet.scan.enabled=false` makes the results match, isolating this
to the native scan.
* Found while auditing the config inventory in #4180.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
