andygrove opened a new issue, #5010:
URL: https://github.com/apache/datafusion-comet/issues/5010
### Describe the bug
Comet's native Parquet scan never applies the hybrid-Julian →
proleptic-Gregorian rebase that Spark applies when reading dates and timestamps
written in the legacy calendar. Every pre-1582 date/timestamp read from such a
file comes back shifted by the calendar offset, silently.
This is one of the configs listed under #4180
(`spark.sql.parquet.datetimeRebaseModeInRead` /
`spark.sql.legacy.parquet.datetimeRebaseModeInRead`), but the reproduction
below needs **no non-default config at all**, because Spark decides whether to
rebase from the `org.apache.spark.legacyDateTime` key/value metadata that it
writes into the file footer, and only falls back to the read config when that
metadata is absent.
Two things are missing on the Comet side:
* the rebase itself is not implemented in the native scan;
* the guard is not wired either. `spark.comet.exceptionOnDatetimeRebase` is
declared in `CometConf.scala:695` and is not read anywhere else in the tree,
and `SparkParquetOptions::use_legacy_date_timestamp_or_ntz`
(`native/core/src/parquet/parquet_support.rs:78`) is only ever initialized to
`false` and never consulted.
This looks like the gap that #1638 originally tracked for
`native_iceberg_compat`; that issue was closed as obsolete when the reader was
removed, but the same gap exists in the current native scan.
### Steps to reproduce
```scala
val path = "/tmp/legacy_dates"
// Write a file in the legacy hybrid calendar. This is byte-for-byte what a
Spark 2.x
// writer produced, and what Spark 3+ produces with
datetimeRebaseModeInWrite=LEGACY.
spark.conf.set("spark.sql.parquet.datetimeRebaseModeInWrite", "LEGACY")
spark.sql("SELECT cast(s as date) as d FROM VALUES
('1000-01-01'),('1990-01-01') AS v(s)")
.write.mode("overwrite").parquet(path)
spark.conf.unset("spark.sql.parquet.datetimeRebaseModeInWrite")
spark.conf.set("spark.comet.enabled", "false")
spark.read.parquet(path).show()
// +----------+
// | d|
// +----------+
// |1000-01-01|
// |1990-01-01|
// +----------+
spark.conf.set("spark.comet.enabled", "true")
spark.read.parquet(path).show()
// +----------+
// | d|
// +----------+
// |1000-01-06| <-- wrong, shifted by the Julian/Gregorian offset
// |1990-01-01|
// +----------+
```
`spark.comet.scan.enabled=false` makes the results match again, which
isolates this to the native scan.
The same applies to `TIMESTAMP_MICROS` columns written with
`datetimeRebaseModeInWrite=LEGACY`:
```
spark: 1000-01-01 12:34:56
comet: 1000-01-06 12:41:58
```
It is not confined to projection. Filters and aggregates over the affected
column silently return the wrong answer rather than obviously-wrong dates:
```sql
SELECT count(*) FROM parquet.`/tmp/legacy_dates` WHERE d = date'1000-01-01'
-- spark: 1
-- comet: 0
```
```sql
SELECT min(d), max(d) FROM parquet.`/tmp/legacy_dates`
-- spark: 1000-01-01, 1990-01-01
-- comet: 1000-01-06, 1990-01-01
```
Not reproduced here, but implied by the same missing code: a file with
**no** `org.apache.spark.legacyDateTime` metadata and no Spark writer version
(i.e. genuinely written by Spark 2.x, Hive, or Impala) is the case the read
config actually governs. Spark's default there is `EXCEPTION`, which fails the
query and asks the user to choose a mode; `LEGACY` rebases; `CORRECTED` reads
as-is. Comet honours none of the three.
### Expected behavior
Either of:
1. Comet's native scan rebases legacy-calendar dates/timestamps the way
Spark does, honouring the file metadata first and
`spark.sql.parquet.datetimeRebaseModeInRead` when the metadata is absent; or
2. Comet falls back to Spark for scans of files carrying
`org.apache.spark.legacyDateTime`, and for any non-default value of the read
config.
Given that the current behaviour is silent data corruption at default
settings, a fallback would be a reasonable first step.
### Additional context
* Reproduced on `apache/main` @ `0761e549a`, default Maven profile (Spark
4.1.2), macOS aarch64, debug build.
* `spark.sql.parquet.int96RebaseModeInRead` has the same problem for INT96
columns; tracked separately.
* Write-side rebase (`datetimeRebaseModeInWrite`, `int96RebaseModeInWrite`)
is not affected in the default configuration, since
`spark.comet.parquet.write.enabled` defaults to `false`.
* Found while auditing the config inventory in #4180.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
