sunchao commented on code in PR #5177:
URL: https://github.com/apache/datafusion-comet/pull/5177#discussion_r3884502327
##########
native/core/src/parquet/parquet_support.rs:
##########
@@ -195,6 +195,21 @@ fn parquet_convert_array(
list_arr.nulls().cloned(),
)))
}
+ (
+ Timestamp(TimeUnit::Millisecond, _),
+ Timestamp(TimeUnit::Microsecond, target_tz),
+ ) => {
+ // Spark's Parquet reader calls the checked `millisToMicros`
conversion for both
+ // direct and dictionary values, independent of CAST evaluation
mode:
+ //
https://github.com/apache/spark/blob/v4.2.0/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetVectorUpdaterFactory.java#L817-L833
+ // `millisToMicros` uses `Math.multiplyExact`:
+ //
https://github.com/apache/spark/blob/v4.2.0/sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/SparkDateTimeUtils.scala#L103-L108
+ let micros = array
+ .as_primitive::<TimestampMillisecondType>()
+ .try_unary::<_, TimestampMicrosecondType, _>(|value|
value.mul_checked(1_000))?
+ .with_timezone_opt(target_tz.clone());
Review Comment:
**[P2] Preserve timestamp pruning before checked conversion**
With a `TimestampType` column stored as `TIMESTAMP_MILLIS` and containing
`9223372036854776` milliseconds, `WHERE ts < TIMESTAMP '1970-01-01 00:00:00'`
returns no rows in Spark 4.1.3 and the base native build (`c067e4e`), but
`5c8de42` throws `Overflow happened on: 9223372036854776 * 1000`. I reproduced
this with plain/dictionary encoding, ANSI on/off, and Comet row-filter pushdown
on/off: all eight Spark/base cases succeed and all eight head cases fail.
The existing `CometCastColumnExpr` prevents DataFusion from recognizing the
timestamp statistics predicate, so Comet converts values from row groups Spark
skips. This new error turns that pruning limitation into a query failure.
Please preserve pruning before the checked conversion and add this filtered
case as a regression test, while retaining overflow errors for values actually
read.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
