sunchao commented on code in PR #5177:
URL: https://github.com/apache/datafusion-comet/pull/5177#discussion_r3887263640
##########
native/core/src/parquet/parquet_support.rs:
##########
@@ -195,6 +195,21 @@ fn parquet_convert_array(
list_arr.nulls().cloned(),
)))
}
+ (
+ Timestamp(TimeUnit::Millisecond, _),
+ Timestamp(TimeUnit::Microsecond, target_tz),
+ ) => {
+ // Spark's Parquet reader calls the checked `millisToMicros`
conversion for both
+ // direct and dictionary values, independent of CAST evaluation
mode:
+ //
https://github.com/apache/spark/blob/v4.2.0/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetVectorUpdaterFactory.java#L817-L833
+ // `millisToMicros` uses `Math.multiplyExact`:
+ //
https://github.com/apache/spark/blob/v4.2.0/sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/SparkDateTimeUtils.scala#L103-L108
+ let micros = array
+ .as_primitive::<TimestampMillisecondType>()
+ .try_unary::<_, TimestampMicrosecondType, _>(|value|
value.mul_checked(1_000))?
+ .with_timezone_opt(target_tz.clone());
Review Comment:
**[P2] Preserve pruning for IN, null-safe equality, and nested timestamp
predicates**
Rechecked `0d78875`: the original `ts < epoch` reproducer now passes, but
this issue remains for three supported predicate forms. With 32 repeated
`9223372036854776` millisecond values in a `TIMESTAMP_MILLIS` column (and the
same value in nested `s.ts`), these predicates return zero rows in Spark 4.1.3
and base `c067e4e`, while the PR head throws `Overflow happened on:
9223372036854776 * 1000`:
```sql
ts IN (TIMESTAMP '1970-01-01 00:00:00', TIMESTAMP '1970-01-02 00:00:00')
ts <=> TIMESTAMP '1970-01-01 00:00:00'
s.ts < TIMESTAMP '1970-01-01 00:00:00'
```
All 24 cases reproduce across plain/dictionary encoding, ANSI on/off, and
row-filter pushdown on/off; dictionary encoding was verified from the footer.
The eight ordinary `<` controls pass on all three builds.
The [new
rewrite](https://github.com/apache/datafusion-comet/blob/0d78875ab6d9ec99d0a1426348c86e7da9208648/native/core/src/parquet/schema_adapter.rs#L596-L634)
skips `InListExpr`, does not handle `IsNotDistinctFrom`, and only matches a
direct `CometCastColumnExpr`, so the nested-field predicate also retains the
opaque conversion. These row groups are still read and converted even though
Spark prunes them.
Could we extend pruning to these forms before checked conversion and add
regression coverage? This is the failing-query direction of the original issue,
separate from the documented case where Comet skips an error on a row discarded
by its filter.
Validation: independent native Parquet scans plus focused Spark 4.1.3/JDK17
comparisons. The base control reused a hash-verified `c067e4e` native library;
production JVM/proto sources are identical. Other Spark versions were not run
locally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
