peterxcli commented on code in PR #5177:
URL: https://github.com/apache/datafusion-comet/pull/5177#discussion_r3887368382
##########
native/core/src/parquet/parquet_support.rs:
##########
@@ -195,6 +195,21 @@ fn parquet_convert_array(
list_arr.nulls().cloned(),
)))
}
+ (
+ Timestamp(TimeUnit::Millisecond, _),
+ Timestamp(TimeUnit::Microsecond, target_tz),
+ ) => {
+ // Spark's Parquet reader calls the checked `millisToMicros`
conversion for both
+ // direct and dictionary values, independent of CAST evaluation
mode:
+ //
https://github.com/apache/spark/blob/v4.2.0/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetVectorUpdaterFactory.java#L817-L833
+ // `millisToMicros` uses `Math.multiplyExact`:
+ //
https://github.com/apache/spark/blob/v4.2.0/sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/SparkDateTimeUtils.scala#L103-L108
+ let micros = array
+ .as_primitive::<TimestampMillisecondType>()
+ .try_unary::<_, TimestampMicrosecondType, _>(|value|
value.mul_checked(1_000))?
+ .with_timezone_opt(target_tz.clone());
Review Comment:
Verified all three and fixed. `IN` and `<=>` now rewrite into the
millisecond domain (list elements rescale with IN's null semantics preserved;
null-safe equality rescales or folds to a constant, and `<=> NULL` becomes `IS
NULL`) — I confirmed DataFusion 54.1's PruningPredicate analyzes both
`InListExpr` and `IsNotDistinctFrom`, so pruning covers them in every config.
The nested case turned out deeper than the rewrite: DataFusion can neither
prune nested-field predicates
([pruning_predicate.rs](https://github.com/apache/datafusion/blob/54.1.0/datafusion/pruning/src/pruning_predicate.rs#L1131)
— "PruningPredicate does not support pruning on nested fields yet") nor
evaluate them as row filters (`can_expr_be_pushed_down_with_schemas` classifies
struct columns non-pushable), and the failing conversion was actually the flat
`ts` column being materialized from row groups nothing could prune. Since Spark
only avoids the error via nested statistics pruning we don't have, scans whose
data filters reference nested fields now fall back to the safe conversion
(overflow → NULL, main's behavior) and the filter discards the rows — matching
Spark's zero-row answers across your matrix. Checked conversion stays for all
other scans, and is scoped to top-level columns for the same reason.
Extended the regression test to your three forms plus the flat control (4
predicates × dictionary × ANSI × rowFilterPushdown), and added native tests for
the IN/null-safe rewrites and the top-level/nested/flag-off conversion split.
Remaining divergence, documented in the code: a nested-predicate scan Spark
*fails* to prune errors in Spark but NULLs in Comet, as does a direct nested
read of overflow — both pre-existing behavior. Filed #5553 to lift both once
DataFusion grows nested-field pruning. (917286124)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
