peterxcli opened a new issue, #5553: URL: https://github.com/apache/datafusion-comet/issues/5553
## Context #5177 makes the native Parquet reader's `TIMESTAMP_MILLIS -> microseconds` conversion checked for **top-level** columns, matching Spark's `millisToMicros` (`Math.multiplyExact`). Two cases intentionally keep the pre-existing safe-cast behavior (overflow -> NULL) instead of erroring like Spark: 1. **Nested fields** (struct children, list/map elements): the conversion inside `parquet_convert_struct_to_struct` / list / map recursion is unchecked. 2. **Scans whose data filters reference nested fields** (`GetStructField`): the whole scan falls back to the safe cast (`SparkParquetOptions.checked_timestamp_overflow = false`, set in `init_datasource_exec`). ## Why Spark only avoids the overflow error for filtered-out values through row-group statistics pruning (`ParquetFilters` supports nested column predicates). DataFusion currently can neither: - prune nested-field predicates — `datafusion-pruning` says "PruningPredicate does not support pruning on nested fields yet" (`pruning_predicate.rs`), nor - evaluate them as Parquet row filters — `can_expr_be_pushed_down_with_schemas` in `datafusion-datasource-parquet/src/row_filter.rs` classifies struct columns as non-pushable. So a checked conversion on these paths would decode row groups Spark prunes and fail queries Spark answers with zero rows (see the review discussion on #5177: `ts IN (...)`, `ts <=> ...`, and `s.ts < ...` reproducers). ## Remaining divergence vs Spark - A nested-predicate scan that Spark *fails* to prune (mixed row-group statistics) throws `ArithmeticException` in Spark but returns NULL for overflowing top-level values in Comet. - A direct read of an overflowing nested `TIMESTAMP_MILLIS` field throws in Spark but returns NULL in Comet (pre-existing behavior on main). ## Proposed work Once DataFusion supports nested-field pruning (and ideally nested row filters), remove the `checked_timestamp_overflow` fallback and extend the checked conversion to nested fields, plus extend the millisecond-domain predicate rewrite in `SparkPhysicalExprAdapter` to `GetStructField`-wrapped conversions. Regression coverage lives in `ParquetReadSuite`: - "TIMESTAMP_MILLIS overflow fails in native scan" - "TIMESTAMP_MILLIS overflow rows skipped by filter pruning do not fail" Related: #5517 (error fidelity for the overflow exception). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
