sam-1112 opened a new pull request, #7443: URL: https://github.com/apache/datafusion-comet/pull/7443
## Which issue does this PR close? Closes #6854 . ## Rationale for this change Comet currently excludes decimal columns from Iceberg residual predicate pushdown. Selective decimal queries therefore cannot use native row-group or page pruning, even though the underlying reader supports it. This change enables decimal pushdown while preserving exact values and safe predicate conversion. ## What changes are included in this PR? - Serialize decimal literals using signed unscaled bytes and the column's precision and scale. Decline literals that require rounding or exceed the column's precision. - Decode decimal literals into typed iceberg-rust `Datum` values, including full 38-digit coefficients, with validation for malformed encodings. - Remove decimals from the unsupported-column gate while retaining UUID, fixed and binary exclusions. - Preserve supported conjuncts in positive AND expressions, retain both OR branches, and require complete conversion beneath NOT. - Enable decimal fuzz coverage and add serialization, predicate-conversion and pruning regressions. The existing post-scan filter continues to enforce the full expression. ## How are these changes tested? - Scala serialization tests cover exact rescaling, signed encoding, precision boundaries, rejected literals, IN, AND, OR, NOT and double NOT. - Rust tests cover decimal decoding, invalid inputs, predicate conversion and schema binding. - End-to-end tests compare native results against Spark for DECIMAL(9,2), DECIMAL(18,2) and DECIMAL(38,2), including negative values, nulls, duplicate rows and precision boundaries. - Page-pruning tests verify physical Parquet encodings, page indexes and pushed decimal literals. A single-file, single-row-group fixture requires selective decimal-only reads to scan less than half the bytes of a full read. - A separate Rust reader test isolates row-group pruning with page selection disabled. Validation performed: - Spark 3.5 and 4.1 serialization and decimal page-pruning tests passed. - Spark 4.1 native Iceberg, fuzz and residual-pushdown integration suites passed, with two existing version-dependent cancellations. - Rust predicate/pruning tests, Cargo fmt, workspace Clippy and Maven Spotless passed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
