comphead opened a new pull request, #3540: URL: https://github.com/apache/iceberg-rust/pull/3540
## Which issue does this PR close? - Closes #3122. ## What changes are included in this PR? A file written before an `int` -> `long`, `float` -> `double`, or decimal precision promotion keeps its old physical type, while the predicate is bound to the table type. Filtered scans of such files dropped matching rows, and for doubles they could return non-matching ones. - **Row filter.** `try_cast_literal` cast the literal down to the file's Arrow type. An out-of-range literal became null, so `x < 2147483648` returned no rows and `x IN (1, 2147483648)` dropped the row where `x = 1`. A double literal was rounded, so `x = 0.1` matched a stored `0.1f32`. `PredicateConverter` now casts a numeric column to the table type before comparing, as `RecordBatchTransformer` does for the output. Other columns keep the literal cast. - **Page index** (`with_row_selection_enabled(true)`). INT32 and FLOAT bounds of a `long` or `double` field were built as `Datum(Long, Int)` and `Datum(Double, Float)`, which never compare, so range and `IN` predicates pruned every page, even for `x < 4`. The bounds are now widened, as in Java's `ParquetConversions.converterFromParquet`. UINT_32 bounds are read as `u32`, so values above `i32::MAX` don't wrap negative. Row-group pruning skips INT32 and FLOAT statistics on `long` and `double` fields, and bloom filters already handle promotion. This takes the approach of #3123, which was closed as stale without review. ## Are these changes tested? Yes. `test_predicate_on_promoted_column` reads `int`, `float`, and UINT_32 files as `long` and `double` under nine predicates, with and without page-index pruning. On `main` it fails on the first case (`v < 2147483648` returns no rows). Of 18 mutants of the new code, the test catches 17. The survivor drops the `is_numeric()` guard, which only spares string columns a cast. A local probe also showed `decimal(9,2)` read as `decimal(30,2)` and `INT(8)` read as `int` losing rows on `main` and passing here. The `iceberg` lib suite passes, and clippy and fmt are clean. ## AI Disclosure Drafted with Claude Code (Anthropic). Opened as a draft pending the author's review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
