hcrosse opened a new pull request, #3623: URL: https://github.com/apache/iceberg-rust/pull/3623
## Which issue does this PR close? - Closes https://github.com/apache/iceberg-rust/issues/3617. ## What changes are included in this PR? Millisecond timestamp columns can lose matching rows during pruning, return wrong rows during filtering, or fail projection. Pruning reads millisecond bounds as microseconds, while the row filter truncates microsecond literals to milliseconds. The three commits apply in this order: 1. Respect Parquet timestamp units in row-group and page bounds, bloom probes, and footer metrics. 2. Widen millisecond columns to microseconds for comparisons instead of truncating literals. 3. Enable millisecond projection once pruning and filtering are correct. Behaviour changes: - `NANOS` columns under a microsecond field get no bounds, and bloom probes never rule out a match. This benefit only shows end to end once https://github.com/apache/iceberg-rust/issues/3616 is fixed. On main, the `INT96` coercion relabels raw `INT64` nanoseconds as microseconds under `timestamp`, and the read fails under `timestamptz`. - Footer metrics return the existing `Statistics ... is not match with field type` error when a bound cannot be converted, rather than writing bounds that are wrong by a factor of 1000. This includes `NANOS` under a microsecond field and millisecond bounds beyond about ±292,000 years. Only `parquet_files_to_data_files`, which is unused, reaches this error, because the writer always writes microseconds. - `arrow_schema_to_schema` now accepts Arrow millisecond timestamps. Writing millisecond batches into the resulting table still fails with `Incompatible type`, so callers must cast before writing. - `RecordBatchTransformer` promotion casts now return an error instead of writing null when a value does not fit, for example a millisecond value beyond about ±292,000 years. Iceberg's type promotions widen and cannot fail this way. In practice, only that overflow and files matched without field IDs and with mismatched types reach this path. - A millisecond column under a `timestamp_ns` or `timestamptz_ns` field now projects as nulls instead of failing with `Unsupported Arrow data type`. Main already projects microsecond columns under nanosecond fields as nulls. Predicates on such columns behave as on main. ## Are these changes tested? I added nine tests, including an end-to-end matrix across four pruning modes: default, page index only, bloom filter only, and all off, with and without timestamp projection. Five tests fail on main. Three use new internal helpers and cannot compile on main, so I checked each by restoring main's behaviour in its helper. One passes on main and guards against widening the column for `IS NULL` and `IS NOT NULL`. Formatting, clippy, documentation, and `cargo test -p iceberg` passed on each commit. Workspace clippy and public API checks passed on the final tree. A separate differential check with a fixed seed found no mismatches across 900 cases. That check is not committed. ## AI Disclosure I used AI coding tools to help draft the code, tests and this description. I reviewed the changes, ran the checks listed above, and can explain the design. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
