linzch3 opened a new pull request, #3621: URL: https://github.com/apache/iceberg-rust/pull/3621
## Which issue does this PR close? - Closes #3616. ## What changes are included in this PR? `coerce_int96_timestamps` decided which columns to coerce by Arrow type (`Timestamp(Nanosecond)`), which INT64 `TIMESTAMP(NANOS)` columns share with INT96. Under a microsecond hint arrow-rs reinterprets the raw INT64 nanosecond values as microseconds, so those columns came back 1000x too large (and timestamptz reads could fail). This passes the file's Parquet `SchemaDescriptor` in and only coerces leaves whose physical type is `INT96`. Coercion is skipped entirely when the file has no INT96 columns. ## Are these changes tested? Yes, with unit tests: - `int96.rs`: an INT64 nanosecond column is left alone, and in a mixed schema only the INT96 leaf is coerced. The existing tests go through a helper that marks nanosecond leaves as INT96. - `pipeline.rs`: `test_read_int64_nanosecond_timestamps_as_microseconds` writes INT64 `TIMESTAMP(NANOS)` (with and without tz) and reads it as Iceberg `timestamp`/`timestamptz`. Without the fix it reads `1970-01-01T00:16:40` instead of `00:00:01`. I ran `cargo test -p iceberg --lib arrow` (299 passed), `cargo clippy -p iceberg --all-targets -- -D warnings` and `cargo fmt` locally. ## AI Disclosure I prepared this change with an AI coding assistant (Grok Bot), then checked the diff and test results myself. The main area of uncertainty is that the visitor matches Arrow leaves to Parquet leaves by order, which relies on `visit_schema` calling `primitive()` once per Parquet leaf in schema order. That holds for structs, lists and maps, and the existing nested INT96 reader tests still pass. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
