ranflarion opened a new pull request, #26464: URL: https://github.com/apache/datafusion/pull/26464
## Which issue does this PR close? - Closes #26463. ## Rationale for this change With `coerce_int96` set, INT96 timestamps inside Hive's two-level lists (how Hive writes `array<timestamp>`), bare `repeated int96` fields and parquet-mr Avro's `array` lists stay `Timestamp(Nanosecond, None)`, so values outside 1677-2262 overflow on decode, which is what `coerce_int96` exists to prevent. ## What changes are included in this PR? `Int96Coercer` no longer rebuilds Parquet paths from Arrow field names, which assumed the three-level `<list>.list.<element>` layout. For each top-level field it records which of its Parquet leaf columns are INT96, in the depth-first order in which arrow-rs maps them to Arrow leaves, and walks the field's Arrow type in that order, so the list or map encoding no longer matters. Top-level fields are still matched by name, so a projected file schema keeps working. The walk now also descends `LargeList` and `FixedSizeList`, and replaces the explicit stack with a short recursion that keeps field metadata via `field_with_new_type`. ## What is the testing strategy for this PR? `coerce_int96_to_resolution_with_two_level_and_legacy_lists` covers a Hive two-level list, a bare repeated field, an Avro-style `array` list of structs, a three-level list, a map value and a non-INT96 nanosecond timestamp, and then a projected, reordered file schema. It fails on main for the first three. The existing nested-type and field-metadata tests pass unchanged. ## Are there any user-facing changes? No API changes. INT96 columns in those list layouts now coerce like the rest. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
