ranflarion opened a new issue, #26463:
URL: https://github.com/apache/datafusion/issues/26463
### Describe the bug
With `datafusion.execution.parquet.coerce_int96` set, INT96 timestamps
inside lists that do not use the three-level LIST layout stay
`Timestamp(Nanosecond, None)`. That covers Hive's two-level lists (`repeated
int96 bag`, how Hive writes `array<timestamp>`), bare `repeated int96` fields,
and parquet-mr Avro's legacy lists whose repeated group is named `array`.
Values outside 1677-09-21 to 2262-04-11 then overflow on decode, which is what
`coerce_int96` is meant to prevent: `0001-01-01T01:02:03.123456` reads back in
1754.
### To Reproduce
```rust
use std::sync::Arc;
use arrow::datatypes::TimeUnit;
use datafusion_datasource_parquet::Int96Coercer;
use parquet::arrow::parquet_to_arrow_schema;
use parquet::schema::{parser::parse_message_type, types::SchemaDescriptor};
let descr = SchemaDescriptor::new(Arc::new(
parse_message_type(
"message m {
optional group hive (LIST) { repeated int96 bag; }
optional group three (LIST) { repeated group list { optional int96
element; } }
}",
)
.unwrap(),
));
let arrow_schema = parquet_to_arrow_schema(&descr, None).unwrap();
let coerced = Int96Coercer::new(&descr, &arrow_schema,
&TimeUnit::Microsecond)
.coerce()
.unwrap();
println!("{:?}", coerced.field_with_name("hive").unwrap().data_type());
println!("{:?}", coerced.field_with_name("three").unwrap().data_type());
```
`hive` keeps `Timestamp(Nanosecond, None)` while `three` becomes
`Timestamp(Microsecond, None)`.
### Expected behavior
Every INT96 leaf coerces to the requested unit, whatever the list encoding.
### Additional context
`coerce_int96_to_resolution_impl` in
`datasource-parquet/src/schema_coercion.rs` finds INT96 leaves by rebuilding
each Arrow leaf's Parquet path, and assumes a list's path is
`<list>.list.<element>`. For `hive` it looks up `hive.list.bag`, which no
column has, since the path is `hive.bag`. It also never descends `LargeList` or
`FixedSizeList`.
### Willingness to contribute
I plan to contribute a fix for this bug independently
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
