ryzhyk opened a new issue, #24786:
URL: https://github.com/apache/datafusion/issues/24786
Reading a Parquet file with `datafusion.execution.parquet.coerce_int96` set
is supposed to only change the time unit that INT96 timestamp columns decode
at. However it also empties the metadata of every struct, list and map field in
the file.
## Cause
In `datafusion/datasource-parquet/src/schema_coercion.rs`, leaf fields are
cloned:
```rust
// :423
fn field_with_new_type(field: &FieldRef, new_type: DataType) -> FieldRef {
Arc::new(field.as_ref().clone().with_data_type(new_type))
}
```
while container fields are constructed fresh, and `Field::new*` starts from
empty metadata:
```rust
// :327
let processed_struct = Field::new_struct(
current_field.name(),
processed_children.as_slice(),
current_field.is_nullable(),
);
// :360
let processed_list = Field::new_list(
current_field.name(),
Arc::clone(&processed_children[0]),
current_field.is_nullable(),
);
// :392
DataType::Map(Arc::clone(&processed_children[0]), *sorted),
```
Name, type and nullability are carried across; `current_field.metadata()` is
never read.
A test that demonstrates the issue:
https://github.com/ryzhyk/coerce_int96_bug/tree/main
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
