mbutrovich commented on code in PR #3354:
URL: https://github.com/apache/iceberg-rust/pull/3354#discussion_r4198176086
##########
crates/iceberg/src/spec/manifest/data_file.rs:
##########
@@ -333,7 +333,9 @@ pub fn read_data_files_from_avro<R: Read>(
FormatVersion::V3 => data_file_schema_v3(partition_type).unwrap(),
};
- let reader = AvroReader::with_schema(&avro_schema, reader)?;
+ let reader = AvroReader::builder(reader)
Review Comment:
> The repeated-definition fallback only runs in `try_from_avro_bytes`, but
`read_data_files_from_avro` opens the same kind of bytes this crate itself
wrote via `write_data_files_to_avro`
The PR description claimed the same thing, and it was wrong.
`read_data_files_from_avro` passes a reader schema, and 0.22 checks the writer
schema for repeated names only when the reader has no reader schema
([`reader/block.rs#L266-L282`](https://github.com/apache/avro-rs/blob/ec5721cb0c80dcde56c1049a004f1d785abd88cf/avro/src/reader/block.rs#L266-L282)).
[`test_read_data_files_with_repeated_named_type_definitions`](https://github.com/apache/iceberg-rust/blob/91486882b29deb0d97560543e0764aa00a2988fd/crates/iceberg/src/spec/manifest/_serde.rs#L436-L480)
writes data files with two `decimal(10, 2)` partition fields, rewrites the
header to repeat the definition as 0.21 wrote it, and reads the same data files
back. I removed the claim from the description. If a later apache-avro release
adds the check on that path, this test fails. Sharing the fallback now would
mean buffering the whole `R: Read` input, because the fallback rewrites the
header and reads the file again. Does it work for
you to share it only if that test fails?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
