mbutrovich commented on code in PR #3354:
URL: https://github.com/apache/iceberg-rust/pull/3354#discussion_r4198182488
##########
crates/iceberg/src/spec/manifest/mod.rs:
##########
@@ -45,9 +47,30 @@ pub struct Manifest {
}
impl Manifest {
- /// Parse manifest metadata and entries from bytes of avro file.
- pub(crate) fn try_from_avro_bytes(bs: &[u8]) -> Result<(ManifestMetadata,
Vec<ManifestEntry>)> {
- let reader = AvroReader::new(bs)?;
+ /// Parse manifest metadata and entries from bytes of avro file. `location`
+ /// names the manifest in warnings.
+ pub(crate) fn try_from_avro_bytes(
+ bs: &[u8],
+ location: Option<&str>,
+ ) -> Result<(ManifestMetadata, Vec<ManifestEntry>)> {
+ let rewritten;
+ let reader = match AvroReader::new(bs) {
+ Ok(reader) => reader,
+ Err(e) if matches!(e.details(),
Details::AmbiguousSchemaDefinition(_)) => {
+ let Some((bs, repeated)) = define_named_types_once(bs)? else {
+ return Err(e.into());
+ };
+ tracing::warn!(
Review Comment:
> I'd dedupe the warning (once per path, or `debug!` after the first) and
note when the shim can be dropped.
The [first such manifest in a
process](https://github.com/apache/iceberg-rust/blob/91486882b29deb0d97560543e0764aa00a2988fd/crates/iceberg/src/spec/manifest/mod.rs#L71-L83)
logs at warn level, and later ones log at debug level. The [comment on the
fallback](https://github.com/apache/iceberg-rust/blob/91486882b29deb0d97560543e0764aa00a2988fd/crates/iceberg/src/spec/manifest/mod.rs#L64-L66)
now says when it can go: when tables no longer contain manifests that
iceberg-rust wrote before it defined each named type once. No release makes
that safe, because old manifests stay in a table until something rewrites them.
The variant match fails to compile if apache-avro renames
`AmbiguousSchemaDefinition`. If apache-avro stops returning it,
[`test_parse_manifest_with_repeated_named_type_definitions`](https://github.com/apache/iceberg-rust/blob/91486882b29deb0d97560543e0764aa00a2988fd/crates/iceberg/src/spec/manifest/mod.rs#L2104-L2118)
fails. The copy happens only for these manifests and cop
ies the file once. I haven't measured it next to the parse, which decodes
every entry.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
