DeviousCardi opened a new pull request, #3452: URL: https://github.com/apache/iceberg-rust/pull/3452
## Which issue does this PR close? - Closes #3399. ## What changes are included in this PR? The v2 manifest Avro schema declared `first_row_id` (142), `content_offset` (144) and `content_size_in_bytes` (145). The spec defines these only for v3, so a `DataFile` with any of them set wrote v3 values into a v2 manifest. The v2 `data_file` schema now has the same fields as Java's `V2Metadata.fileType`, keeping `referenced_data_file` (143). The writer previously used the v2 schema for v3 manifests too, so v3 manifests now get their own `manifest_schema_v3`, which keeps all four fields. Things the diff doesn't show: - v3-only values set on a `DataFile` are dropped when it is written to a v2 manifest, not rejected. Java behaves the same way: its v2 `DataFileWrapper` exposes only the v2 positions. The v1 writer already drops them the same way. - Reading is unchanged. `Manifest::parse_avro` reads with the writer schema, so v2 manifests that older iceberg-rust versions wrote with these fields still parse, and the values are returned as before. Gating v3 fields when reading v1/v2 is a separate item on #2411. - Nothing on a v2 table depends on these fields. Row lineage is v3-only, and deletion vectors (which use the content fields) are v3-only. - `write_data_files_to_avro` and `read_data_files_from_avro` use `data_file_schema_v2` for `FormatVersion::V2`, so they also stop carrying the three fields for v2. ## Are these changes tested? Yes, with unit tests: - A new writer test writes the same `DataFile`, with all four fields set, to a v2 and a v3 manifest. It checks the `data_file` field ids in the written Avro schema and the data file read back from each manifest. - `test_parse_manifest_without_each_field` now writes with the v3 schema and runs for both v2 and v3 metadata. Its v2 case is a manifest as older versions wrote it, with 142/144/145 present. ## AI Disclosure The code and tests were written with Claude Code (Claude Opus 5.5), then reviewed by the author. Areas a reviewer should check: - Dropping the values silently on the v2 path, versus returning an error from `add_file`/`add_entry`. I followed Java, but an error would surface caller bugs earlier. - `read_data_files_from_avro` with `FormatVersion::V2` now uses a reader schema without the three fields, so it returns `None` for them even if the bytes contain them. I found no callers outside tests; if something external depends on this, it is a behaviour change. - The v3 field order (`first_row_id`, `referenced_data_file`, `content_offset`, `content_size_in_bytes`) is the one that already existed, and it matches Java's `V3Metadata.fileType`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
