raghuvanshraj opened a new pull request, #3699: URL: https://github.com/apache/iceberg-rust/pull/3699
## Which issue does this PR close? - Closes #2546. ## What changes are included in this PR? The Parquet writer now accepts a variant field. It writes the field as a group with the `VARIANT` logical type and the field id of the Iceberg field, with a required binary `metadata` leaf and an optional binary `value` leaf. The leaves carry no field id, which is how iceberg-java writes them. The annotation comes from enabling the `variant_experimental` feature of `parquet`, which maps the `arrow.parquet.variant` extension that the schema converter already attaches to the `VARIANT` logical type. Data file metrics follow iceberg-java's `ParquetMetrics`: value and null counts come from the `metadata` leaf, the column size adds both leaves, and an unshredded variant has no lower or upper bound. Shredded writes (`write.parquet.shred-variants`) are not part of this PR. One thing to know: parquet 60.1 writes the `VARIANT` annotation without `specification_version`. apache/arrow-rs#11961 adds `specification_version = 1` but is not in a release yet. Spark 4.1's own Parquet reader refused such a file when I tried in September (Spark 4.1.1 on Glue); I have not checked whether Iceberg's Spark reader does the same. The test asserts `LogicalType::variant(None)`, so it will flag the change when the parquet dependency moves to a release with the fix. ## Are these changes tested? Unit tests in `parquet_writer.rs`: - `test_index_by_parquet_path_variant`: both leaves of a variant map to its field id at the top level and inside a struct, a list and a map. - `test_parquet_writer_variant_roundtrip`: writes an optional top-level variant and a required variant inside a struct across two row groups, checks the Parquet schema (logical type, field ids, leaf types), the data file metrics, and reads the rows back through `ArrowReader`, comparing JSON row for row. I also read the Spark 4.1.1 / Iceberg 1.11 unshredded files from my earlier test with this branch, since `variant_experimental` changes the reader's schema inference as well. Same rows as before. ## AI Disclosure Written with an AI coding assistant. I read the full diff and ran the tests above. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
