c-thiel commented on code in PR #2880:
URL: https://github.com/apache/iceberg-rust/pull/2880#discussion_r4491849162
##########
crates/iceberg/src/arrow/record_batch_transformer.rs:
##########
@@ -233,6 +233,62 @@ pub(crate) enum ChildPlan {
}
impl PromotePlan {
+ /// Plans one field, dispatching on the target field so a variant is seen
by its extension.
+ fn build_field(
+ source: &Field,
+ target: &Field,
+ snapshot_schema: &IcebergSchema,
+ constant_fields: &HashMap<i32, ColumnConstant>,
+ ) -> Result<Self> {
+ if is_variant_field(target) {
+ return Self::build_variant(source.data_type(), target.data_type());
+ }
+ Self::build(
+ source.data_type(),
+ target.data_type(),
+ snapshot_schema,
+ constant_fields,
+ )
+ }
+
+ /// A variant's `metadata` and `value` carry no field ids, so they are
matched by name.
+ /// When reading data files, projection reports unsupported or malformed
storage before this
+ /// runs.
+ fn build_variant(source: &DataType, target: &DataType) -> Result<Self> {
Review Comment:
Agreed, done in 524f5fddc. `validate_unshredded_variant` now lives in
`arrow/schema.rs`, and `build_variant` calls it with the field id from the
target field, so both paths report the same errors, including
`FeatureUnsupported` for shredded storage. It also rejects an optional metadata
as `DataInvalid` with the column name and field id. I am checking this now on
the schema, not per batch: the Parquet spec says metadata is required, the
Arrow extension says it is non-nullable, and every writer I looked at
(iceberg-java, Spark, Trino, DuckDB, Arrow C++, arrow-rs) writes it as
required. This is a bit stricter than Java, which doesn't check the repetition.
An optional value is still accepted, since the table type reads value as
nullable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
