yuhao-su commented on PR #2188:
URL: https://github.com/apache/iceberg-rust/pull/2188#issuecomment-4897749198
Could we attach the canonical Arrow extension name to variant fields in
ToArrowSchemaConverter::variant()? i.e. return the Struct{metadata, value}
field with ARROW:extension:name = arrow.parquet.variant (matches
parquet_variant_compute::VariantType::NAME in arrow-rs).
Two reasons:
Consumers can't distinguish a variant column from a regular struct<metadata:
binary, value: binary> in the converted schema. The extension name is the
standard signal downstream engines key on to decode the column as variant
instead of a plain struct.
Consistency with file-derived schemas: parquet-rs already attaches this
extension when reading VARIANT-annotated files, so schemas read from data files
carry it while schema_to_arrow_schema output doesn't. Anything that reconciles
the two (e.g. RecordBatchTransformer) ends up stripping the extension from
emitted batches.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
