c-thiel opened a new issue, #3606:
URL: https://github.com/apache/iceberg-rust/issues/3606
### Is your feature request related to a problem or challenge?
#2880 reads unshredded variant columns from Parquet: a variant group stored
as `metadata` and `value` binaries. A projected variant whose group contains a
`typed_value` field fails with `FeatureUnsupported` ("Reading shredded variant
column `v` (field id N) is not supported yet").
Iceberg v3 allows writers to shred variants following the Parquet [variant
shredding
spec](https://github.com/apache/parquet-format/blob/master/VariantShredding.md).
iceberg-java can write such files (`write.parquet.shred-variants`, off by
default) and reads them. Tables containing shredded files are unreadable from
iceberg-rust today if a query projects a shredded variant.
Unprojected shredded variants do not block reads of other columns.
### Describe the solution you'd like
Read shredded variant groups and return each row's variant, as the
unshredded path does today.
Requirements from the shredding spec:
- Rebuild each row from `value` and `typed_value`, following the spec's
rules for which of the two holds the row (including null and missing values).
- Partially shredded objects: merge the fields in `typed_value` with the
fields still encoded in `value`.
- Shredded arrays and nested shredded objects, recursively.
- Shredded primitive types mapped to their variant encodings (decimals,
timestamps, UUID, etc.).
- Match the columns inside a variant group by name, since they carry no
field ids.
arrow-rs provides `unshred_variant` in `parquet-variant-compute`, which
folds `typed_value` back into `value`. Using it would add that crate as a
dependency.
Related: #2411 (v3 tracking), #2880 (unshredded read). Reference
implementation in iceberg-go: apache/iceberg-go#986.
### Willingness to contribute
None
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
