antonyprasad-db commented on issue #1819:
URL:
https://github.com/apache/iceberg-python/issues/1819#issuecomment-5987658572
A precise reproduction, and a scoping question.
**What breaks today, on pyiceberg 0.12.0.** A format version 3 table
carrying a VARIANT column cannot be opened. Table metadata validation raises
before any column is read:
```
1 validation error for TableMetadataV3
schemas.0.fields.1.type
Value error, Unsupported field type: 'variant'
```
The schema in that reproduction has two fields, `id` as a `long` and
`payload` as `variant`. The `long` is collateral: the table is unreadable, not
just the Variant column. That is a different failure shape from "Variant values
are unsupported", and it is the one users hit first. Swapping that single field
type for any registered type parses the identical metadata fine, so the failure
is type registration in the metadata parser rather than anything on the data
path.
**The scoping question.** Value decoding clearly waits on the Arrow chain,
which is still open: `apache/arrow#50121` and `#50122` are in review with
`#50131` and `#50132` behind them. Metadata deserialization does not depend on
any of it. Would the project accept registering `VariantType` as a parsed,
representable type ahead of value support, so that a table with a Variant
column opens and its other columns read, with explicit unsupported errors on
Variant value access?
I would rather state the cost than undersell it. Registering a new primitive
means supplying visitor behaviour in `schema.py`, the Avro resolver on both
read and write, `io/pyarrow.py` and `conversions.py`. `UnknownType` fills all
of those cheaply because its values are always null, so it is not a precedent
for a type whose values exist but cannot be decoded. For Variant those paths
would need to raise a clear `NotImplementedError`, which is a new pattern here
and a maintainer call rather than mine.
If that scoping is acceptable I will take the registration half and keep it
narrow. If you would rather hold the issue whole until the Arrow chain lands,
say so and I will not duplicate effort.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
