Tishj opened a new issue, #18190:
URL: https://github.com/apache/iceberg/issues/18190
### Proposed Change
In Appendix A: Parquet format, under the `Notes` section, there's a comment
on the `UNKNOWN` type:
> Omit from data files
Due to the nature of Parquet's format, it's not always possible to omit a
field while keeping the structural integrity of the format.
Definition and repetition levels are stored per-leaf, and are computed from
the count of optional/repeated ancestors on the path from root to that leaf. A
group node (struct, list, map) has no storage of its own.
How this manifests:
- STRUCT (this includes the schema root) with all fields omitted: there is
no way to indicate the distinction between a row being `NULL` or `{}`.
- LIST with omitted element: there is no way to indicate whether a list is
`NULL` or `[]`.
This has (I presume) unintended consequences that disable these cases:
- UNKNOWN LIST element
- STRUCT with only UNKNOWN fields
- table with only UNKNOWN columns
I propose replacing this existing requirement with the requirement of using
`LogicalType.NULL`
(https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#unknown-always-null)
instead.
Due to the `NULL` type having no value data, the overhead compared to
omitting the field entirely is negligible, and only influenced by the number of
pages, not by the number of rows.
By no longer omitting the field for UNKNOWN columns/fields, the limitations
listed above all disappear.
This proposal has no effect on readers, because of the existing requirement:
> When reading an unknown column, any corresponding column must be ignored
and replaced with null values.
It also does not invalidate existing writers because of this existing reader
requirement.
It only lifts the limitations that existing writers following the spec
currently suffer from.
### Proposal document
_No response_
### Specifications
- [x] Table
- [ ] View
- [ ] REST
- [ ] Puffin
- [ ] Encryption
- [ ] Other
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
