laskoviymishka opened a new pull request, #16972: URL: https://github.com/apache/iceberg/pull/16972
This adds collation support for string types to the v3 spec, following the discussion in the dev-list thread. A string field can carry a collation (e.g. icu.en_US-ci) that defines case-insensitive, accent-insensitive, or locale-aware comparison and ordering, and a new data_file.collation_bounds field carries collation-aware min/max so collated columns can still be pruned The spec changes: - A collation attribute on string fields (provider-qualified, e.g. icu.en_US-ci; utf8 means byte order), stored unversioned in the schema so any compatible engine can read. - A new data_file.collation_bounds field (id 147): per column, a list of collation-aware lower/upper bounds stored as original values and tagged with the collation and the implementation version they were selected under. - Reader/writer rules: byte-order lower_bounds/upper_bounds are still written for collated columns so collation-unaware engines stay correct, but must not be used to prune predicates on a collated column; a collation_bounds entry may be used only on an exact collation + version match. Two decisions I'd flag for discussion. - First, bounds store original values rather than ICU sort keys: sort keys aren't stable across UCA/CLDR/ICU versions, so per-file versioning plus an exact-match read gate degrades gracefully where a pinned global version would break. - Second, I put collation_bounds on data_file as a standalone v3 field, but it could instead live in the v4 content_stats typed-stats struct - worth deciding which. Backed by a working reference implementation in iceberg-go: https://github.com/apache/iceberg-go/pull/1318. Rationale and the differences from the [original proposal](https://docs.google.com/document/d/1m8b7u97uteHYjXk-4DNglJSpQO8OcZOCzW2tApCNTW4/edit?tab=t.0#heading=h.y1ant4w2163k) are written up separately ([proposal doc](https://github.com/laskoviymishka/iceberg/blob/collation-support-proposal/format/collation-support-proposal.md)). See also the [dev-list thread](https://lists.apache.org/thread/zdvc0gmqjgws8whwxv0ztn7h1jqclzyk). Feedback very welcome. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
