laskoviymishka commented on code in PR #16972: URL: https://github.com/apache/iceberg/pull/16972#discussion_r3501702072
########## format/spec.md: ########## @@ -345,6 +346,33 @@ For example, a struct column `point` with fields `x` (default 0) and `y` (defaul Default values are attributes of fields in schemas and serialized with fields in the JSON format. See [Appendix C](#appendix-c-json-serialization). +#### Collations + +A `string` field may carry a **collation**, an attribute that changes how the field's values are compared and ordered without changing how they are stored. Collations enable case-insensitive, accent-insensitive, and locale-aware comparison and sorting. A collation only affects comparison: the stored value is returned unchanged (a value written as `'appLE'` is read back as `'appLE'`). + +A field's collation is stored as a `collation` attribute on the field (see [Appendix C](#appendix-c-json-serialization)). The attribute is allowed only on `string` fields. If a field has no `collation` attribute, comparison defaults to UTF-8 byte order, which is the behavior of all prior versions. + +A collation is identified by a provider-qualified name of the form `<provider>.<name>`, for example `icu.en_US-ci`. The provider names the library that defines the collation (`icu` for collations defined by the [Unicode Collation Algorithm](https://unicode.org/reports/tr10/) over [CLDR](https://cldr.unicode.org/) locale data; other providers may define engine-specific collations such as case-folding variants). The name selects a locale and optional modifiers for case sensitivity (`ci`/`cs`), accent sensitivity (`ai`/`as`), trimming, and case folding. The reserved name `utf8` denotes UTF-8 byte-order comparison. + +The schema stores the collation **name without a version**, so any engine that supports the collation can read the table. UCA, DUCET, CLDR, and ICU collation orders are [not stable across versions](https://unicode.org/reports/tr10/#Non-Goals), so collation-aware metrics carry the implementation version they were produced under (see below) and a reader uses them only when it can produce the same order. + +##### Collation Bounds + +Because a collation can reorder values (for example `'a' < 'B'` under a case-insensitive collation, but `'a' > 'B'` in byte order), the byte-order `lower_bounds` and `upper_bounds` cannot be used to evaluate predicates on a collated column. Collation-aware bounds are stored separately in `data_file.collation_bounds`, a map from column id to a list of `collation_bound` structs: + +| Field id, name | Type | Description | +|----------------|------|-------------| +| **`151 collation`** | `string` | Collation the bounds were produced for, e.g. `icu.en_US-ci` | Review Comment: Agreed, let's go with the schema-declared registry: the string field keeps its UTF-8 metrics under its own id, plus a map of collation -> metric field id, and a data file populates only the collation fields it has valid bounds for. Field-id native and resolvable at planning time, I like it. On versioning I think we're actually saying the same thing, let me confirm with an example so we don't talk past each other. German (de) is a good case. The letter ä has code point U+00E4, so in UTF-8 byte order it sorts after z, but German collation sorts it with a, at the front. So the byte-order lower/upper bounds already bound the wrong thing, which is why the column needs collation bounds at all. Now the version point. German collation can order ä two ways: like 'a' (the standard/dictionary order) or like 'ae' (the phonebook order). Say a file holds "Apfel" and "Äpfel". Under the ä-as-'a' order, "Apfel" < "Äpfel", so the writer stores lower="Apfel". If a later collation version switches the tailoring to ä-as-'ae', then "Äpfel" reads as "Aepfel" and the second letter decides -- e < p -- so "Äpfel" < "Apfel", i.e. the real minimum is now "Äpfel". A reader on the new version running col < "Af" should match the file ("Aepfel" < "Af"), but if it trusts the old bound lower="Apfel" it concludes nothing is below "Af", skips the file, and silently drops the row. So the reader has to know which version produced the bound, and decide, from its own library, whether that ordering matches the one it will use. That "produced under version X" tag is all I meant by encoding the version: provenance, not a validity range. I'm not proposing the format declare which library versions a bound is valid for - that's engine knowledge. And the version I mean is ICU's collator version (Collator.getVersion()), which by design changes only when collation results can change - ICU recommends storing it alongside any persisted collation data for exactly this reason. So most bumps don't produce a new entry at all: a maintenance release like ICU 74.1 -> 74.2, or a Unicode update that only adds new code points (new emoji, a new script), leaves every German letter's weight untouched. getVersion() for de is identical, the old bound is still exactly valid, and the engine just reuses it — no new field id, no new bounds. A new entry shows up only when getVersion() actually moves, like the ä-tailoring change above. That's your version-bump point: if the ordering doesn't change, the engine keeps using the existing collation. Given that, the separate valid-for-versions set I floated earlier is no required, so can be skiped. The (name, collator version) on the registered entry carries what a reader needs. I'll rework the bounds to this shape, collation metrics moving into content_stats keyed by the registered field ids. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
