sfc-gh-aloeser commented on code in PR #16972: URL: https://github.com/apache/iceberg/pull/16972#discussion_r4426713171
########## format/spec.md: ########## @@ -386,9 +387,81 @@ For example, a struct column `point` with fields `x` (default 0) and `y` (defaul Default values are attributes of fields in schemas and serialized with fields in the JSON format. See [Appendix C](#appendix-c-json-serialization). +#### Collations + +A `string` field may have a collation that defines how its values are compared and ordered, for example case-insensitive, accent-insensitive, or locale-aware. A collation does not change how values are stored: a value written as `'appLE'` is read as `'appLE'`. + +Any `string` field may have a collation, including nested fields. A struct field uses the `collation` attribute; a list or map uses `element-collation`, `key-collation`, or `value-collation` (see [Appendix C](#appendix-c-json-serialization)). A string field without a collation uses UTF-8 byte order. + +A collation changes comparison only. The rest of the format stays byte-based, because collation order differs between implementations and versions: + +* Partition transforms, `bucket`, and byte-order bounds use UTF-8 bytes. Values that are equal under a collation can be in different partitions or buckets, so readers must not use partition values to prune a predicate evaluated under a collation. +* Identifier fields compare bytes. +* Map keys are distinct by bytes: `map['ABC']` and `map['abc']` are separate entries under a case-insensitive key collation. A lookup under the collation can match both, and the result is engine-defined. +* Sort orders do not use collations. + +SQL comparison, equality, ordering, grouping, and distinctness are the engine's responsibility. + +##### Collation Names + +A collation name has the form `<provider>.<name>`, for example `icu.en_US-ci`. The provider is the library that defines the collation. This spec defines one provider, `icu`: the [Unicode Collation Algorithm](https://unicode.org/reports/tr10/) with [CLDR](https://cldr.unicode.org/) locale data. Adding a provider requires a spec change. A reader must not use a collation from a provider it does not recognize. + +``` +specifier = icu-spec | utf8-spec +icu-spec = "icu" "." locale [ "-" case-mod ] [ "-" accent-mod ] [ "-" trim-mod ] +utf8-spec = "utf8" [ "-" trim-mod ] +locale = language [ "_" region ] ; e.g. en, de, en_US, de_DE +case-mod = "cs" | "ci" ; case sensitive (default) / insensitive +accent-mod = "as" | "ai" ; accent sensitive (default) / insensitive +trim-mod = "ltrim" | "rtrim" | "trim" ; trim leading / trailing / both spaces (U+0020) +``` Review Comment: lets also document how this translates into ICU modifiers, just to make sure all engines agree. If there's room for interpretation, engines may produce incompatible metadata. See [this section](https://docs.google.com/document/d/1m8b7u97uteHYjXk-4DNglJSpQO8OcZOCzW2tApCNTW4/edit?tab=t.50g4t7yl7w7t#heading=h.jviurbu2812) of the original proposal for reference. ########## format/spec.md: ########## @@ -386,9 +387,81 @@ For example, a struct column `point` with fields `x` (default 0) and `y` (defaul Default values are attributes of fields in schemas and serialized with fields in the JSON format. See [Appendix C](#appendix-c-json-serialization). +#### Collations + +A `string` field may have a collation that defines how its values are compared and ordered, for example case-insensitive, accent-insensitive, or locale-aware. A collation does not change how values are stored: a value written as `'appLE'` is read as `'appLE'`. + +Any `string` field may have a collation, including nested fields. A struct field uses the `collation` attribute; a list or map uses `element-collation`, `key-collation`, or `value-collation` (see [Appendix C](#appendix-c-json-serialization)). A string field without a collation uses UTF-8 byte order. + +A collation changes comparison only. The rest of the format stays byte-based, because collation order differs between implementations and versions: + +* Partition transforms, `bucket`, and byte-order bounds use UTF-8 bytes. Values that are equal under a collation can be in different partitions or buckets, so readers must not use partition values to prune a predicate evaluated under a collation. +* Identifier fields compare bytes. +* Map keys are distinct by bytes: `map['ABC']` and `map['abc']` are separate entries under a case-insensitive key collation. A lookup under the collation can match both, and the result is engine-defined. +* Sort orders do not use collations. + +SQL comparison, equality, ordering, grouping, and distinctness are the engine's responsibility. + +##### Collation Names + +A collation name has the form `<provider>.<name>`, for example `icu.en_US-ci`. The provider is the library that defines the collation. This spec defines one provider, `icu`: the [Unicode Collation Algorithm](https://unicode.org/reports/tr10/) with [CLDR](https://cldr.unicode.org/) locale data. Adding a provider requires a spec change. A reader must not use a collation from a provider it does not recognize. + +``` +specifier = icu-spec | utf8-spec +icu-spec = "icu" "." locale [ "-" case-mod ] [ "-" accent-mod ] [ "-" trim-mod ] +utf8-spec = "utf8" [ "-" trim-mod ] +locale = language [ "_" region ] ; e.g. en, de, en_US, de_DE +case-mod = "cs" | "ci" ; case sensitive (default) / insensitive +accent-mod = "as" | "ai" ; accent sensitive (default) / insensitive +trim-mod = "ltrim" | "rtrim" | "trim" ; trim leading / trailing / both spaces (U+0020) +``` + +Modifier order and case are fixed, so equal collations have equal names. Case- and accent-insensitive collations require a locale: `ł` and `l` are equal under `icu.en-ai` but not under `icu.pl-ai`. + +Without a trim modifier, trailing spaces are significant (NO PAD). `rtrim` gives PAD SPACE behavior, for example `icu.en_US-ci-rtrim`. Review Comment: > `rtrim` gives PAD SPACE behavior I don't think this is completely true. While trimming and padding give the same results in most common scenarios, they behave differently for code points below U+0020 (space). With padding, we can get some unintuitive behaviors. For example: with pad semantics, U+000A (newline) sorts before an empty string, as does any code point below U+0020. With trimming, empty string <= any string with characters. This seems more intuitive (to me at least), and it's nice to have a value that can safely be used as a lower bound. For that reason, I much prefer the trimming version -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
