nssalian opened a new pull request, #17726: URL: https://github.com/apache/iceberg/pull/17726
## Rationale for this change Variant metadata stores field names in a dictionary. When its `sorted_strings` flag is set, [VariantEncoding.md](https://github.com/apache/parquet-format/blob/master/VariantEncoding.md) requires those names to be in UTF-8 byte order. Iceberg sorts and looks them up with Java's `String.compareTo`, which is UTF-16 order. For ASCII and most characters the two orders are identical, but they differ for characters above U+FFFF. So Iceberg can flag a dictionary as `sorted` when it is not actually UTF-8-ordered. Iceberg reads its own files fine because it sorts and searches the same way, but a spec-conforming reader (iceberg-go, iceberg-rust) searches by UTF-8 and would fail to find those fields. ## Changes Sort and look up Variant field names with the existing UTF-8 comparator `Comparators.charSequences()` at every site: - `Variants.metadata` - sets the dictionary `sorted_strings` flag - `VariantUtil.find` - reader lookup over field names - `ShreddedObject` - shredded object field order - `RecordConverter` (kafka-connect) and `ParquetMetrics` - sort field names before building the dictionary No on-disk format or spec change - this conforms Iceberg to the existing spec. Files with only ASCII or common field names are unaffected; only names above U+FFFF change ordering. ## Testing `TestVariantMetadataFieldOrdering` and a new `TestShreddedObject` case check that a dictionary is flagged `sorted` only when it is actually UTF-8-ordered, and that fields with characters above U+FFFF are found on read. The tests fail on the old `String.compareTo` code and pass on the fix (verified by reverting the change and watching them go red). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
