rraulinio opened a new issue, #1248: URL: https://github.com/apache/iceberg-go/issues/1248
### Apache Iceberg version main (development) ### Please describe the bug 🐞 # Description When projecting `NOT STARTS WITH` predicates through a `truncate[N]` string partition transform, iceberg-go checks whether the filter prefix fits within the truncate width. That check currently uses `len(StringLiteral)`, which counts bytes. But the string truncate transform itself truncates by Unicode characters/code points, not bytes. For ASCII strings those are the same, but for UTF-8 strings they can differ. Example: - partition transform: `truncate[2](name)` - filter: `name NOT STARTS WITH "éé"` - `"éé"` is 2 characters, but 4 bytes The prefix fits within `truncate[2]` in terms of Iceberg string truncate semantics, but the current byte-length check treats it as too long. That can make the projection choose the wrong fallback behavior for multibyte UTF-8 prefixes. The fix is to count string literals with `utf8.RuneCountInString` while keeping binary/fixed literals byte-counted. This matches the existing truncate implementation, which already truncates strings on UTF-8 code point boundaries. This is separate from, but adjacent to, the `NOT STARTS WITH` truncate projection bug in #1192/#1193. It was found while reviewing that same area. # Expected behavior String truncate projection should compare string prefix length using the same unit as the string truncate transform: Unicode characters/code points. Binary and fixed truncate projection should continue using byte length. # Impact This can affect pruning/projection correctness for tables partitioned by string `truncate[N]` when filters use multibyte UTF-8 prefixes. # Proposed fix Use `utf8.RuneCountInString(string(l))` for `StringLiteral` in the truncate projection length helper, and add regression coverage with prefixes such as `"é"`, `"éé"`, and `"ééa"`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
