wgtmac commented on code in PR #760:
URL: https://github.com/apache/iceberg-cpp/pull/760#discussion_r3490460950
##########
src/iceberg/util/string_util.h:
##########
@@ -41,29 +40,46 @@ concept FromChars = requires(const char* p, T& v) {
std::from_chars(p, p, v); };
class ICEBERG_EXPORT StringUtils {
public:
- // NOTE: These convert ASCII letters only; all other bytes, including
non-ASCII
- // (multibyte UTF-8) bytes, are passed through unchanged.
- // See https://github.com/apache/iceberg-cpp/issues/613.
- static std::string ToLower(std::string_view str) {
- return str | std::ranges::views::transform(ToLowerAscii) |
- std::ranges::to<std::string>();
- }
-
+ /// \brief Lower-case a UTF-8 string using Unicode simple (1:1) case mapping.
+ ///
+ /// Intended for case-insensitive name matching, similar to Iceberg Java's
+ /// toLowerCase(Locale.ROOT). The mapping is locale-independent, matching
the intent
+ /// of Locale.ROOT. It uses simple (1:1) case mapping rather than Java's
full case
+ /// mapping, so results differ for a few code points; e.g. U+0130 (capital I
with dot
+ /// above) maps to U+0069 ("i") here, but to U+0069 U+0307 ("i" + combining
dot above)
+ /// in Java. For ASCII and the large majority of letters the two agree.
+ ///
+ /// Invalid UTF-8 input is returned unchanged.
Review Comment:
Why not using `Result<std::string>` so we can return an error for invalid
UTF8 input.
##########
src/iceberg/util/string_util.h:
##########
@@ -41,29 +40,46 @@ concept FromChars = requires(const char* p, T& v) {
std::from_chars(p, p, v); };
class ICEBERG_EXPORT StringUtils {
public:
- // NOTE: These convert ASCII letters only; all other bytes, including
non-ASCII
- // (multibyte UTF-8) bytes, are passed through unchanged.
- // See https://github.com/apache/iceberg-cpp/issues/613.
- static std::string ToLower(std::string_view str) {
- return str | std::ranges::views::transform(ToLowerAscii) |
- std::ranges::to<std::string>();
- }
-
+ /// \brief Lower-case a UTF-8 string using Unicode simple (1:1) case mapping.
+ ///
+ /// Intended for case-insensitive name matching, similar to Iceberg Java's
+ /// toLowerCase(Locale.ROOT). The mapping is locale-independent, matching
the intent
+ /// of Locale.ROOT. It uses simple (1:1) case mapping rather than Java's
full case
+ /// mapping, so results differ for a few code points; e.g. U+0130 (capital I
with dot
+ /// above) maps to U+0069 ("i") here, but to U+0069 U+0307 ("i" + combining
dot above)
+ /// in Java. For ASCII and the large majority of letters the two agree.
+ ///
+ /// Invalid UTF-8 input is returned unchanged.
+ /// See https://github.com/apache/iceberg-cpp/issues/613.
+ static std::string ToLower(std::string_view str);
+
+ /// \brief Upper-case the ASCII letters (a-z) in a string; all other bytes,
including
+ /// multi-byte UTF-8 sequences, are left unchanged.
+ ///
+ /// Deliberately ASCII-only and, unlike ToLower, not Unicode-aware. It is
only used to
+ /// normalize ASCII enum/codec strings (e.g. "gzip" -> "GZIP", "all" ->
"ALL") for
+ /// case-insensitive comparison. A Unicode upper-case is intentionally not
provided:
+ /// simple case mapping would be wrong for some letters (e.g. "ß" (U+00DF)
would stay
+ /// unchanged instead of becoming "SS"), and no caller needs it.
static std::string ToUpper(std::string_view str) {
return str | std::ranges::views::transform(ToUpperAscii) |
std::ranges::to<std::string>();
}
+ /// \brief Case-insensitive equality; compares the ToLower forms of both
operands and
+ /// therefore inherits ToLower's Unicode simple-mapping behavior.
static bool EqualsIgnoreCase(std::string_view lhs, std::string_view rhs) {
- return std::ranges::equal(
- lhs, rhs, [](char lc, char rc) { return ToLowerAscii(lc) ==
ToLowerAscii(rc); });
+ return ToLower(lhs) == ToLower(rhs);
}
+ /// \brief Case-insensitive prefix test, comparing the ToLower forms of both
inputs.
+ ///
+ /// Inherits ToLower's Unicode simple-mapping behavior. The whole strings are
+ /// lower-cased rather than byte-slicing str to prefix.size(), because
ToLower can
+ /// change a string's byte length (e.g. "İ" (U+0130) is two bytes but maps
to "i"),
+ /// so a byte slice could split a code point or reject a valid match.
static bool StartsWithIgnoreCase(std::string_view str, std::string_view
prefix) {
- if (str.size() < prefix.size()) {
- return false;
- }
- return EqualsIgnoreCase(str.substr(0, prefix.size()), prefix);
+ return ToLower(str).starts_with(ToLower(prefix));
Review Comment:
ditto
##########
src/iceberg/util/string_util.h:
##########
@@ -41,29 +40,46 @@ concept FromChars = requires(const char* p, T& v) {
std::from_chars(p, p, v); };
class ICEBERG_EXPORT StringUtils {
public:
- // NOTE: These convert ASCII letters only; all other bytes, including
non-ASCII
- // (multibyte UTF-8) bytes, are passed through unchanged.
- // See https://github.com/apache/iceberg-cpp/issues/613.
- static std::string ToLower(std::string_view str) {
- return str | std::ranges::views::transform(ToLowerAscii) |
- std::ranges::to<std::string>();
- }
-
+ /// \brief Lower-case a UTF-8 string using Unicode simple (1:1) case mapping.
+ ///
+ /// Intended for case-insensitive name matching, similar to Iceberg Java's
+ /// toLowerCase(Locale.ROOT). The mapping is locale-independent, matching
the intent
+ /// of Locale.ROOT. It uses simple (1:1) case mapping rather than Java's
full case
+ /// mapping, so results differ for a few code points; e.g. U+0130 (capital I
with dot
+ /// above) maps to U+0069 ("i") here, but to U+0069 U+0307 ("i" + combining
dot above)
+ /// in Java. For ASCII and the large majority of letters the two agree.
+ ///
+ /// Invalid UTF-8 input is returned unchanged.
+ /// See https://github.com/apache/iceberg-cpp/issues/613.
+ static std::string ToLower(std::string_view str);
+
+ /// \brief Upper-case the ASCII letters (a-z) in a string; all other bytes,
including
+ /// multi-byte UTF-8 sequences, are left unchanged.
+ ///
+ /// Deliberately ASCII-only and, unlike ToLower, not Unicode-aware. It is
only used to
+ /// normalize ASCII enum/codec strings (e.g. "gzip" -> "GZIP", "all" ->
"ALL") for
+ /// case-insensitive comparison. A Unicode upper-case is intentionally not
provided:
+ /// simple case mapping would be wrong for some letters (e.g. "ß" (U+00DF)
would stay
+ /// unchanged instead of becoming "SS"), and no caller needs it.
static std::string ToUpper(std::string_view str) {
return str | std::ranges::views::transform(ToUpperAscii) |
std::ranges::to<std::string>();
}
+ /// \brief Case-insensitive equality; compares the ToLower forms of both
operands and
+ /// therefore inherits ToLower's Unicode simple-mapping behavior.
static bool EqualsIgnoreCase(std::string_view lhs, std::string_view rhs) {
- return std::ranges::equal(
- lhs, rhs, [](char lc, char rc) { return ToLowerAscii(lc) ==
ToLowerAscii(rc); });
+ return ToLower(lhs) == ToLower(rhs);
Review Comment:
Since `ToLower` returns as is for invalid UTF8 input, this line is a UB
instead.
##########
src/iceberg/util/string_util.h:
##########
@@ -41,29 +40,46 @@ concept FromChars = requires(const char* p, T& v) {
std::from_chars(p, p, v); };
class ICEBERG_EXPORT StringUtils {
public:
- // NOTE: These convert ASCII letters only; all other bytes, including
non-ASCII
- // (multibyte UTF-8) bytes, are passed through unchanged.
- // See https://github.com/apache/iceberg-cpp/issues/613.
- static std::string ToLower(std::string_view str) {
Review Comment:
One concern I still have is that the current ASCII-only behavior was a
reasonable trade-off for most existing call sites: metrics modes, UUIDs, HTTP
header/property names, enum-like strings, etc. are overwhelmingly ASCII. Moving
`ToLower`, `EqualsIgnoreCase`, and `StartsWithIgnoreCase` to always decode
UTF-8 and allocate lower-cased strings means the common path pays for Unicode
support even when it is not needed.
This feels a bit unfortunate for a minor/non-ASCII case. In SQL engines we
often keep ASCII-only fast paths for identifier/string handling, because most
real-world names are ASCII and these helpers can sit on hot-ish paths.
Could we think about preserving the fast path here? For example:
- keep the existing byte-wise ASCII path when both inputs are ASCII;
- only fall back to utf8proc when non-ASCII bytes are actually present;
- make `EqualsIgnoreCase` streaming/short-circuiting instead of
materializing two lower-cased strings;
- avoid lower-casing the whole `str` in `StartsWithIgnoreCase` when only the
prefix needs to be compared.
I’m supportive of improving Unicode behavior, but I’d prefer not to regress
the common ASCII path globally if we can avoid it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
