airborne12 commented on code in PR #67918:
URL: https://github.com/apache/doris/pull/67918#discussion_r4065296421
##########
be/src/storage/index/inverted/tokenizer/icu/icu_tokenizer.cpp:
##########
@@ -55,13 +55,23 @@ Token* ICUTokenizer::next(Token* token) {
utf8Str_.clear();
int32_t length = std::min(end - start, LUCENE_MAX_WORD_LEN);
auto subString = buffer_.tempSubString(start, length);
+ sourceUtf8Str_.clear();
+ subString.toUTF8String(sourceUtf8Str_);
if (this->lowercase) {
subString.toLower().toUTF8String(utf8Str_);
} else {
subString.toUTF8String(utf8Str_);
}
token->setNoCopy(utf8Str_.data(), 0,
static_cast<int32_t>(utf8Str_.size()));
+ int32_t source_start = 0;
+ int32_t source_end = 0;
+ if (start >= 0 && length >= 0 && advance_source_offset(start,
source_start) &&
+ advance_source_offset(start + length, source_end)) {
Review Comment:
Fixed in 31450230. The ICU cap now backs up one UTF-16 unit when the limit
would split a lead/trail surrogate pair, so the term and source endpoint stay
on the same Unicode scalar boundary. The new supplementary-boundary test
verifies term bytes, source offsets, provenance, and reset reuse.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
