airborne12 commented on code in PR #67918:
URL: https://github.com/apache/doris/pull/67918#discussion_r4061525960
##########
be/src/storage/index/inverted/token_filter/icu_normalizer_filter.cpp:
##########
@@ -60,13 +64,45 @@ Token* ICUNormalizerFilter::next(Token* t) {
_output_buffer.clear();
result16.toUTF8String(_output_buffer);
+ if (std::string_view(buffer, length) != std::string_view(_output_buffer)) {
+ int32_t offset = 0;
+ int32_t rune_count = 0;
+ const auto normalized_length =
static_cast<int32_t>(_output_buffer.size());
+ while (offset < normalized_length) {
+ UChar32 code_point;
+ U8_NEXT(_output_buffer, offset, normalized_length, code_point);
+ DORIS_CHECK_GE(code_point, 0);
+ ++rune_count;
+ }
+
+ const int32_t source_length = t->endOffset() - t->startOffset();
+ DORIS_CHECK_GE(source_length, 0);
+ _source_byte_offsets.assign(rune_count + 1, 0);
Review Comment:
Fixed in 6ecc7e14. ICUNormalizerFilter now retains the source-offset opt-in
flag, skips provenance state when no downstream consumer requests it, and
represents changed-token provenance as one compact conservative [0,
source_length] span when enabled. Pinyin consumes that span without forcing
dense normalizer maps. Added ASAN coverage for both disabled and enabled
large-input paths; exact-map getters remain empty and the compact span is
verified. Validation: 203/203 related ASAN tests, full ASAN BE build, and
clang-tidy passed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
