github-actions[bot] commented on code in PR #67918:
URL: https://github.com/apache/doris/pull/67918#discussion_r4060346329
##########
be/src/storage/index/inverted/token_filter/icu_normalizer_filter.cpp:
##########
@@ -60,13 +64,45 @@ Token* ICUNormalizerFilter::next(Token* t) {
_output_buffer.clear();
result16.toUTF8String(_output_buffer);
+ if (std::string_view(buffer, length) != std::string_view(_output_buffer)) {
+ int32_t offset = 0;
+ int32_t rune_count = 0;
+ const auto normalized_length =
static_cast<int32_t>(_output_buffer.size());
+ while (offset < normalized_length) {
+ UChar32 code_point;
+ U8_NEXT(_output_buffer, offset, normalized_length, code_point);
+ DORIS_CHECK_GE(code_point, 0);
+ ++rune_count;
+ }
+
+ const int32_t source_length = t->endOffset() - t->startOffset();
+ DORIS_CHECK_GE(source_length, 0);
+ _source_byte_offsets.assign(rune_count + 1, 0);
Review Comment:
[P1] Do not build these two dense provenance maps unless a downstream
consumer requested them. A valid `empty -> icu_normalizer` analyzer passes the
entire value as one token; with default `nfkc_cf`, a 100 MiB mostly-lowercase
ASCII value containing one uppercase byte takes this branch and allocates about
800 MiB across the two `int32_t` vectors, even without Pinyin. Analyzed values
bypass `ignore_above`, so concurrent writers can exhaust BE memory. This is
separate from the earlier `ICUNormalizerCharFilter` byte-map thread and is a
resource consequence of the conservative provenance fix. Retain the enable flag
in this filter, skip unused maps, and represent enabled whole-span provenance
compactly (or bound/reserve it), with large-input coverage.
##########
be/src/storage/index/inverted/tokenizer/icu/icu_tokenizer.cpp:
##########
@@ -73,6 +83,24 @@ void ICUTokenizer::reset() {
if (!buffer_.isEmpty() && buffer_.isBogus()) {
_CLTHROWT(CL_ERR_Runtime, "Failed to convert UTF-8 string to
UnicodeString.");
}
+ utf16ToUtf8Offset_.assign(buffer_.length() + 1, 0);
Review Comment:
[P1] Avoid materializing an entry for every UTF-16 code unit in the full
input. A 100 MiB ASCII value makes `reset()` allocate roughly 400 MiB here
before the first token, on top of the input and the roughly 200 MiB
`UnicodeString`; analyzed values are not capped by `ignore_above`, so
concurrent index writers can exhaust the BE memory limit. This table also
supplies ordinary token spans, so simply gating it on Pinyin provenance would
be incorrect, but the break iterator advances monotonically: track the UTF-8
cursor incrementally or use compact checkpoints/runs while preserving exact
spans. This is distinct from the earlier `ICUNormalizerCharFilter` dense-table
thread, whose fix does not cover this tokenizer-owned table. Please add
large-input peak-memory coverage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
