github-actions[bot] commented on code in PR #67918:
URL: https://github.com/apache/doris/pull/67918#discussion_r4090843928
##########
be/src/storage/index/inverted/analyzer/custom_analyzer.cpp:
##########
@@ -70,12 +70,17 @@ ReaderPtr CustomAnalyzer::init_reader(ReaderPtr reader) {
}
TokenStreamComponentsPtr CustomAnalyzer::create_components() {
- auto tk = _tokenizer->create();
- TokenStreamPtr ts = tk;
- for (const auto& filter : _token_filters) {
- ts = filter->create(ts);
+ try {
+ auto tk = _tokenizer->create();
+ TokenStreamPtr ts = tk;
+ for (const auto& filter : _token_filters) {
+ ts = filter->create(ts);
+ }
+ return std::make_shared<TokenStreamComponents>(tk, ts);
+ } catch (const CLuceneError& e) {
+ throw Exception(ErrorCode::INVERTED_INDEX_ANALYZER_ERROR,
Review Comment:
[P1] Convert this lazy failure at the direct SEARCH tokenization boundary.
Analyzer/provider construction can succeed, then CustomAnalyzer::tokenStream()
reaches this catch on first use (the new IK tokenizer does exactly that when
Dictionary::initial() fails), but CLucene SEARCH TERM/PHRASE/ANY/ALL and the
SNII TERM minimum-should-match pre-analysis invoke get_analyse_result() outside
a catch. VSearchExpr calls this SEARCH path directly, so IFunction's generic
exception conversion does not cover it and the Doris Exception escapes the
Status API. Please route these calls through a Result helper (or wrap them) and
add a provider-succeeds/first-token-stream-fails case.
##########
be/src/storage/index/inverted/inverted_index_iterator.cpp:
##########
@@ -69,12 +70,21 @@ Status InvertedIndexIterator::read_from_index(const
IndexParam& param) {
// The execution context carries reader selection separately from analyzer
execution.
const std::string& analyzer_key =
(i_param->analyzer_ctx != nullptr) ?
i_param->analyzer_ctx->analyzer_key : "";
- auto reader =
- DORIS_TRY(select_best_reader(i_param->column_type,
i_param->query_type, analyzer_key));
+ const std::string& legacy_analyzer_key =
+ (i_param->analyzer_ctx != nullptr) ?
i_param->analyzer_ctx->legacy_analyzer_key : "";
+ auto reader = DORIS_TRY(select_best_reader(i_param->column_type,
i_param->query_type,
+ analyzer_key,
legacy_analyzer_key));
if (UNLIKELY(reader == nullptr)) {
return Status::Error<ErrorCode::INVERTED_INDEX_CLUCENE_ERROR>(
"inverted index reader is null");
}
+ // Check the reader that runs the query, not the first candidate of its
type, because the
+ // analyzer decides which index is selected and the two can disagree on
support_phrase.
+ if (is_phrase_query(i_param->query_type) &&
Review Comment:
[P1] Apply the selected-reader phrase-capability check to direct SEARCH too.
This guard only runs from InvertedIndexIterator::read_from_index(), but SEARCH
PHRASE calls FieldReaderResolver::resolve(), which invokes select_best_reader()
directly, then queries that reader or builds a PhraseQuery/MultiPhraseQuery
without ever entering read_from_index(). An index selected with
support_phrase=false can therefore reach a positional SEARCH query even though
equivalent MATCH is rejected. Please validate the returned reader in the shared
selection/binding path and cover both direct SEARCH execution formats.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
