airborne12 commented on code in PR #67918:
URL: https://github.com/apache/doris/pull/67918#discussion_r4093228577
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/InvertedIndexUtil.java:
##########
@@ -423,17 +488,27 @@ public static boolean isAnalyzerMatched(Map<String,
String> properties, String a
buildAnalyzerIdentity(properties));
}
+ String resolvedAnalyzer = resolveAnalyzerName(normalizedAnalyzer);
String preferredAnalyzer =
InvertedIndexProperties.getPreferredAnalyzer(properties);
if (!Strings.isNullOrEmpty(preferredAnalyzer)) {
- return normalizedAnalyzer.equalsIgnoreCase(preferredAnalyzer);
+ return
resolvedAnalyzer.equals(resolveAnalyzerName(preferredAnalyzer))
+ && (!INVERTED_INDEX_PARSER_IK.equals(resolvedAnalyzer)
+ || matchesBuiltinIkDefaults(properties));
}
String parser =
InvertedIndexProperties.getInvertedIndexParser(properties);
if (Strings.isNullOrEmpty(parser)) {
- return normalizedAnalyzer.equalsIgnoreCase("default")
- ||
normalizedAnalyzer.equalsIgnoreCase(INVERTED_INDEX_PARSER_NONE);
+ return resolvedAnalyzer.equals("default")
+ || resolvedAnalyzer.equals(INVERTED_INDEX_PARSER_NONE);
}
- return normalizedAnalyzer.equalsIgnoreCase(parser);
+ return resolvedAnalyzer.equals(parser.trim().toLowerCase(Locale.ROOT))
+ && (!INVERTED_INDEX_PARSER_IK.equals(resolvedAnalyzer)
Review Comment:
Confirmed and fixed in 2e22e296ea6.
The regression is real. On master `isAnalyzerMatched()` compared names only,
so `MATCH ... USING ANALYZER ik` bound any `parser=ik` index. 45015525902 made
explicit built-in IK select by effective configuration
(`matchesBuiltinIkDefaults`), which is what a column with several IK indexes
needs, but it also turned the only IK index of a pre-upgrade table into a miss:
`parser=ik` without a mode is `ik_smart`, the built-in default is
`ik_max_word`, the identities differ, and `visitMatch` threw `No inverted index
found for analyzer 'ik'` for SQL that worked before.
Fix: `OlapTable.filterIndexesByAnalyzer()` keeps the configuration match
first, and only when nothing matches it falls back to the single index that
carries the requested name (`InvertedIndexUtil.isAnalyzerNameMatched()`, the
pre-45015525902 check split out of `isAnalyzerMatched()`). Two differently
configured legacy indexes stay ambiguous and still fail, and
`isAnalyzerMatched()` itself is unchanged, so the existing assertions on it
hold. The fallback is coherent end to end: `MatchPredicate` serializes the
selected index's own `parser_mode`/`lower_case` (`ik_smart`/true for such an
index), and BE builds both the reader key and the query analyzer from those
fields, so the query is tokenized the way the index was.
Tests:
`InvertedIndexPropertiesTest.testExplicitBuiltinIkFallsBackToTheOnlyLegacyIndex`
covers the single legacy index (selected, Thrift carries `ik_smart` +
lowercase), legacy plus default (`analyzer=ik` still wins), and two legacy
indexes (still null); with the fallback reverted the single-index assertion
fails. `test_analyzer_identity_semantics` gains a table whose only index is
`PROPERTIES("parser"="ik")`: `MATCH 'abc' USING ANALYZER ik` returns rows 1 and
4, `MATCH '清华大学'` returns row 3, and `MATCH '清华'` returns nothing, which is
what a smart-mode binding gives and what an `ik_max_word` binding would not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
