github-actions[bot] commented on code in PR #67918:
URL: https://github.com/apache/doris/pull/67918#discussion_r4093531911
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/invertedindex/AnalyzerIdentityBuilder.java:
##########
@@ -116,6 +319,30 @@ private static String resolveAnalyzerIdentity(String
analyzerName, String defaul
}
}
+ /** Whether BE builds the built-in normalizer for this name; an exact
legacy policy shadows it. */
+ private static boolean isBuiltinNormalizerBinding(String name) {
+ try {
+ Env env = Env.getCurrentEnv();
+ if (env != null && env.getIndexPolicyMgr() != null) {
+ return env.getIndexPolicyMgr().getTopLevelBuiltin(
+ name, IndexPolicy.BUILTIN_NORMALIZERS) != null;
+ }
+ } catch (RuntimeException e) {
+ // Fall through to the name-only answer.
+ }
+ return IndexPolicy.BUILTIN_NORMALIZERS.contains(
+ Strings.nullToEmpty(name).trim().toLowerCase(Locale.ROOT));
+ }
+
+ /**
+ * BE builds a built-in normalizer as the keyword tokenizer plus the
built-in token filter of
+ * the canonical name, so it shares the identity of that custom pipeline.
+ */
+ private static String builtinNormalizerIdentity(String name) {
+ return
buildIdentityFromPolicyProperties(IndexPolicyTypeEnum.NORMALIZER,
+ Map.of(IndexPolicy.PROP_TOKEN_FILTER,
name.trim().toLowerCase(Locale.ROOT)));
Review Comment:
[P1] Keep the built-in normalizer identity out of nested policy lookup. The
existing replay coverage establishes that a normalized-only `LOWERCASE`
NORMALIZER does not shadow the top-level built-in, but older releases also
allowed a case-distinct TOKEN_FILTER `LOWERCASE` (for example,
`type=asciifolding`). Here `normalizer=LowerCase` still reaches BE's built-in
lowercase normalizer, while canonicalizing its component to `lowercase` lets
TOKEN_FILTER normalized fallback resolve that legacy filter. FE can then reject
it plus a named ASCII-folding normalizer as duplicates although their runtime
analyzers differ. Please encode keyword-plus-built-in-lowercase directly and
cover this wrong-family collision in CREATE/ALTER tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
