github-actions[bot] commented on code in PR #67918:
URL: https://github.com/apache/doris/pull/67918#discussion_r4056591606
##########
fe/fe-core/src/main/java/org/apache/doris/indexpolicy/IndexPolicy.java:
##########
@@ -59,7 +59,8 @@ public class IndexPolicy implements Writable,
GsonPostProcessable {
public static final String PROP_TOKEN_FILTER = "token_filter";
public static final String PROP_CHAR_FILTER = "char_filter";
public static final Set<String> BUILTIN_TOKENIZERS = ImmutableSet.of(
- "empty", "ngram", "edge_ngram", "keyword", "standard",
"char_group", "basic", "icu", "pinyin");
+ "empty", "ngram", "edge_ngram", "keyword", "standard",
"char_group", "basic", "icu", "pinyin",
Review Comment:
[P1] Keep `IF NOT EXISTS` idempotent for replayed tokenizer names. Before
this change, `ik_smart` and `ik_max_word` were legal user policy names, so an
upgraded FE can legitimately replay one. `createIndexPolicy()` now checks this
expanded built-in set before its existing-name/`ifNotExists` return, so
repeating the same `CREATE INVERTED INDEX TOKENIZER IF NOT EXISTS ik_smart ...`
starts throwing a built-in-name conflict even though no policy would be
created. Resolve the existing-policy no-op first when `IF NOT EXISTS` is set,
while still rejecting a fresh creation with either reserved name, and cover the
replayed legacy-policy case.
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/InvertedIndexUtil.java:
##########
@@ -337,23 +339,43 @@ private static void
checkInvertedIndexProperties(Map<String, String> properties,
// dict_compression now silently ignores by V2/V3 inverted index
}
- // Normalize analyzer and normalizer names to lowercase for
case-insensitive matching
+ // Canonicalize built-ins while retaining the exact spelling of a
resolved legacy policy.
normalizeInvertedIndexProperties(properties);
}
/**
- * Normalize analyzer and normalizer names in index properties to
lowercase.
- * This ensures case-insensitive matching between table creation and query
time.
+ * Canonicalize analyzer and normalizer names in index properties. Legacy
metadata may contain
+ * case-distinct policy names, so a resolved custom policy must keep its
exact stored name.
*/
private static void normalizeInvertedIndexProperties(Map<String, String>
properties) {
+ normalizeResolvedPolicyName(properties,
INVERTED_INDEX_ANALYZER_NAME_KEY);
+ normalizeResolvedPolicyName(properties,
INVERTED_INDEX_NORMALIZER_NAME_KEY);
AnalyzerKeyNormalizer.normalizeInvertedIndexProperties(
properties,
- INVERTED_INDEX_ANALYZER_NAME_KEY,
- INVERTED_INDEX_NORMALIZER_NAME_KEY,
INVERTED_INDEX_PARSER_KEY,
INVERTED_INDEX_PARSER_KEY_ALIAS);
}
+ private static void normalizeResolvedPolicyName(Map<String, String>
properties, String key) {
+ String name = properties.get(key);
+ if (name == null || name.isEmpty()) {
+ return;
+ }
+ properties.put(key, resolveAnalyzerName(name));
+ }
+
+ /** Resolve built-in names and retain the stored spelling of custom
policies. */
+ public static String resolveAnalyzerName(String name) {
+ String trimmedName = name.trim();
+ // Match the BE writer's case-sensitive built-in dispatch before
policy lookup.
+ if (IndexPolicy.BUILTIN_ANALYZERS.contains(trimmedName)
+ || IndexPolicy.BUILTIN_NORMALIZERS.contains(trimmedName)) {
+ return trimmedName;
+ }
+ IndexPolicy policy =
Env.getCurrentEnv().getIndexPolicyMgr().getPolicyByName(trimmedName);
Review Comment:
[P1] Validate the exact legacy binding before treating this as a built-in. A
Turkish-locale old FE could persist a TOKENIZER named `IK` (legacy lowercasing
produced dotless `ık`, so it did not collide with built-in `ik`). After replay,
`validateAnalyzerExists("IK")` now normalizes to `ik` and returns as a built-in
without checking that exact policy, while this resolver preserves `IK`; BE
exact-first lookup then selects the TOKENIZER and fails with "Analyzer policy
not found". Thus CREATE/ALTER can pass FE validation and produce an index/query
configuration that fails only at BE setup. This is distinct from the earlier
valid-exact-analyzer binding fix: resolve and type/validity-check an exact
saved policy first, using the built-in fallback only when no exact policy
exists, and cover a replayed wrong-type `IK`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
