airborne12 commented on code in PR #67918:
URL: https://github.com/apache/doris/pull/67918#discussion_r4056693839
##########
fe/fe-core/src/main/java/org/apache/doris/analysis/InvertedIndexUtil.java:
##########
@@ -337,23 +339,43 @@ private static void
checkInvertedIndexProperties(Map<String, String> properties,
// dict_compression now silently ignores by V2/V3 inverted index
}
- // Normalize analyzer and normalizer names to lowercase for
case-insensitive matching
+ // Canonicalize built-ins while retaining the exact spelling of a
resolved legacy policy.
normalizeInvertedIndexProperties(properties);
}
/**
- * Normalize analyzer and normalizer names in index properties to
lowercase.
- * This ensures case-insensitive matching between table creation and query
time.
+ * Canonicalize analyzer and normalizer names in index properties. Legacy
metadata may contain
+ * case-distinct policy names, so a resolved custom policy must keep its
exact stored name.
*/
private static void normalizeInvertedIndexProperties(Map<String, String>
properties) {
+ normalizeResolvedPolicyName(properties,
INVERTED_INDEX_ANALYZER_NAME_KEY);
+ normalizeResolvedPolicyName(properties,
INVERTED_INDEX_NORMALIZER_NAME_KEY);
AnalyzerKeyNormalizer.normalizeInvertedIndexProperties(
properties,
- INVERTED_INDEX_ANALYZER_NAME_KEY,
- INVERTED_INDEX_NORMALIZER_NAME_KEY,
INVERTED_INDEX_PARSER_KEY,
INVERTED_INDEX_PARSER_KEY_ALIAS);
}
+ private static void normalizeResolvedPolicyName(Map<String, String>
properties, String key) {
+ String name = properties.get(key);
+ if (name == null || name.isEmpty()) {
+ return;
+ }
+ properties.put(key, resolveAnalyzerName(name));
+ }
+
+ /** Resolve built-in names and retain the stored spelling of custom
policies. */
+ public static String resolveAnalyzerName(String name) {
+ String trimmedName = name.trim();
+ // Match the BE writer's case-sensitive built-in dispatch before
policy lookup.
+ if (IndexPolicy.BUILTIN_ANALYZERS.contains(trimmedName)
+ || IndexPolicy.BUILTIN_NORMALIZERS.contains(trimmedName)) {
+ return trimmedName;
+ }
+ IndexPolicy policy =
Env.getCurrentEnv().getIndexPolicyMgr().getPolicyByName(trimmedName);
Review Comment:
Fixed in `191a43c4a77`.
`validateAnalyzerExists()` and `validateNormalizerExists()` now resolve an
exact saved policy while holding the policy-manager read lock before
considering the normalized built-in fallback. If that exact policy exists, its
type and validity are checked directly. A replayed TOKENIZER named `IK`
therefore fails analyzer validation instead of being accepted as built-in `ik`;
the same rule applies to a wrong-type exact `LOWERCASE` policy, while canonical
lowercase built-in names still work when no exact policy exists.
`PolicyValidatorTests.testExactLegacyPolicyPrecedesBuiltinValidation` covers
both wrong-type legacy bindings and the built-in fallback. The test failed
before the change because `IK` was accepted, then passed with the fix. The
complete `PolicyValidatorTests` class passed 38/38 after the master rebase, and
`./build.sh --fe` completed successfully with zero Checkstyle violations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
