924060929 commented on code in PR #68998:
URL: https://github.com/apache/doris/pull/68998#discussion_r4255623057
##########
fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java:
##########
@@ -922,6 +927,42 @@ private List<ConnectorPartitionInfo>
restrictToPin(HudiTableHandle handle, List<
return atPin;
}
+ private List<String> hmsPartitionPaths(HudiTableHandle handle,
List<String> names) {
+ List<String> encodedNames = names.stream().filter(name ->
name.indexOf('%') >= 0)
+ .collect(Collectors.toList());
+ if (encodedNames.isEmpty()) {
+ return names;
+ }
+ // Hive sync can register an already escaped Hudi value, which HMS
escapes again in its name.
+ // The storage location identifies the physical path regardless of the
sync extractor's decoding
+ // policy. Decode that path only in parsePartitionValues, preserving
literal percent sequences.
+ List<HmsPartitionInfo> partitions = hmsClient.getPartitions(
Review Comment:
[P2] Independently confirmed at `0d0163fc17d6151c60c9cc4a711b311034bb5a93`.
The new location lookup runs before partition pruning, while `CachingHmsClient`
caches names and partition objects independently. A stale encoded name
therefore makes a query for a surviving partition fail when the object lookup
misses; the merge-base path did not perform this strict lookup. Please use
`getExistingPartitions`, associate returned objects by partition identity
rather than list index, and omit confirmed deleted names while retaining
malformed-result errors. Add a test with a cached encoded name whose HMS
partition has been deleted.
##########
fe/fe-filesystem/fe-filesystem-spi/src/main/java/org/apache/doris/filesystem/spi/GlobPattern.java:
##########
@@ -0,0 +1,112 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.filesystem.spi;
+
+import java.util.LinkedHashSet;
+import java.util.Set;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+/** Numeric-range normalization shared by object storage and Hadoop glob
listing. */
+public final class GlobPattern {
+ private static final int MAX_EXPANSIONS = 10000;
+ private static final Pattern RANGE =
Pattern.compile("(-?\\d+)\\.\\.(-?\\d+)");
+ private static final Pattern SIMPLE_RANGE =
Pattern.compile("\\d+\\.\\.\\d+");
+ private static final Pattern BRACE_GROUP =
Pattern.compile("\\{([^{}]*)\\}");
Review Comment:
[P2] Independently confirmed at `0d0163fc17d6151c60c9cc4a711b311034bb5a93`
against the merge-base normalizer. For `data_{1..2,\{special\}}.csv`, the outer
group is not matched, so the numeric alternative remains literal and
`data_1.csv` / `data_2.csv` are omitted. If the literal-brace alternative
exists, discovery can still succeed with an incomplete file set. Please parse
brace boundaries with escape awareness, reusing or extracting the existing
`findClosingBrace` / `splitBraceAlternatives` logic where appropriate, and add
this mixed numeric-range/literal-brace case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
