github-actions[bot] commented on code in PR #69385:
URL: https://github.com/apache/doris/pull/69385#discussion_r4374743527
##########
fe/fe-filesystem/fe-filesystem-spi/src/main/java/org/apache/doris/filesystem/spi/S3CompatibleFileSystem.java:
##########
@@ -863,6 +882,57 @@ public static Pattern compileGlobPattern(String
keyPattern) {
return
Pattern.compile(globToRegex(ObjectStorageGlob.expandNumericRanges(keyPattern)));
}
+ private static List<String> expandDeterministicKeyPattern(String pattern,
int maxPaths) {
+ // Keep the original deterministic-path boundary, including escaped
glob literals.
+ if (maxPaths <= 0 || pattern.isEmpty() ||
pattern.endsWith(DIR_MARKER_SUFFIX)
+ || pattern.indexOf('*') >= 0 || pattern.indexOf('?') >= 0 ||
pattern.indexOf('\\') >= 0) {
+ return null;
+ }
+ List<String> expanded = ObjectStorageGlob.expandGlobKeys(pattern,
maxPaths);
+ if (expanded == null || expanded.stream().anyMatch(key ->
key.isEmpty() || key.endsWith(DIR_MARKER_SUFFIX))) {
+ return null;
+ }
+ List<String> keys = new ArrayList<>(new LinkedHashSet<>(expanded));
+ keys.sort(ObjectStorageGlob::compareUtf8Binary);
+ return keys;
+ }
+
+ private RemoteObject headObjectIfPresent(String path) throws IOException {
+ try {
+ return objStorage.headObject(path);
+ } catch (IOException e) {
+ if (isNotFoundError(e)) {
+ return null;
+ }
+ throw e;
+ }
+ }
+
+ private GlobListing globListDeterministicKeys(String bucket, String
patternPrefix, String base,
+ List<String> keys, Pattern matcher) throws IOException {
+ List<FileEntry> files = new ArrayList<>();
+ String lastMatchedKey = "";
+ for (String key : keys) {
+ // Prefix expansion can include keys excluded by character-class
intersections.
+ if (!matcher.matcher(key).matches()) {
+ continue;
+ }
+ RemoteObject obj = headObjectIfPresent(base + key);
Review Comment:
[P2] Preserve object identity for slash-prefixed expansions
For `s3://bucket/{/,a}foo.csv` with both `foo.csv` and `afoo.csv` present,
`/foo.csv` passes the matcher, but `base + key` becomes `s3://bucket//foo.csv`.
`S3ObjStorage.headObject()` reparses that URI through `ObjectStorageUri`, which
strips leading key slashes, so it requests and returns `foo.csv`; this helper
then emits it. HEAD therefore reads/imports a root file outside the glob,
whereas LIST filters that returned key out. This differs from the existing
character-class finding: the candidate matches correctly and changes identity
only during the later URI parse. Exclude candidates that cannot survive this
URI round trip, or preserve the requested key and verify the returned identity
before emitting it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
