Gabriel39 commented on code in PR #68998:
URL: https://github.com/apache/doris/pull/68998#discussion_r4257554103
##########
fe/fe-filesystem/fe-filesystem-hdfs-base/src/main/java/org/apache/doris/filesystem/hdfs/DFSFileSystem.java:
##########
@@ -277,7 +279,7 @@ public List<FileEntry> listFiles(Location location) throws
IOException {
}
private static boolean containsGlob(String path) {
- return path.contains("*") || path.contains("?") || path.contains("[");
+ return path.contains("*") || path.contains("?") || path.contains("[")
|| path.contains("{");
Review Comment:
Fixed in b3039f6f488. The cleanup consumer now enumerates the literal target
using FileSystem.list, filters out directories, closes the iterator, and then
removes the collected files. It retains the staging-directory exclusion. I
audited the other callers before choosing this approach: backup and Parquet
metadata still rely on listFiles glob behavior, so removing that behavior would
introduce a separate regression.
Added an actual HiveConnectorTransaction regression over
DFSFileSystem/Hadoop local storage with relative staging and table directory
names containing {}, *, ? and []. It verifies the old file is removed, the new
file is committed, and a neighboring glob-matching directory remains untouched.
The brace-directory test failed before the fix. All existing filesystem
discovery tests remain passing.
##########
fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java:
##########
@@ -794,7 +799,8 @@ private List<ConnectorPartitionInfo>
collectPartitions(HudiTableHandle handle, b
// HoodieTableMetaClient built just to read a timeline zone -
plus a fallback that
// would have published epoch 0 as a partition's last-modified
time for any
// non-numeric pin, the day it became reachable.
- return buildPartitionInfos(hmsNames, partKeyNames,
latestInstantMillis(handle));
+ return buildPartitionInfos(hmsPartitionPaths(handle, hmsNames),
Review Comment:
Fixed in b3039f6f488. Physical-path parsing now takes an explicit encoding
flag from HoodieTableConfig.getUrlEncodePartitioning(). Missing/false preserves
literal percent sequences; true unescapes once. The flag is propagated through
listing, pruning, pinned-partition filtering, snapshot scans and COW/MOR
incremental scans, so the displayed values and scan values use the same
contract. HMS name decoding for partition-object identity is kept separate.
Added tests using real persisted hoodie.properties for missing, false and
true settings. They distinguish raw ':' and '%3A' in both
listPartitions/listPartitionNames and applyFilter; the default-setting test
reproduced the colon collision before the fix. Additional tests cover
metadata/pinned listings, positional and prefixed paths, percent/slash/plus
values, and MOR native/JNI ranges. The final scoped run passed 906 tests with
no failures, errors or skips; FE Checkstyle passed for all 23 reactor modules.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
