github-actions[bot] commented on code in PR #68998:
URL: https://github.com/apache/doris/pull/68998#discussion_r4256817233
##########
fe/fe-filesystem/fe-filesystem-hdfs-base/src/main/java/org/apache/doris/filesystem/hdfs/DFSFileSystem.java:
##########
@@ -277,7 +279,7 @@ public List<FileEntry> listFiles(Location location) throws
IOException {
}
private static boolean containsGlob(String path) {
- return path.contains("*") || path.contains("?") || path.contains("[");
+ return path.contains("*") || path.contains("?") || path.contains("[")
|| path.contains("{");
Review Comment:
[P1] Preserve literal directory listings when enabling brace globs
For an unpartitioned Hive table at `hdfs://nn/warehouse/table_{prod}` with
`hive.staging_dir=.doris_staging`, `INSERT OVERWRITE` calls
`HiveConnectorTransaction.deleteTargetPathContents`, which uses
`listFiles(targetPath)` to remove old files before moving staged files into the
table. This new `{` check sends that literal directory through `globStatus`:
[Hadoop interprets `{prod}` as a
group](https://github.com/apache/hadoop/blob/rel/release-3.4.2/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/GlobPattern.java#L102-L114),
matching `table_prod`, and [returns an empty array when it finds no
match](https://github.com/apache/hadoop/blob/rel/release-3.4.2/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Globber.java#L355-L377).
Cleanup therefore succeeds without deleting old files, and the overwrite
leaves old and new rows together. The previous direct listing read the literal
directory correctly. Preserve literal-directory behavior for the
se callers while retaining brace-glob discovery, and cover a relative-staging
overwrite into a directory containing braces.
##########
fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java:
##########
@@ -794,7 +799,8 @@ private List<ConnectorPartitionInfo>
collectPartitions(HudiTableHandle handle, b
// HoodieTableMetaClient built just to read a timeline zone -
plus a fallback that
// would have published epoch 0 as a partition's last-modified
time for any
// non-numeric pin, the day it became reachable.
- return buildPartitionInfos(hmsNames, partKeyNames,
latestInstantMillis(handle));
+ return buildPartitionInfos(hmsPartitionPaths(handle, hmsNames),
Review Comment:
[P2] Preserve literal percent values in unencoded Hudi partitions
With `use_hive_sync_partition=true`, [Hudi's default
`hoodie.datasource.write.partitionpath.urlencode=false`](https://github.com/apache/hudi/blob/release-1.0.2/hudi-common/src/main/java/org/apache/hudi/keygen/constant/KeyGeneratorOptions.java#L33-L37)
permits the literal STRING value `%3A` at `s3://b/t/dt=%3A`. [The Hive-style
sync extractor retains
`%3A`](https://github.com/apache/hudi/blob/release-1.0.2/hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/HiveStylePartitionValueExtractor.java#L32-L39),
so HMS exposes `dt=%253A`. Previously `buildPartitionInfos` decoded that HMS
name once and published `%3A`; the new location substitution supplies `dt=%3A`,
which `parsePartitionValues` decodes into `:`. Consequently
`partition_values()` now reports the wrong logical value and `SHOW PARTITIONS`
changes the name; separate raw `:` and `%3A` partitions can also collapse in
name-keyed consumers. Preserve the table's encoding contract when deriving
logical values from physical p
aths. Add a raw `%3A` location case alongside the existing encoded `%253A`
case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
