924060929 commented on code in PR #68998:
URL: https://github.com/apache/doris/pull/68998#discussion_r4278181403
##########
fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorTransaction.java:
##########
@@ -1142,7 +1143,17 @@ private void deleteTargetPathContents(String targetPath,
String excludedChildPat
}
deleteDir(dir);
}
- List<FileEntry> files =
getFileSystem().listFiles(Location.of(targetPath));
+ // Storage implementations may treat listFiles arguments as globs.
Overwrite must enumerate
+ // the literal target, or a table name such as table_{prod} can
leave old rows behind.
+ List<FileEntry> files = new ArrayList<>();
+ try (FileIterator iterator =
getFileSystem().list(Location.of(targetPath))) {
+ while (iterator.hasNext()) {
+ FileEntry file = iterator.next();
+ if (!file.isDirectory()) {
Review Comment:
[P1] Independently confirmed by tracing the code at
`b3039f6f48862c5ba02ef6ab5bb486daefb9c802`. This does not require an upgrade or
metadata from an older master build: a fresh deployment with a newly created
unpartitioned Hive table at the OneLake path above and relative
`hive.staging_dir=.doris_staging` can reach the same path.
`prepareAlterTable` calls this cleanup before `asyncRenameFiles`. Unlike the
immediate-child DFS listing covered by the new test, `AzureFileSystem.list()`
returns descendant blobs, including `.doris_staging/.../new.parquet`, with
`isDirectory=false`. The directory-loop exclusion therefore does not protect
the staged source; the file loop deletes both it and existing table files. The
subsequent move fails, and this relative-staging branch has no old-data
restoration task.
Please exclude the staging root and all its descendants from the file
deletion set, using path-boundary-aware comparisons such as the existing
`pathsEqual` / `isSubDirectory` helpers. Add a transaction regression using the
real `AzureFileSystem` with a recording object store, verifying that staged
inputs survive cleanup and the overwrite commits successfully. This conclusion
is source-based; I have not run a live OneLake test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
