pvary commented on code in PR #12254:
URL: https://github.com/apache/iceberg/pull/12254#discussion_r2024776922
##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/DeleteOrphanFilesSparkAction.java:
##########
@@ -302,21 +303,29 @@ private Dataset<FileURI> actualFileIdentDS() {
private Dataset<String> listedFileDS() {
List<String> subDirs = Lists.newArrayList();
List<String> matchingFiles = Lists.newArrayList();
+ boolean isSupportPrefixOperations = table.io() instanceof
SupportsPrefixOperations;
- Predicate<FileStatus> predicate = file -> file.getModificationTime() <
olderThanTimestamp;
PathFilter pathFilter =
PartitionAwareHiddenPathFilter.forSpecs(table.specs());
- // list at most MAX_DRIVER_LISTING_DEPTH levels and only dirs that have
- // less than MAX_DRIVER_LISTING_DIRECT_SUB_DIRS direct sub dirs on the
driver
- listDirRecursively(
- location,
- predicate,
- hadoopConf.value(),
- MAX_DRIVER_LISTING_DEPTH,
- MAX_DRIVER_LISTING_DIRECT_SUB_DIRS,
- subDirs,
- pathFilter,
- matchingFiles);
+ if (isSupportPrefixOperations) {
+ Predicate<org.apache.iceberg.io.FileInfo> predicate =
+ fileInfo -> fileInfo.createdAtMillis() < olderThanTimestamp;
+ listDirRecursivelyWithFileIO(
+ (SupportsPrefixOperations) table.io(), location, predicate,
pathFilter, matchingFiles);
+ } else {
+ Predicate<FileStatus> predicate = file -> file.getModificationTime() <
olderThanTimestamp;
+ // list at most MAX_DRIVER_LISTING_DEPTH levels and only dirs that have
+ // less than MAX_DRIVER_LISTING_DIRECT_SUB_DIRS direct sub dirs on the
driver
+ listDirRecursivelyWithHadoop(
Review Comment:
So basically if the FileIO doesn't implement the `SupportsPrefixOperations`
(like OSS, ECS), then we will still try to list the files through Hadoop
libraries, but not through the HadoopFileIO.
It is good to have this for backward compatibility, but if we don't have
tests for this codepath, then we might cause unintended issues later. So please
create tests which exercise this codepath
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org
