ismailsimsek commented on code in PR #11906:
URL: https://github.com/apache/iceberg/pull/11906#discussion_r1903140118
##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/DeleteOrphanFilesSparkAction.java:
##########
@@ -292,14 +293,37 @@ private Dataset<FileURI> validFileIdentDS() {
private Dataset<FileURI> actualFileIdentDS() {
StringToFileURI toFileURI = new StringToFileURI(equalSchemes,
equalAuthorities);
+ Dataset<String> dataList;
if (compareToFileList == null) {
- return toFileURI.apply(listedFileDS());
+ dataList =
+ table.io() instanceof SupportsPrefixOperations ? listWithPrefix() :
listWithoutPrefix();
} else {
- return toFileURI.apply(filteredCompareToFileList());
+ dataList = filteredCompareToFileList();
}
+
+ return toFileURI.apply(dataList);
+ }
+
+ @VisibleForTesting
+ Dataset<String> listWithPrefix() {
+ List<String> matchingFiles = Lists.newArrayList();
+ PathFilter pathFilter =
PartitionAwareHiddenPathFilter.forSpecs(table.specs());
+
+ Iterator<org.apache.iceberg.io.FileInfo> iterator =
+ ((SupportsPrefixOperations)
table.io()).listPrefix(location).iterator();
+ while (iterator.hasNext()) {
+ org.apache.iceberg.io.FileInfo fileInfo = iterator.next();
+ if (fileInfo.createdAtMillis() < olderThanTimestamp
+ && pathFilter.accept(new Path(fileInfo.location()))) {
+ matchingFiles.add(fileInfo.location());
+ }
+ }
+ JavaRDD<String> matchingFileRDD =
sparkContext().parallelize(matchingFiles, 1);
+ return spark().createDataset(matchingFileRDD.rdd(), Encoders.STRING());
}
- private Dataset<String> listedFileDS() {
+ @VisibleForTesting
+ Dataset<String> listWithoutPrefix() {
Review Comment:
had to make it package private in-orderto test using `VisibleForTesting`.
not sure how else to call `listWithoutPrefix()` method.
By default: Current executions are using `HadoopFileIO` and its implementing
`DelegateFileIO` -> `SupportPrefixOperations` which calls new method
`listWithPrefix`
[mentioned in this comment
too](https://github.com/apache/iceberg/pull/7914#issuecomment-2212775830)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org
