ismailsimsek commented on code in PR #11906:
URL: https://github.com/apache/iceberg/pull/11906#discussion_r1920398798
##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/DeleteOrphanFilesSparkAction.java:
##########
@@ -292,19 +294,49 @@ private Dataset<FileURI> validFileIdentDS() {
private Dataset<FileURI> actualFileIdentDS() {
StringToFileURI toFileURI = new StringToFileURI(equalSchemes,
equalAuthorities);
+ Dataset<String> dataList;
if (compareToFileList == null) {
- return toFileURI.apply(listedFileDS());
+ dataList =
+ table.io() instanceof SupportsPrefixOperations ? listWithPrefix() :
listWithoutPrefix();
} else {
- return toFileURI.apply(filteredCompareToFileList());
+ dataList = filteredCompareToFileList();
}
+
+ return toFileURI.apply(dataList);
+ }
+
+ @VisibleForTesting
+ Dataset<String> listWithPrefix() {
Review Comment:
@danielcweeks having trouble how to get sub-folders and list them
separately using existing classes.
does it makes sense to add new method to
[SupportsPrefixOperations](https://github.com/apache/iceberg/blob/4d0f40c5b98c16320e2ad795cb5c9a0cd3b1d360/api/src/main/java/org/apache/iceberg/io/SupportsPrefixOperations.java#L36)
interface?
something like below:
`Iterable<FileInfo> listPrefix(String prefix, boolean currentDirectory);`
with cloud libraries `delimiter` (`/`) parameter could be used for this
So the idea is to
1- first make call and get sub-directories (using new method,
`delimiter="/"`).
2- then do full listing per prefix + sub-directory.
- [GCP `delimiter`
example](https://github.com/googleapis/java-storage/blob/e80d2e00df3b310a6ea03c321473fbbd9711686a/samples/snippets/src/main/java/com/example/storage/object/ListObjectsWithPrefix.java#L37-L57)
- [azure lib `delimiter`
parametter](https://azure.github.io/azure-storage-java-async/com/microsoft/azure/storage/blob/ListBlobsOptions.html)
- [aws `delimiter`
parametter](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html#API_ListObjectsV2_RequestSyntax)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org
