amogh-jahagirdar commented on code in PR #10229:
URL: https://github.com/apache/iceberg/pull/10229#discussion_r1581079152
##########
spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/DeleteOrphanFilesSparkAction.java:
##########
@@ -398,12 +399,14 @@ static List<String> findOrphanFiles(
spark.sparkContext().register(conflicts);
Column joinCond =
actualFileIdentDS.col("path").equalTo(validFileIdentDS.col("path"));
-
- List<String> orphanFiles =
- actualFileIdentDS
- .joinWith(validFileIdentDS, joinCond, "leftouter")
- .mapPartitions(new FindOrphanFiles(prefixMismatchMode, conflicts),
Encoders.STRING())
- .collectAsList();
+ PatriciaTrie<String> files = new PatriciaTrie<>();
+ actualFileIdentDS
+ .joinWith(validFileIdentDS, joinCond, "leftouter")
+ .mapPartitions(new FindOrphanFiles(prefixMismatchMode, conflicts),
Encoders.STRING())
+ .foreach(
+ path -> {
+ files.put(path, null);
Review Comment:
Foreach on this local reference won't work, in the end we have to collect it
at which point all the string paths are materialized in memory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org
