aarushigupta132 opened a new pull request, #17796: URL: https://github.com/apache/iceberg/pull/17796
## Summary This draft adds a `parallel_deletes` option to the Spark 4.1 `remove_orphan_files` procedure. When enabled, orphan files are deleted from Spark executors rather than sequentially on the driver. The option is disabled by default, preserving the existing driver-side deletion behavior. `FileIO` implementations that support bulk operations delete one Spark partition at a time; other implementations delete files individually within each partition. Orphan paths continue to flow back to the driver so the procedure can produce its existing result count and path sample. ## Motivation `remove_orphan_files` currently identifies orphan files with Spark but performs deletion on the driver. Even when listing and the metadata anti-join are distributed, deletion throughput is therefore limited by one node. Executor-side deletion allows the work to scale with the number of Spark partitions. This is complementary to #17790, which distributes prefix-based file listing. The cached orphan-file dataset remains the correct input to deletion because files must not be removed until the metadata anti-join and prefix-mismatch validation have completed. ## Open questions and follow-ups This is a draft because the API and failure semantics need review. In particular: * `dry_run => true` currently supplies a no-op delete function, which the parallel path bypasses. The two options must be rejected together or made safe before this is ready to merge. * The current implementation materializes a complete Spark partition before invoking bulk deletion. It should retain bounded batches comparable to the existing `DELETE_GROUP_SIZE`. * The interaction with `max_concurrent_deletes` and custom delete functions should be validated explicitly. * Executor-side retries and speculative execution can cause a partition to be deleted more than once. The expected idempotency and failure-reporting behavior should be documented and tested. * Driver-side logs become executor-side logs for the actual delete operations when this option is enabled. ## Testing Tests are still to be added for executor-side bulk and non-bulk deletion, dry-run behavior, option interactions, partition batching, and task retry behavior. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
