Akshat-Jain opened a new pull request, #20425: URL: https://github.com/apache/druid/pull/20425
### Description `DurableStorageCleaner.run()` collects every stray file path into a single in-memory HashSet and only calls `StorageConnector.deleteFiles` once the entire listing has completed. Memory is therefore `O(number of stray files in durable storage)`, with no bound. This caused issues on a deployment where we seem to have a lot of those files leaked in S3, causing the overlord to run into OOM. This PR improves the logic to flush the buffer to `deleteFiles` every 100k paths. 100k is a reasonably-sized number which should be higher than what every cloud provider limits the batch size to, while still being decent-sized to do well in the case of large number of files. <hr> This PR has: - [x] been self-reviewed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
