vrajat opened a new issue, #14566: URL: https://github.com/apache/pinot/issues/14566
Segments are deleted one by one from deep store during a table delete operation. If there are a large number of segments, the segment deletion phase may take a long time. e.g. it took 54 minutes to delete 60K segments from an AWS S3 deep store. (Logs pasted below). Some storage systems support a BULK DELETE operation such as: * S3 (DELETE API)[https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObjects.html] * GCS (Batch Request)[https://cloud.google.com/storage/docs/batch] These APIs can be used for large tables hosted in these storage systems. ``` pinotController.2024-11-25.1.log.gz:2024/11/25 09:24:46.235 INFO [PinotHelixResourceManager] [grizzly-http-server-3] Deleting table nation_OFFLINE: Start pinotController.2024-11-25.1.log.gz:2024/11/25 09:24:46.237 INFO [PinotHelixResourceManager] [grizzly-http-server-3] Deleting table nation_OFFLINE: Removed from broker resource pinotController.2024-11-25.1.log.gz:2024/11/25 09:24:47.003 INFO [PinotHelixResourceManager] [grizzly-http-server-3] Deleting table nation_OFFLINE: Removed ideal state pinotController.2024-11-25.1.log.gz:2024/11/25 10:18:58.712 INFO [PinotHelixResourceManager] [grizzly-http-server-3] Deleting table nation_OFFLINE: Removed stored segments pinotController.2024-11-25.1.log.gz:2024/11/25 10:20:51.764 INFO [PinotHelixResourceManager] [grizzly-http-server-3] Deleting table nation_OFFLINE: Removed segment metadata pinotController.2024-11-25.1.log.gz:2024/11/25 10:20:51.766 INFO [PinotHelixResourceManager] [grizzly-http-server-3] Deleting table nation_OFFLINE: Removed instance partitions pinotController.2024-11-25.1.log.gz:2024/11/25 10:20:51.769 INFO [PinotHelixResourceManager] [grizzly-http-server-3] Deleting table nation_OFFLINE: Removed tier instance partitions pinotController.2024-11-25.1.log.gz:2024/11/25 10:20:51.771 INFO [PinotHelixResourceManager] [grizzly-http-server-3] Deleting table nation_OFFLINE: Removed segment lineage pinotController.2024-11-25.1.log.gz:2024/11/25 10:20:51.783 INFO [PinotHelixResourceManager] [grizzly-http-server-3] Deleting table nation_OFFLINE: Removed all minion task metadata pinotController.2024-11-25.1.log.gz:2024/11/25 10:20:51.786 INFO [PinotHelixResourceManager] [grizzly-http-server-3] Deleting table nation_OFFLINE: Removed table config pinotController.2024-11-25.1.log.gz:2024/11/25 10:20:51.786 INFO [PinotHelixResourceManager] [grizzly-http-server-3] Deleting table nation_OFFLINE: Finish ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
