dang-stripe opened a new issue, #10956: URL: https://github.com/apache/pinot/issues/10956
We've noticed segments in S3 are not getting deleted from the Deleted_Segments directory. This seems to be because when SegmentDeletionManager calls S3PinotFS.listFiles, it only returns files and not directories and always returns an empty list, causing the `removeAgedDeletedSegments` call to terminate. The Deleted_Segments directory only contains a directory for each table w/ deleted segments. We found this log line that confirms this which reports 0 files being returned. ``` [2023-06-21 17:02:32.807244] INFO [S3PinotFS] [pool-17-thread-1:169] Listed 0 files from URI: s3://some-bucket/pinot/pinot-cluster-1/prod-1/Deleted_Segments, is recursive: false ``` https://github.com/apache/pinot/blob/master/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/SegmentDeletionManager.java#L290 https://github.com/apache/pinot/blob/master/pinot-plugins/pinot-file-system/pinot-s3/src/main/java/org/apache/pinot/plugin/filesystem/S3PinotFS.java#L486-L499 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
