9aman commented on code in PR #15142:
URL: https://github.com/apache/pinot/pull/15142#discussion_r2000894701
##########
pinot-spi/src/main/java/org/apache/pinot/spi/config/table/SegmentsValidationAndRetentionConfig.java:
##########
@@ -55,6 +55,8 @@ public class SegmentsValidationAndRetentionConfig extends
BaseJsonConfig {
// For more usage of this field, please refer to this design doc:
https://tinyurl.com/f63ru4sb
private String _peerSegmentDownloadScheme;
+ private int _untrackedSegmentsDeletionBatchSize;
Review Comment:
In case we but the default size as 100 segments, the time taken for the run
for a table with 100 K segments will be:
- Time taken to list all the 100 K segments: ~ 20 seconds (it take 200 ms to
list 1000 segments)
- All the segments will be listed irrespective of the batch size.
- Assuming each path listed if of 100 characters, the size will be 200
bytes. For 100 K it will be ~20 MB
- Time taken to delete 100 segments: ~25 seconds
Total time taken ~ 1 minute for each table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
