somandal commented on issue #15683: URL: https://github.com/apache/pinot/issues/15683#issuecomment-2940967343
> > Today we don't clean up ZK job status ZNodes. Thus they can grow indefinitely and we may start hitting the ZNode size limitations. We should periodically clean up older job statuses > > This isn't done periodically today, but there's a hard limit of 100 jobs of each type beyond which older ones will be cleaned up - > > [pinot/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java](https://github.com/apache/pinot/blob/c15440466a0032c5f74e55940792fb16cd719760/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java#L2552-L2558) > > Lines 2552 to 2558 in [c154404](/apache/pinot/commit/c15440466a0032c5f74e55940792fb16cd719760) > > if (jobMetadataMap.size() > CommonConstants.ControllerJob.MAXIMUM_CONTROLLER_JOBS_IN_ZK) { > jobMetadataMap = jobMetadataMap.entrySet().stream().sorted((v1, v2) -> Long.compare( > Long.parseLong(v2.getValue().get(CommonConstants.ControllerJob.SUBMISSION_TIME_MS)), > Long.parseLong(v1.getValue().get(CommonConstants.ControllerJob.SUBMISSION_TIME_MS)))) > .collect(Collectors.toList()).subList(0, CommonConstants.ControllerJob.MAXIMUM_CONTROLLER_JOBS_IN_ZK) > .stream().collect(Collectors.toMap(Map.Entry::getKey, Map.Entry::getValue)); > } > This isn't ideal, though, and we could maybe make this configurable at least. Yes, we should make it configurable for sure. If we do progress tracking for `SegmentRelocator` this can actually result in weird issues where the 100 limit may not be enough - if `SegmentRelocator` runs every hour and handles a lot of tables. We should also ideally not remove entries that are still `IN_PROGRESS` for rebalance - as this can completely mess up our objective of not allowing multiple rebalance jobs if these entries are cleaned up. This code is job type agnostic though, so it doesn't look into the specifics of each job type metadata. Perhaps we should split this out to allow special handling for each job type or something. Can you look into this improvement as a follow-up @yashmayya? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
