swaminathanmanish commented on code in PR #10359: URL: https://github.com/apache/pinot/pull/10359#discussion_r1128673327
########## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java: ########## @@ -2126,13 +2141,15 @@ public boolean addNewReloadAllSegmentsJob(String tableNameWithType, String jobId jobMetadata.put(CommonConstants.ControllerJob.JOB_TYPE, ControllerJobType.RELOAD_ALL_SEGMENTS.toString()); jobMetadata.put(CommonConstants.ControllerJob.SUBMISSION_TIME_MS, Long.toString(System.currentTimeMillis())); jobMetadata.put(CommonConstants.ControllerJob.MESSAGE_COUNT, Integer.toString(numberOfMessagesSent)); - return addControllerJobToZK(jobId, jobMetadata); + return addControllerJobToZK(jobId, jobMetadata, Review Comment: Looked at the code (addControllerJobToZK). We have an upper bound on how many Znodes we can create under a resource directory (MAXIMUM_CONTROLLER_JOBS_IN_ZK = 100). Beyond this, we remove the older jobs based on submission time. ########## pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java: ########## @@ -2126,13 +2141,15 @@ public boolean addNewReloadAllSegmentsJob(String tableNameWithType, String jobId jobMetadata.put(CommonConstants.ControllerJob.JOB_TYPE, ControllerJobType.RELOAD_ALL_SEGMENTS.toString()); jobMetadata.put(CommonConstants.ControllerJob.SUBMISSION_TIME_MS, Long.toString(System.currentTimeMillis())); jobMetadata.put(CommonConstants.ControllerJob.MESSAGE_COUNT, Integer.toString(numberOfMessagesSent)); - return addControllerJobToZK(jobId, jobMetadata); + return addControllerJobToZK(jobId, jobMetadata, Review Comment: We are using PERSISTENT mode for all our Zookeeper updates (not just this one). I dont think this gets up cleaned automatically. Need to check if we have any periodic zookeeper clean up mechanism. https://zookeeper.apache.org/doc/r3.1.2/api/org/apache/zookeeper/CreateMode.html#PERSISTENT -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org