dang-stripe opened a new issue, #8948: URL: https://github.com/apache/pinot/issues/8948
We had a few segments that were not assigned to any servers due to metadata push happening when the controller was shutting down while it was getting replaced. The segments were not added to the ideal state yet. In order to fix the issue, we deleted the segments and re-ran the segment metadata push job. We were unable to reset the segments since they did not exist in ideal state yet Is it possible to avoid this race condition or make the segments recover themselves after startup? ``` [2022-06-18 07:41:14.084079] 2022/06/18 07:41:14.083 INFO [BaseControllerStarter] [Thread-1] Shutting down executor service [2022-06-18 07:41:14.084149] 2022/06/18 07:41:14.084 INFO [BaseControllerStarter] [Thread-1] Disconnecting helix controller zk manager ... [2022-06-18 07:41:14.088673] 2022/06/18 07:41:14.085 ERROR [PinotHelixResourceManager] [jersey-server-managed-async-executor-46] Caught exception while adding segment: table1_65_2022-06-17-18_2022-06-18-05_0 to IdealState for table: table1_OFFLINE, deleting segment ZK metadata [2022-06-18 07:41:14.088707] java.lang.RuntimeException: Caught exception while updating ideal state for resource: table1_OFFLINE [2022-06-18 07:41:14.088739] at org.apache.pinot.common.utils.helix.HelixHelper.updateIdealState(HelixHelper.java:160) ~[pinot-all-0.9.0-2022-06-15-4b2ba75da-SNAPSHOT-jar-with-dependencies.jar:0.9.0-2022-06-15-4b2ba75da-SNAPSHOT-4b2ba75da26eb821937444b097465d63686d5cc3] [2022-06-18 07:41:14.088775] at org.apache.pinot.common.utils.helix.HelixHelper.updateIdealState(HelixHelper.java:180) ~[pinot-all-0.9.0-2022-06-15-4b2ba75da-SNAPSHOT-jar-with-dependencies.jar:0.9.0-2022-06-15-4b2ba75da-SNAPSHOT-4b2ba75da26eb821937444b097465d63686d5cc3] [2022-06-18 07:41:14.088807] at org.apache.pinot.controller.helix.core.PinotHelixResourceManager.assignTableSegment(PinotHelixResourceManager.java:1814) ~[pinot-all-0.9.0-2022-06-15-4b2ba75da-SNAPSHOT-jar-with-dependencies.jar:0.9.0-2022-06-15-4b2ba75da-SNAPSHOT-4b2ba75da26eb821937444b097465d63686d5cc3] [2022-06-18 07:41:14.088861] at org.apache.pinot.controller.helix.core.PinotHelixResourceManager.addNewSegment(PinotHelixResourceManager.java:1800) ~[pinot-all-0.9.0-2022-06-15-4b2ba75da-SNAPSHOT-jar-with-dependencies.jar:0.9.0-2022-06-15-4b2ba75da-SNAPSHOT-4b2ba75da26eb821937444b097465d63686d5cc3] [2022-06-18 07:41:14.088895] at org.apache.pinot.controller.api.upload.ZKOperator.processNewSegment(ZKOperator.java:243) ~[pinot-all-0.9.0-2022-06-15-4b2ba75da-SNAPSHOT-jar-with-dependencies.jar:0.9.0-2022-06-15-4b2ba75da-SNAPSHOT-4b2ba75da26eb821937444b097465d63686d5cc3] [2022-06-18 07:41:14.088926] at org.apache.pinot.controller.api.upload.ZKOperator.completeSegmentOperations(ZKOperator.java:77) ~[pinot-all-0.9.0-2022-06-15-4b2ba75da-SNAPSHOT-jar-with-dependencies.jar:0.9.0-2022-06-15-4b2ba75da-SNAPSHOT-4b2ba75da26eb821937444b097465d63686d5cc3] [2022-06-18 07:41:14.088979] at org.apache.pinot.controller.api.resources.PinotSegmentUploadDownloadRestletResource.completeZkOperations(PinotSegmentUploadDownloadRestletResource.java:401) ~[pinot-all-0.9.0-2022-06-15-4b2ba75da-SNAPSHOT-jar-with-dependencies.jar:0.9.0-2022-06-15-4b2ba75da-SNAPSHOT-4b2ba75da26eb821937444b097465d63686d5cc3] [2022-06-18 07:41:14.089031] at org.apache.pinot.controller.api.resources.PinotSegmentUploadDownloadRestletResource.uploadSegment(PinotSegmentUploadDownloadRestletResource.java:303) ~[pinot-all-0.9.0-2022-06-15-4b2ba75da-SNAPSHOT-jar-with-dependencies.jar:0.9.0-2022-06-15-4b2ba75da-SNAPSHOT-4b2ba75da26eb821937444b097465d63686d5cc3] [2022-06-18 07:41:14.089081] at org.apache.pinot.controller.api.resources.PinotSegmentUploadDownloadRestletResource.uploadSegmentAsMultiPartV2(PinotSegmentUploadDownloadRestletResource.java:533) ~[pinot-all-0.9.0-2022-06-15-4b2ba75da-SNAPSHOT-jar-with-dependencies.jar:0.9.0-2022-06-15-4b2ba75da-SNAPSHOT-4b2ba75da26eb821937444b097465d63686d5cc3] [2022-06-18 07:41:14.089104] at jdk.internal.reflect.GeneratedMethodAccessor202.invoke(Unknown Source) ~[?:?] [2022-06-18 07:41:14.089129] at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] [2022-06-18 07:41:14.089148] at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?] [2022-06-18 07:41:14.089172] at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) ~[pinot-all-0.9.0-2022-06-15-4b2ba75da-SNAPSHOT-jar-with-dependencies.jar:0.9.0-2022-06-15-4b2ba75da-SNAPSHOT-4b2ba75da26eb821937444b097465d63686d5cc3] [2022-06-18 07:41:14.089221] at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124) ~[pinot-all-0.9.0-2022-06-15-4b2ba75da-SNAPSHOT-jar-with-dependencies.jar:0.9.0-2022-06-15-4b2ba75da-SNAPSHOT-4b2ba75da26eb821937444b097465d63686d5cc3] [2022-06-18 07:41:14.089272] at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167) ~[pinot-all-0.9.0-2022-06-15-4b2ba75da-SNAPSHOT-jar-with-dependencies.jar:0.9.0-2022-06-15-4b2ba75da-SNAPSHOT-4b2ba75da26eb821937444b097465d63686d5cc3] [2022-06-18 07:41:14.089328] at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$VoidOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:159) ~[pinot-all-0.9.0-2022-06-15-4b2ba75da-SNAPSHOT-jar-with-dependencies.jar:0.9.0-2022-06-15-4b2ba75da-SNAPSHOT-4b2ba75da26eb821937444b097465d63686d5cc3] [2022-06-18 07:41:14.089376] at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79) ~[pinot-all-0.9.0-2022-06-15-4b2ba75da-SNAPSHOT-jar-with-dependencies.jar:0.9.0-2022-06-15-4b2ba75da-SNAPSHOT-4b2ba75da26eb821937444b097465d63686d5cc3] [2022-06-18 07:41:14.089425] at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469) ~[pinot-all-0.9.0-2022-06-15-4b2ba75da-SNAPSHOT-jar-with-dependencies.jar:0.9.0-2022-06-15-4b2ba75da-SNAPSHOT-4b2ba75da26eb821937444b097465d63686d5cc3] [2022-06-18 07:41:14.089472] at org.glassfish.jersey.server.model.ResourceMethodInvoker.lambda$apply$0(ResourceMethodInvoker.java:381) ~[pinot-all-0.9.0-2022-06-15-4b2ba75da-SNAPSHOT-jar-with-dependencies.jar:0.9.0-2022-06-15-4b2ba75da-SNAPSHOT-4b2ba75da26eb821937444b097465d63686d5cc3] -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
