Ferix9288 opened a new issue, #11727: URL: https://github.com/apache/pinot/issues/11727
Thread from Pinot slack: https://apache-pinot.slack.com/archives/C011C9JHN7R/p1696234954299969?thread_ts=1695932162.685919&cid=C011C9JHN7R Effectively the scenario is as follows: 1. We start a table with 1 pool / 1 replica group (pool tagged as FirstHalfReplicationGroups) 2. Then we edit the table to 2 pools via the tag `AllReplicationGroups` and bumping the config from 1 to 2 number of replica groups. 3. Finally we also would like to avoid the circumstance in which it shuffles the existing segments in RG0 so we would like to enable `minimizeDataMovement` true. However, we get the following error: ``` Start rebalancing table: e5297e14-b7cc-4aac-a79c-87e0120211c9_OFFLINE with dryRun: true, reassignInstances: true, includeConsuming: false, bootstrap: false, downtime: false, minReplicasToKeepUpForNoDowntime: 1, enableStrictReplicaGroup: false, bestEfforts: false, externalViewCheckIntervalInMs: 1000, externalViewStabilizationTimeoutInMs: 3600000 For rebalanceId: 1a23a96c-51d2-4784-a859-f3107f7a4125, processing instance partitions for table: e5297e14-b7cc-4aac-a79c-87e0120211c9_OFFLINE Reassigning OFFLINE instances for table: e5297e14-b7cc-4aac-a79c-87e0120211c9_OFFLINE Starting e5297e14-b7cc-4aac-a79c-87e0120211c9_OFFLINE instance assignment for table e5297e14-b7cc-4aac-a79c-87e0120211c9_OFFLINE Starting instance tag/pool selection for table: e5297e14-b7cc-4aac-a79c-87e0120211c9_OFFLINE with hash: 957874249 4 enabled instances have the tag: AllReplicationGroups for table: e5297e14-b7cc-4aac-a79c-87e0120211c9_OFFLINE Number instances for each pool: {1=2, 2=2} for table: e5297e14-b7cc-4aac-a79c-87e0120211c9_OFFLINE Selecting all 2 pools: [1, 2] for table: e5297e14-b7cc-4aac-a79c-87e0120211c9_OFFLINE No instance constraint is configured, using default hash-based-rotate instance constraint Rotating instances for table: e5297e14-b7cc-4aac-a79c-87e0120211c9_OFFLINE with hash: 957874249 Starting instance replica-group/partition selection for table: e5297e14-b7cc-4aac-a79c-87e0120211c9_OFFLINE with hash: 957874249 from pools: [1, 2] Selecting 2 replica-groups from pool: {1=[1], 2=[0]} for table: e5297e14-b7cc-4aac-a79c-87e0120211c9_OFFLINE Selecting 2 instances per replica-group for table: e5297e14-b7cc-4aac-a79c-87e0120211c9_OFFLINE Selecting 1 partitions, 2 instances per partition within a replica-group for table: e5297e14-b7cc-4aac-a79c-87e0120211c9_OFFLINE For rebalanceId: 1a23a96c-51d2-4784-a859-f3107f7a4125, caught exception while fetching/calculating instance partitions for table: e5297e14-b7cc-4aac-a79c-87e0120211c9_OFFLINE, aborting the rebalance java.lang.NullPointerException: Cannot invoke "java.util.Collection.size()" because "c" is null at java.util.LinkedHashSet.<init>(LinkedHashSet.java:169) ~[?:?] at org.apache.pinot.controller.helix.core.assignment.instance.InstanceReplicaGroupPartitionSelector.selectInstances(InstanceReplicaGroupPartitionSelector.java:184) ~[pinot-all-1.1.0-SNAPSHOT-jar-with-dependencies.jar:1.1.0-SNAPSHOT-ae1681273c4929587729d71c6dc9526c62ca1fcc] at org.apache.pinot.controller.helix.core.assignment.instance.InstanceAssignmentDriver.getInstancePartitions(InstanceAssignmentDriver.java:98) ~[pinot-all-1.1.0-SNAPSHOT-jar-with-dependencies.jar:1.1.0-SNAPSHOT-ae1681273c4929587729d71c6dc9526c62ca1fcc] at org.apache.pinot.controller.helix.core.assignment.instance.InstanceAssignmentDriver.assignInstances(InstanceAssignmentDriver.java:61) ~[pinot-all-1.1.0-SNAPSHOT-jar-with-dependencies.jar:1.1.0-SNAPSHOT-ae1681273c4929587729d71c6dc9526c62ca1fcc] at org.apache.pinot.controller.helix.core.rebalance.TableRebalancer.getInstancePartitions(TableRebalancer.java:563) ~[pinot-all-1.1.0-SNAPSHOT-jar-with-dependencies.jar:1.1.0-SNAPSHOT-ae1681273c4929587729d71c6dc9526c62ca1fcc] at org.apache.pinot.controller.helix.core.rebalance.TableRebalancer.getInstancePartitionsMap(TableRebalancer.java:497) ~[pinot-all-1.1.0-SNAPSHOT-jar-with-dependencies.jar:1.1.0-SNAPSHOT-ae1681273c4929587729d71c6dc9526c62ca1fcc] at org.apache.pinot.controller.helix.core.rebalance.TableRebalancer.rebalance(TableRebalancer.java:214) ~[pinot-all-1.1.0-SNAPSHOT-jar-with-dependencies.jar:1.1.0-SNAPSHOT-ae1681273c4929587729d71c6dc9526c62ca1fcc] at org.apache.pinot.controller.helix.core.PinotHelixResourceManager.rebalanceTable(PinotHelixResourceManager.java:3115) ~[pinot-all-1.1.0-SNAPSHOT-jar-with-dependencies.jar:1.1.0-SNAPSHOT-ae1681273c4929587729d71c6dc9526c62ca1fcc] at org.apache.pinot.controller.api.resources.PinotTableRestletResource.rebalance(PinotTableRestletResource.java:642) ~[pinot-all-1.1.0-SNAPSHOT-jar-with-dependencies.jar:1.1.0-SNAPSHOT-ae1681273c4929587729d71c6dc9526c62ca1fcc] at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?] at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[?:?] at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?] at java.lang.reflect.Method.invoke(Method.java:568) ~[?:?] at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) ~[pinot-all-1.1.0-SNAPSHOT-jar-with-dependencies.jar:1.1.0-SNAPSHOT-ae1681273c4929587729d71c6dc9526c62ca1fcc] at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:134) [pinot-all-1.1.0-SNAPSHOT-jar-with-dependencies.jar:1.1.0-SNAPSHOT-ae1681273c4929587729d71c6dc9526c62ca1fcc] at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:177) [pinot-all-1.1.0-SNAPSHOT-jar-with-dependencies.jar:1.1.0-SNAPSHOT-ae1681273c4929587729d71c6dc9526c62ca1fcc] at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219) [pinot-all-1.1.0-SNAPSHOT-jar-with-dependencies.jar:1.1.0-SNAPSHOT-ae1681273c4929587729d71c6dc9526c62ca1fcc] at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:81) [pinot-all-1.1.0-SNAPSHOT-jar-with-dependencies.jar:1.1.0-SNAPSHOT-ae1681273c4929587729d71c6dc9526c62ca1fcc] at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478) [pinot-all-1.1.0-SNAPSHOT-jar-with-dependencies.jar:1.1.0-SNAPSHOT-ae1681273c4929587729d71c6dc9526c62ca1fcc] at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400) [pinot-all-1.1.0-SNAPSHOT-jar-with-dependencies.jar:1.1.0-SNAPSHOT-ae1681273c4929587729d71c6dc9526c62ca1fcc] at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81) [pinot-all-1.1.0-SNAPSHOT-jar-with-dependencies.jar:1.1.0-SNAPSHOT-ae1681273c4929587729d71c6dc9526c62ca1fcc] at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:256) [pinot-all-1.1.0-SNAPSHOT-jar-with-dependencies.jar:1.1.0-SNAPSHOT-ae1681273c4929587729d71c6dc9526c62ca1fcc] at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) [pinot-all-1.1.0-SNAPSHOT-jar-with-dependencies.jar:1.1.0-SNAPSHOT-ae1681273c4929587729d71c6dc9526c62ca1fcc] at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) [pinot-all-1.1.0-SNAPSHOT-jar-with-dependencies.jar:1.1.0-SNAPSHOT-ae1681273c4929587729d71c6dc9526c62ca1fcc] at org.glassfish.jersey.internal.Errors.process(Errors.java:292) [pinot-all-1.1.0-SNAPSHOT-jar-with-dependencies.jar:1.1.0-SNAPSHOT-ae1681273c4929587729d71c6dc9526c62ca1fcc] at org.glassfish.jersey.internal.Errors.process(Errors.java:274) [pinot-all-1.1.0-SNAPSHOT-jar-with-dependencies.jar:1.1.0-SNAPSHOT-ae1681273c4929587729d71c6dc9526c62ca1fcc] at org.glassfish.jersey.internal.Errors.process(Errors.java:244) [pinot-all-1.1.0-SNAPSHOT-jar-with-dependencies.jar:1.1.0-SNAPSHOT-ae1681273c4929587729d71c6dc9526c62ca1fcc] at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) [pinot-all-1.1.0-SNAPSHOT-jar-with-dependencies.jar:1.1.0-SNAPSHOT-ae1681273c4929587729d71c6dc9526c62ca1fcc] at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:235) [pinot-all-1.1.0-SNAPSHOT-jar-with-dependencies.jar:1.1.0-SNAPSHOT-ae1681273c4929587729d71c6dc9526c62ca1fcc] at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684) [pinot-all-1.1.0-SNAPSHOT-jar-with-dependencies.jar:1.1.0-SNAPSHOT-ae1681273c4929587729d71c6dc9526c62ca1fcc] at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:356) [pinot-all-1.1.0-SNAPSHOT-jar-with-dependencies.jar:1.1.0-SNAPSHOT-ae1681273c4929587729d71c6dc9526c62ca1fcc] at org.glassfish.grizzly.http.server.HttpHandler$1.run(HttpHandler.java:200) [pinot-all-1.1.0-SNAPSHOT-jar-with-dependencies.jar:1.1.0-SNAPSHOT-ae1681273c4929587729d71c6dc9526c62ca1fcc] at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:569) [pinot-all-1.1.0-SNAPSHOT-jar-with-dependencies.jar:1.1.0-SNAPSHOT-ae1681273c4929587729d71c6dc9526c62ca1fcc] at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:549) [pinot-all-1.1.0-SNAPSHOT-jar-with-dependencies.jar:1.1.0-SNAPSHOT-ae1681273c4929587729d71c6dc9526c62ca1fcc] at java.lang.Thread.run(Thread.java:833) [?:?] ``` I also thought this might be because of a off by one indexing error so I tried changing the pool from (1 -> 0 and 2 -> 1) but no luck either. Also tried changing to `FD_AWARE_INSTANCE_PARTITION_SELECTOR` (but perhaps not using it correctly). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
