Hi Shawn,
Thanks for replying.
Solr version: 6.1.0
Zookeeper: 3.3.6
Solr log errors below are from around the 21:33:39 timestamp:
2016-08-21 21:33:37.135 WARN (main-SendThread(172.28.128.3:2181)) [ ]
o.a.z.ClientCnxn Session 0x156aefeba2d0004 for server
172.28.128.3/172.28.128.3:2181, unexpected error, closing socket connection and
attempting reconnect
java.io.IOException: Xid out of order. Got Xid 1438 with err 0 expected Xid
1437 for a packet with details: clientPath:null serverPath:null finished:false
header:: 1437,14 replyHeader:: 0,0,-4 request::
org.apache.zookeeper.MultiTransactionRecord@95ad06de response::
org.apache.zookeeper.MultiResponse@0
at
org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:798)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:94)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
2016-08-21 21:33:37.144 ERROR
(recoveryExecutor-3-thread-1-processing-n:10.0.2.15:8983_solr
x:mycollection_shard1_replica1 s:shard1 c:mycollection r:core_node1)
[c:mycollection s:shard1 r:core_node1 x:mycollection_shard1_replica1]
o.a.s.c.RecoveryStrategy Error while trying to recover.
core=mycollection_shard1_replica1:org.apache.solr.common.SolrException: No
registered leader was found after waiting for 4000ms , collection: mycollection
slice: shard1
at
org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:718)
at
org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:704)
at
org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:305)
at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:221)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$22(ExecutorUtil.java:229)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2016-08-21 21:33:37.144 INFO
(recoveryExecutor-3-thread-1-processing-n:10.0.2.15:8983_solr
x:mycollection_shard1_replica1 s:shard1 c:mycollection r:core_node1)
[c:mycollection s:shard1 r:core_node1 x:mycollection_shard1_replica1]
o.a.s.c.RecoveryStrategy Replay not started, or was not successful... still
buffering updates.
2016-08-21 21:33:37.144 INFO
(recoveryExecutor-3-thread-1-processing-n:10.0.2.15:8983_solr
x:mycollection_shard1_replica1 s:shard1 c:mycollection r:core_node1)
[c:mycollection s:shard1 r:core_node1 x:mycollection_shard1_replica1]
o.a.s.c.RecoveryStrategy RecoveryStrategy has been closed
2016-08-21 21:33:37.149 INFO
(recoveryExecutor-3-thread-1-processing-n:10.0.2.15:8983_solr
x:mycollection_shard1_replica1 s:shard1 c:mycollection r:core_node1)
[c:mycollection s:shard1 r:core_node1 x:mycollection_shard1_replica1]
o.a.s.c.RecoveryStrategy Finished recovery process, successful=[false]
2016-08-21 21:33:37.237 INFO
(zkCallback-4-thread-168-processing-n:10.0.2.15:8983_solr) [ ]
o.a.s.c.c.ConnectionManager Watcher
org.apache.solr.common.cloud.ConnectionManager@7d41da47
name:ZooKeeperConnection Watcher:172.28.128.3:2181 got event WatchedEvent
state:Disconnected type:None path:null path:null type:None
2016-08-21 21:33:37.237 INFO
(zkCallback-4-thread-168-processing-n:10.0.2.15:8983_solr) [ ]
o.a.s.c.c.ConnectionManager zkClient has disconnected
2016-08-21 21:33:39.062 WARN (main-SendThread(172.28.128.3:2181)) [ ]
o.a.z.ClientCnxnSocket Connected to an old server; r-o mode will be unavailable
2016-08-21 21:33:39.063 INFO
(zkCallback-4-thread-171-processing-n:10.0.2.15:8983_solr) [ ]
o.a.s.c.c.ConnectionManager Watcher
org.apache.solr.common.cloud.ConnectionManager@7d41da47
name:ZooKeeperConnection Watcher:172.28.128.3:2181 got event WatchedEvent
state:SyncConnected type:None path:null path:null type:None
2016-08-21 21:33:39.071 WARN (main-SendThread(172.28.128.3:2181)) [ ]
o.a.z.ClientCnxn Session 0x156aefeba2d0004 for server
172.28.128.3/172.28.128.3:2181, unexpected error, closing socket connection and
attempting reconnect
java.io.IOException: Xid out of order. Got Xid 1441 with err 0 expected Xid
1440 for a packet with details: clientPath:null serverPath:null finished:false
header:: 1440,14 replyHeader:: 0,0,-4 request::
org.apache.zookeeper.MultiTransactionRecord@95ad06de response::
org.apache.zookeeper.MultiResponse@0
at
org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:798)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:94)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
2016-08-21 21:33:39.174 INFO
(zkCallback-4-thread-168-processing-n:10.0.2.15:8983_solr) [ ]
o.a.s.c.c.ConnectionManager Watcher
org.apache.solr.common.cloud.ConnectionManager@7d41da47
name:ZooKeeperConnection Watcher:172.28.128.3:2181 got event WatchedEvent
state:Disconnected type:None path:null path:null type:None
2016-08-21 21:33:39.175 INFO
(zkCallback-4-thread-168-processing-n:10.0.2.15:8983_solr) [ ]
o.a.s.c.c.ConnectionManager zkClient has disconnected
2016-08-21 21:33:40.023 INFO (qtp110456297-14) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/info/logging
params={wt=json&_=1471815019974&since=0} status=0 QTime=0
2016-08-21 21:33:40.622 WARN (main-SendThread(172.28.128.3:2181)) [ ]
o.a.z.ClientCnxnSocket Connected to an old server; r-o mode will be unavailable
2016-08-21 21:33:40.623 INFO
(zkCallback-4-thread-171-processing-n:10.0.2.15:8983_solr) [ ]
o.a.s.c.c.ConnectionManager Watcher
org.apache.solr.common.cloud.ConnectionManager@7d41da47
name:ZooKeeperConnection Watcher:172.28.128.3:2181 got event WatchedEvent
state:SyncConnected type:None path:null path:null type:None
2016-08-21 21:33:42.177 WARN (main-SendThread(172.28.128.3:2181)) [ ]
o.a.z.ClientCnxn Session 0x156aefeba2d0004 for server
172.28.128.3/172.28.128.3:2181, unexpected error, closing socket connection and
attempting reconnect
java.io.IOException: Xid out of order. Got Xid 1444 with err 0 expected Xid
1443 for a packet with details: clientPath:null serverPath:null finished:false
header:: 1443,14 replyHeader:: 0,0,-4 request::
org.apache.zookeeper.MultiTransactionRecord@95ad06de response::
org.apache.zookeeper.MultiResponse@0
at
org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:798)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:94)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
2016-08-21 21:33:42.280 INFO
(zkCallback-4-thread-168-processing-n:10.0.2.15:8983_solr) [ ]
o.a.s.c.c.ConnectionManager Watcher
org.apache.solr.common.cloud.ConnectionManager@7d41da47
name:ZooKeeperConnection Watcher:172.28.128.3:2181 got event WatchedEvent
state:Disconnected type:None path:null path:null type:None
2016-08-21 21:33:42.280 INFO
(zkCallback-4-thread-168-processing-n:10.0.2.15:8983_solr) [ ]
o.a.s.c.c.ConnectionManager zkClient has disconnected
2016-08-21 21:33:43.911 WARN (main-SendThread(172.28.128.3:2181)) [ ]
o.a.z.ClientCnxnSocket Connected to an old server; r-o mode will be unavailable
2016-08-21 21:33:43.912 INFO
(zkCallback-4-thread-171-processing-n:10.0.2.15:8983_solr) [ ]
o.a.s.c.c.ConnectionManager Watcher
org.apache.solr.common.cloud.ConnectionManager@7d41da47
name:ZooKeeperConnection Watcher:172.28.128.3:2181 got event WatchedEvent
state:SyncConnected type:None path:null path:null type:None
2016-08-21 21:33:46.782 WARN (main-SendThread(172.28.128.3:2181)) [ ]
o.a.z.ClientCnxn Session 0x156aefeba2d0004 for server
172.28.128.3/172.28.128.3:2181, unexpected error, closing socket connection and
attempting reconnect
java.io.IOException: Xid out of order. Got Xid 1447 with err 0 expected Xid
1446 for a packet with details: clientPath:null serverPath:null finished:false
header:: 1446,14 replyHeader:: 0,0,-4 request::
org.apache.zookeeper.MultiTransactionRecord@95ad06de response::
org.apache.zookeeper.MultiResponse@0
at
org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:798)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:94)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
2016-08-21 21:33:46.885 INFO
(zkCallback-4-thread-168-processing-n:10.0.2.15:8983_solr) [ ]
o.a.s.c.c.ConnectionManager Watcher
org.apache.solr.common.cloud.ConnectionManager@7d41da47
name:ZooKeeperConnection Watcher:172.28.128.3:2181 got event WatchedEvent
state:Disconnected type:None path:null path:null type:None
2016-08-21 21:33:46.885 INFO
(zkCallback-4-thread-168-processing-n:10.0.2.15:8983_solr) [ ]
o.a.s.c.c.ConnectionManager zkClient has disconnected
2016-08-21 21:33:48.760 WARN (main-SendThread(172.28.128.3:2181)) [ ]
o.a.z.ClientCnxnSocket Connected to an old server; r-o mode will be unavailable
2016-08-21 21:33:48.761 INFO
(zkCallback-4-thread-171-processing-n:10.0.2.15:8983_solr) [ ]
o.a.s.c.c.ConnectionManager Watcher
org.apache.solr.common.cloud.ConnectionManager@7d41da47
name:ZooKeeperConnection Watcher:172.28.128.3:2181 got event WatchedEvent
state:SyncConnected type:None path:null path:null type:None
2016-08-21 21:33:50.027 INFO (qtp110456297-16) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/info/logging
params={wt=json&_=1471815019974&since=0} status=0 QTime=0
2016-08-21 21:33:52.887 WARN (main-SendThread(172.28.128.3:2181)) [ ]
o.a.z.ClientCnxn Session 0x156aefeba2d0004 for server
172.28.128.3/172.28.128.3:2181, unexpected error, closing socket connection and
attempting reconnect
java.io.IOException: Xid out of order. Got Xid 1450 with err 0 expected Xid
1449 for a packet with details: clientPath:null serverPath:null finished:false
header:: 1449,14 replyHeader:: 0,0,-4 request::
org.apache.zookeeper.MultiTransactionRecord@95ad06de response::
org.apache.zookeeper.MultiResponse@0
at
org.apache.zookeeper.ClientCnxn$SendThread.readResponse(ClientCnxn.java:798)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:94)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1081)
2016-08-21 21:33:52.990 INFO
(zkCallback-4-thread-168-processing-n:10.0.2.15:8983_solr) [ ]
o.a.s.c.c.ConnectionManager Watcher
org.apache.solr.common.cloud.ConnectionManager@7d41da47
name:ZooKeeperConnection Watcher:172.28.128.3:2181 got event WatchedEvent
state:Disconnected type:None path:null path:null type:None
2016-08-21 21:33:52.990 INFO
(zkCallback-4-thread-168-processing-n:10.0.2.15:8983_solr) [ ]
o.a.s.c.c.ConnectionManager zkClient has disconnected
2016-08-21 21:33:54.718 WARN (main-SendThread(172.28.128.3:2181)) [ ]
o.a.z.ClientCnxnSocket Connected to an old server; r-o mode will be unavailable
2016-08-21 21:33:54.719 INFO
(zkCallback-4-thread-171-processing-n:10.0.2.15:8983_solr) [ ]
o.a.s.c.c.ConnectionManager Watcher
org.apache.solr.common.cloud.ConnectionManager@7d41da47
name:ZooKeeperConnection Watcher:172.28.128.3:2181 got event WatchedEvent
state:SyncConnected type:None path:null path:null type:None
2016-08-21 21:34:00.028 INFO (qtp110456297-220) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/info/logging
params={wt=json&_=1471815019974&since=0} status=0 QTime=0
2016-08-21 21:34:00.493 ERROR (qtp110456297-11) [c:mycollection s:shard1
r:core_node1 x:mycollection_shard1_replica1] o.a.s.c.ShardLeaderElectionContext
There was a problem trying to register as the
leader:org.apache.solr.common.SolrException: Could not register as the leader
because creating the ephemeral registration node in ZooKeeper failed
at
org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:218)
at
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:417)
at
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)
at
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)
at
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)
at
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:216)
at
org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:670)
at
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:441)
at
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)
at
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)
at
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)
at
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:216)
at
org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:670)
at
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:441)
at
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)
at
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)
at
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)
at
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:216)
at
org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:670)
at
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:441)
at
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)
at
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)
at
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)
at
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:216)
at
org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:670)
at
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:441)
at
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)
at
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)
at
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)
at
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:216)
at
org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:670)
at
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:441)
at
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)
at
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)
at
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)
at
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:216)
at
org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:670)
at
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:441)
at
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)
at
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)
at
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)
at
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:216)
at
org.apache.solr.cloud.ShardLeaderElectionContext.rejoinLeaderElection(ElectionContext.java:670)
at
org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:441)
at
org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)
at
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)
at
org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)
at
org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1040)
at org.apache.solr.cloud.ZkController.register(ZkController.java:851)
at org.apache.solr.cloud.ZkController.register(ZkController.java:806)
at org.apache.solr.core.ZkContainer$2.run(ZkContainer.java:183)
at org.apache.solr.core.ZkContainer.registerInZk(ZkContainer.java:212)
at
org.apache.solr.core.CoreContainer.registerCore(CoreContainer.java:695)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:819)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:749)
at
org.apache.solr.handler.admin.CoreAdminOperation$1.call(CoreAdminOperation.java:119)
at
org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:367)
at
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:158)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:156)
at
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:663)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:445)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:518)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.zookeeper.ZooKeeper.multiInternal(ZooKeeper.java:935)
at org.apache.zookeeper.ZooKeeper.multi(ZooKeeper.java:915)
at
org.apache.solr.common.cloud.SolrZkClient$11.execute(SolrZkClient.java:572)
at
org.apache.solr.common.cloud.SolrZkClient$11.execute(SolrZkClient.java:569)
at
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
at
org.apache.solr.common.cloud.SolrZkClient.multi(SolrZkClient.java:569)
at
org.apache.solr.cloud.ShardLeaderElectionContextBase$1.execute(ElectionContext.java:201)
at
org.apache.solr.common.util.RetryUtil.retryOnThrowable(RetryUtil.java:49)
at
org.apache.solr.common.util.RetryUtil.retryOnThrowable(RetryUtil.java:42)
at
org.apache.solr.cloud.ShardLeaderElectionContextBase.runLeaderProcess(ElectionContext.java:183)
... 86 more
2016-08-21 21:34:00.493 INFO (qtp110456297-11) [c:mycollection s:shard1
r:core_node1 x:mycollection_shard1_replica1] o.a.s.c.ShardLeaderElectionContext
There may be a better leader candidate than us - going back into recovery
2016-08-21 21:34:00.494 INFO (qtp110456297-11) [c:mycollection s:shard1
r:core_node1 x:mycollection_shard1_replica1] o.a.s.c.ElectionContext Canceling
election
/collections/mycollection/leader_elect/shard1/election/96456851103481860-core_node1-n_0000000014
2016-08-21 21:34:00.494 WARN (main-SendThread(172.28.128.3:2181)) [ ]
o.a.z.ClientCnxn Session 0x156aefeba2d0004 for server
172.28.128.3/172.28.128.3:2181, unexpected error, closing socket connection and
attempting reconnect
Thanks again,
Chris
On 22/08/2016, 14:11, "Shawn Heisey" <apa...@elyograg.org> wrote:
On 8/22/2016 6:20 AM, Chris Rogers wrote:
> It’s then that I start seeing lots of errors in the Solr logs, and lots
of repetitive messages appearing in Zookeeper:
>
> These two Solr errors over and over:
>
> java.io.IOException: Xid out of order. Got Xid 1299 with err 0 expected
Xid 1298 for a packet with details: clientPath:null serverPath:null
finished:false header:: 1298,14 replyHeader:: 0,0,-4 request::
org.apache.zookeeper.MultiTransactionRecord@95acc4f3 response::
org.apache.zookeeper.MultiResponse@0
That appears to be one log message, but you said there were two. Also,
this message is incomplete. It is missing the timestamp at the
beginning and appears to have been cut off at the end too. I think the
message probably had *many* more lines of output that weren't included.
> And this from Zookeeper:
<snip>
> 2016-08-21 21:33:39,147 - WARN
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:ZooKeeperServer@593] - Dropping
packet at server of type 14
> 2016-08-21 21:33:39,154 - WARN
[NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@634] -
EndOfStreamException: Unable to read additional data from client sessionid
0x156aefeba2d0004, likely client has closed socket
That seems to be saying that Solr closed the connection to zookeeper. I
have no idea what might be wrong, based just on what's been provided
here. This section of logging seems to contain everything related to
the specific connection from port 54548, and if that's true, then it
does not appear to have been a timeout.
Is there anything in the solr.log file at the timestamp at or near
21:33:39.154(when zookeeper thought the connection was closed)?
What version of Solr? What version of zookeeper did you install on the
other node?
Thanks,
Shawn
