Hello, I am seeing "ZookeeperServer not running" WARM messages in zookeeper logs which is causing the Solr client connections to timeout...
What could be the problem? ZK: 3.4.10 Zookeeper.out == 2018-10-22 06:04:51,071 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message format version), 5 (n.leader), 0xf00000461 (n.zxid), 0x10 (n.round), FOLLOWING (n.state), 4 (n.sid), 0xf (n.peerEpoch) LOOKING (my state) 2018-10-22 06:04:51,093 [myid:2] - INFO [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:2182:NIOServerCnxnFactory@192] - Accepted socket connection from /192.72.25.177:39514 2018-10-22 06:04:51,094 [myid:2] - WARN [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:2182:NIOServerCnxn@373] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running 2018-10-22 06:04:51,094 [myid:2] - INFO [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:2182:NIOServerCnxn@1044] - Closed socket connection for client /192.72.25.177:39514 (no session established for client) 2018-10-22 06:04:51,138 [myid:2] - INFO [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:2182:NIOServerCnxnFactory@192] - Accepted socket connection from /192.3.101.219:56298 2018-10-22 06:04:51,138 [myid:2] - WARN [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:2182:NIOServerCnxn@373] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running 2018-10-22 06:04:51,139 [myid:2] - INFO [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:2182:NIOServerCnxn@1044] - Closed socket connection for client /192.3.101.219:56298 (no session established for client) 2018-10-22 06:04:51,250 [myid:2] - INFO [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:2182:NIOServerCnxnFactory@192] - Accepted socket connection from /192.72.27.181:46414 2018-10-22 06:04:51,250 [myid:2] - WARN [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:2182:NIOServerCnxn@373] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running 2018-10-22 06:04:51,250 [myid:2] - INFO [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:2182:NIOServerCnxn@1044] - Closed socket connection for client /192.72.27.181:46414 (no session established for client) 2018-10-22 06:04:51,275 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message format version), 4 (n.leader), 0xf00000461 (n.zxid), 0x192 (n.round), LOOKING (n.state), 4 (n.sid), 0xf (n.peerEpoch) LOOKING (my state) 2018-10-22 06:04:51,275 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message format version), 4 (n.leader), 0xf00000461 (n.zxid), 0x192 (n.round), LOOKING (n.state), 2 (n.sid), 0xf (n.peerEpoch) LOOKING (my state) 2018-10-22 06:04:51,275 [myid:2] - INFO [WorkerReceiver[myid=2]:FastLeaderElection@600] - Notification: 1 (message format version), 4 (n.leader), 0xf00000461 (n.zxid), 0x192 (n.round), LOOKING (n.state), 1 (n.sid), 0xf (n.peerEpoch) LOOKING (my state) 2018-10-22 06:04:51,309 [myid:2] - INFO [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:2182:NIOServerCnxnFactory@192] - Accepted socket connection from /192.72.5.212:38944 2018-10-22 06:04:51,309 [myid:2] - WARN [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:2182:NIOServerCnxn@373] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running 2018-10-22 06:04:51,309 [myid:2] - INFO [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:2182:NIOServerCnxn@1044] - Closed socket connection for client /192.72.5.212:38944 (no session established for client) 2018-10-22 06:04:51,356 [myid:2] - INFO [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:2182:NIOServerCnxnFactory@192] - Accepted socket connection from /192.72.7.201:59310 2018-10-22 06:04:51,356 [myid:2] - WARN [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:2182:NIOServerCnxn@373] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running 2018-10-22 06:04:51,356 [myid:2] - INFO [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:2182:NIOServerCnxn@1044] - Closed socket connection for client /192.72.7.201:59310 (no session established for client) 2018-10-22 06:04:51,402 [myid:2] - INFO [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:2182:NIOServerCnxnFactory@192] - Accepted socket connection from /192.3.101.219:56302 2018-10-22 06:04:51,402 [myid:2] - WARN [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:2182:NIOServerCnxn@373] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running 2018-10-22 06:04:51,402 [myid:2] - INFO [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:2182:NIOServerCnxn@1044] - Closed socket connection for client /192.3.101.219:56302 (no session established for client) 2018-10-22 06:04:51,467 [myid:2] - INFO [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:2182:NIOServerCnxnFactory@192] - Accepted socket connection from /192.72.7.205:46694 2018-10-22 06:04:51,468 [myid:2] - WARN [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:2182:NIOServerCnxn@373] - Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running 2018-10-22 06:04:51,468 [myid:2] - INFO [NIOServerCxn.Factory: 0.0.0.0/0.0.0.0:2182:NIOServerCnxn@1044] - Closed socket connection for client /192.72.7.205:46694 (no session established for client) 2018-10-22 06:04:51,476 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2182:QuorumPeer@935] - FOLLOWING 2018-10-22 06:04:51,476 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2182:ZooKeeperServer@173] - Created server with tickTime 2000 minSessionTimeout 4000 maxSessionTimeout 40000 datadir /app/zookeeper-3.4.10/log/version-2 snapdir /app/zookeeper-3.4.10/data/version-2 2018-10-22 06:04:51,476 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2182:Follower@64] - FOLLOWING - LEADER ELECTION TOOK - 1008 2018-10-22 06:04:51,477 [myid:2] - INFO [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2182:QuorumPeer$QuorumServer@167] - Resolved hostname: srch0120.whc.dc02.us.adp to address: srch0120/ 192.3.35.243 solr.log 2018-10-22 10:02:21.466 WARN (main-SendThread(srch0118:2182)) [ ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 26675ms for sessionid 0x5665c67cb0d000b 2018-10-22 10:02:48.891 WARN (main-SendThread(srch0119:2182)) [ ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 26667ms for sessionid 0x5665c67cb0d000b 2018-10-22 10:02:57.720 WARN (main-SendThread(srch0117:2182)) [ ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 8007ms for sessionid 0x5665c67cb0d000b 2018-10-22 10:03:06.776 WARN (main-SendThread(srch0120:2182)) [ ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 8006ms for sessionid 0x5665c67cb0d000b 2018-10-22 10:03:15.734 WARN (main-SendThread(srch0121:2182)) [ ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 8004ms for sessionid 0x5665c67cb0d000b 2018-10-22 10:03:24.568 WARN (main-SendThread(srch0118:2182)) [ ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 8003ms for sessionid 0x5665c67cb0d000b 2018-10-22 10:03:33.844 WARN (main-SendThread(srch0119:2182)) [ ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 8001ms for sessionid 0x5665c67cb0d000b 2018-10-22 10:03:42.377 WARN (main-SendThread(srch0117:2182)) [ ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 8008ms for sessionid 0x5665c67cb0d000b 2018-10-22 10:03:51.138 WARN (main-SendThread(srch0120:2182)) [ ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 8004ms for sessionid 0x5665c67cb0d000b 2018-10-22 10:04:10.229 WARN (main-SendThread(srch0119:2182)) [ ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 8008ms for sessionid 0x5665c67cb0d000b 2018-10-22 10:04:20.669 WARN (main-SendThread(srch0119:2182)) [ ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 9272ms for sessionid 0x5665c67cb0d000b 2018-10-22 10:04:34.076 WARN (main-SendThread(srch0119:2182)) [ ] o.a.z.ClientCnxn Client session timed out, have not heard from server in 9873ms for sessionid 0x5665c67cb0d000b
