Sorry, I believe this can be disregarded. There were changes made to system time that likely caused this state. Apologies, Ryan
On Mon, Sep 29, 2014 at 8:24 AM, Ryan Cutter <ryancut...@gmail.com> wrote: > Solr 4.7.2 went down during a period of little activity. Wondering if > anyone has an idea about what's going on, thanks! > > INFO - 2014-09-26 15:35:00.152; > org.apache.solr.cloud.DistributedQueue$LatchChildWatcher; LatchChildWatcher > fired on path: null state: Disconnected type None > > then eventually: > > WARN - 2014-09-26 15:35:00.377; > org.apache.solr.cloud.OverseerCollectionProcessor; Overseer cannot talk to > ZK > > and: > > WARN - 2014-09-26 15:35:00.454; > org.apache.solr.cloud.Overseer$ClusterStateUpdater; Solr cannot talk to ZK, > exiting Overseer main queue loop > org.apache.zookeeper.KeeperException$SessionExpiredException: > KeeperErrorCode = Session expired for /overseer/queue > at > org.apache.zookeeper.KeeperException.create(KeeperException.java:127) > at > org.apache.zookeeper.KeeperException.create(KeeperException.java:51) > at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468) > at > org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:257) > at > org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:254) > at > org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73) > at > org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:254) > at > org.apache.solr.cloud.DistributedQueue.orderedChildren(DistributedQueue.java:89) > at > org.apache.solr.cloud.DistributedQueue.peek(DistributedQueue.java:411) > at > org.apache.solr.cloud.DistributedQueue.peek(DistributedQueue.java:391) > at > org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:173) > at java.lang.Thread.run(Thread.java:662) > as well as: > ERROR - 2014-09-26 15:35:21.025; org.apache.solr.common.SolrException; > There was a problem finding the leader in > zk:org.apache.solr.common.SolrException: Could not get leader props > at > org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:934) > at > org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:898) > at > org.apache.solr.cloud.ZkController.waitForLeaderToSeeDownState(ZkController.java:1422) > at > org.apache.solr.cloud.ZkController.registerAllCoresAsDown(ZkController.java:370) > at > org.apache.solr.cloud.ZkController.access$000(ZkController.java:87) > at > org.apache.solr.cloud.ZkController$1.command(ZkController.java:222) > at > org.apache.solr.common.cloud.ConnectionManager$1$1.run(ConnectionManager.java:166) > Caused by: org.apache.zookeeper.KeeperException$NoNodeException: > KeeperErrorCode = NoNode for /collections/collection1/leaders/shard1 > at > org.apache.zookeeper.KeeperException.create(KeeperException.java:111) > at > org.apache.zookeeper.KeeperException.create(KeeperException.java:51) > at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1151) > at > org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:274) > at > org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:271) > at > org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73) > at > org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:271) > at > org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:912) > ... 6 more > > In ZK's log at about the same time: > > 2014-09-26 15:35:00,000 [myid:] - INFO [SessionTracker:ZooKeeperServer@334] > - Expiring session 0x144e09d7b910008, timeout of 30000ms exceeded > 2014-09-26 15:35:00,000 [myid:] - WARN [NIOServerCxn.Factory: > 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@349] - caught end of stream exception > EndOfStreamException: Unable to read additional data from client sessionid > 0x144e09d7b910007, likely client has closed socket > at > org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220) > at > org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:224) > at java.lang.Thread.run(Thread.java:662) > > as well as: > > 2014-09-26 15:35:00,534 [myid:] - INFO [ProcessThread(sid:0 > cport:-1)::PrepRequestProcessor@617] - Got user-level KeeperException > when processing sessionid:0x144e09d7b91000a type:delete cxid:0x8 zxid:0xbdd > txntype:-1 reqpath:n/a Error Path:/overseer_elect/leader > Error:KeeperErrorCode = NoNode for /overseer_elect/leader > 2014-09-26 15:35:00,572 [myid:] - INFO [ProcessThread(sid:0 > cport:-1)::PrepRequestProcessor@617] - Got user-level KeeperException > when processing sessionid:0x144e09d7b91000a type:create cxid:0xd zxid:0xbdf > txntype:-1 reqpath:n/a Error Path:/overseer Error:KeeperErrorCode = > NodeExists for /overseer > >
