Re: Solr Prod Issue | KeeperErrorCode = ConnectionLoss for /overseer_elect/leader

Wed, 05 Jul 2017 05:28:17 -0700

From the fact that someone has tried to access /etc/passwd file via your Solr (see all those WARN messages), it seems you have it exposed to the world, unless of course it's a security scanner you use internally. Internet is a hostile place, and the very first thing I would do is shield Solr from external traffic. Even if it's your own security scanning, I wouldn't do it until you have the system stable.
Doing the above you'll reduce noise in the logs and might be able to better identify the issue. 


Losing the Zookeeper connection is typically a Java garbage collection 
issue. If GC causes too long pauses, the connection may time out. So I 
would recommend you start by reading 
https://wiki.apache.org/solr/SolrPerformanceProblems and 
https://wiki.apache.org/solr/ShawnHeisey. Also make sure that 
Zookeeper's Java settings are good.


--Ere

Bhalla, Rahat kirjoitti 5.7.2017 klo 11.05:

Hi

I’m not sure if any of you have had a chance to see this email yet.


We had a reoccurrence of the Issue Today, and I’m attaching the Logs 
from today as well inline below.



Please let me know if any of you have seen this issue before as this 
would really help me to get to the root of the problem to fix it. I’m a 
little lost here and not entirely sure what to do.


Thanks,

Rahat Bhalla


8696248 [qtp778720569-28] [ WARN] 2017-07-04 01:40:20 
(HttpParser.java:parseNext:1391) - parse exception: 
java.lang.IllegalArgumentException: No Authority for 
HttpChannelOverHttp@30a86e14{r=0,c=false,a=IDLE,uri=null}


java.lang.IllegalArgumentException: No Authority


         at 
org.eclipse.jetty.http.HostPortHttpField.<init>(HostPortHttpField.java:43)



         at 
org.eclipse.jetty.http.HttpParser.parsedHeader(HttpParser.java:877)



         at 
org.eclipse.jetty.http.HttpParser.parseHeaders(HttpParser.java:1050)



         at 
org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:1266)



         at 
org.eclipse.jetty.server.HttpConnection.parseRequestBuffer(HttpConnection.java:344)



         at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:227)



         at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)


         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)


         at 
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)



         at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)



         at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)



         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)



         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)


         at java.lang.Thread.run(Unknown Source)


8697308 [qtp778720569-21] [ WARN] 2017-07-04 01:40:21 
(HttpParser.java:parseNext:1364) - bad HTTP parsed: 400 Bad URI for 
HttpChannelOverHttp@127bbbb6{r=16,c=false,a=IDLE,uri=/../../../../../../../../../../etc/passwd}



8697338 [qtp778720569-29] [ WARN] 2017-07-04 01:40:21 
(HttpParser.java:parseNext:1364) - bad HTTP parsed: 400 No Host for 
HttpChannelOverHttp@50a994ce{r=29,c=false,a=IDLE,uri=null}



8697388 [qtp778720569-21] [ WARN] 2017-07-04 01:40:22 
(HttpParser.java:parseNext:1364) - bad HTTP parsed: 400 Bad URI for 
HttpChannelOverHttp@19a624ec{r=1,c=false,a=IDLE,uri=//prod-solr-node01.healthplan.com:9080/solr/admin/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/%2e%2e/etc/passwd}



8697401 [qtp778720569-27] [ WARN] 2017-07-04 01:40:22 
(URIUtil.java:decodePath:348) - 
/solr/admin/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/%c0%ae%c0%ae/etc/passwd 
org.eclipse.jetty.util.Utf8Appendable$NotUtf8Exception: Not valid UTF8! 
byte C0 in state 0



8697444 [qtp778720569-25] [ WARN] 2017-07-04 01:40:22 
(URIUtil.java:decodePath:348) - 
/solr/admin/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/%e0%80%ae%e0%80%ae/etc/passwd 
org.eclipse.jetty.util.Utf8Appendable$NotUtf8Exception: Not valid UTF8! 
byte 80 in state 4



8697475 [qtp778720569-26] [ WARN] 2017-07-04 01:40:22 
(URIUtil.java:decodePath:348) - 
/solr/admin/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/%f0%80%80%ae%f0%80%80%ae/etc/passwd 
org.eclipse.jetty.util.Utf8Appendable$NotUtf8Exception: Not valid UTF8! 
byte 80 in state 6



8697500 [qtp778720569-29] [ WARN] 2017-07-04 01:40:22 
(URIUtil.java:decodePath:348) - 
/solr/admin/%f8%80%80%80%ae%f8%80%80%80%ae/%f8%80%80%80%ae%f8%80%80%80%ae/%f8%80%80%80%ae%f8%80%80%80%ae/%f8%80%80%80%ae%f8%80%80%80%ae/%f8%80%80%80%ae%f8%80%80%80%ae/%f8%80%80%80%ae%f8%80%80%80%ae/%f8%80%80%80%ae%f8%80%80%80%ae/%f8%80%80%80%ae%f8%80%80%80%ae/%f8%80%80%80%ae%f8%80%80%80%ae/%f8%80%80%80%ae%f8%80%80%80%ae/%f8%80%80%80%ae%f8%80%80%80%ae/%f8%80%80%80%ae%f8%80%80%80%ae/etc/passwd 
org.eclipse.jetty.util.Utf8Appendable$NotUtf8Exception: Not valid UTF8! 
byte F8 in state 0



8706641 [qtp778720569-27] [ WARN] 2017-07-04 01:40:31 
(HttpParser.java:parseNext:1364) - bad HTTP parsed: 400 Unknown Version 
for HttpChannelOverHttp@7fcd594a{r=54,c=false,a=IDLE,uri=null}



8707033 [qtp778720569-20] [ WARN] 2017-07-04 01:40:31 
(HttpParser.java:parseNext:1364) - bad HTTP parsed: 400 Unknown Version 
for HttpChannelOverHttp@66740d77{r=54,c=false,a=IDLE,uri=null}



8719390 [qtp778720569-23] [ WARN] 2017-07-04 01:40:44 
(HttpParser.java:<init>:1740) - Illegal character 0xA in 
state=HEADER_IN_NAME for buffer 
HeapByteBuffer@41ed5e30[p=220,l=273,c=8192,r=53]={POST 
/wp-content/...name="qqfile"\r\n<<<filename="QualysU...ys-Scan: 
VM\r\n\r\n>>>\n\r\nq=*%3A*&df=tex...\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00}



8719391 [qtp778720569-23] [ WARN] 2017-07-04 01:40:44 
(HttpParser.java:parseNext:1364) - bad HTTP parsed: 400 Illegal 
character 0xA for HttpChannelOverHttp@235b51f6{r=2,c=false,a=IDLE,uri=null}



8949147 [qtp778720569-27] [ WARN] 2017-07-04 01:44:33 
(HttpParser.java:parseNext:1364) - bad HTTP parsed: 400 HTTP/0.9 not 
supported for HttpChannelOverHttp@62d6261e{r=0,c=false,a=IDLE,uri=null}



8949149 [qtp778720569-26] [ WARN] 2017-07-04 01:44:33 
(HttpParser.java:parseNext:1364) - bad HTTP parsed: 400 HTTP/0.9 not 
supported for HttpChannelOverHttp@33f62c0{r=0,c=false,a=IDLE,uri=null}



8949163 [qtp778720569-26] [ WARN] 2017-07-04 01:44:33 
(HttpParser.java:parseNext:1364) - bad HTTP parsed: 400 HTTP/0.9 not 
supported for HttpChannelOverHttp@6d58f7d7{r=0,c=false,a=IDLE,uri=null}



67338023 
[OverseerCollectionConfigSetProcessor-98247501534199809-prod-solr-node01:9080_solr-n_0000000158] 
[ERROR] 2017-07-04 05:57:42 (SolrException.java:log:159) - 
:org.apache.zookeeper.KeeperException$ConnectionLossException: 
KeeperErrorCode = ConnectionLoss for /overseer/collection-queue-work



         at 
org.apache.zookeeper.KeeperException.create(KeeperException.java:99)



         at 
org.apache.zookeeper.KeeperException.create(KeeperException.java:51)


         at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1472)


         at 
org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:331)



         at 
org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:328)



         at 
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)



         at 
org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:328)



         at 
org.apache.solr.cloud.DistributedQueue.fetchZkChildren(DistributedQueue.java:299)



         at 
org.apache.solr.cloud.DistributedQueue.firstChild(DistributedQueue.java:279)



         at 
org.apache.solr.cloud.DistributedQueue.peekElements(DistributedQueue.java:328)



         at 
org.apache.solr.cloud.OverseerTaskQueue.peekTopN(OverseerTaskQueue.java:236)



         at 
org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:226)


         at java.lang.Thread.run(Unknown Source)


67365997 
[OverseerCollectionConfigSetProcessor-98247501534199809-prod-solr-node01:9080_solr-n_0000000158] 
[ERROR] 2017-07-04 05:58:10 (OverseerTaskProcessor.java:amILeader:392) -



org.apache.zookeeper.KeeperException$ConnectionLossException: 
KeeperErrorCode = ConnectionLoss for /overseer_elect/leader



         at 
org.apache.zookeeper.KeeperException.create(KeeperException.java:99)



         at 
org.apache.zookeeper.KeeperException.create(KeeperException.java:51)


         at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)


         at 
org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:348)



         at 
org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:345)



         at 
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)



         at 
org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:345)



         at 
org.apache.solr.cloud.OverseerTaskProcessor.amILeader(OverseerTaskProcessor.java:384)



         at 
org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:191)


         at java.lang.Thread.run(Unknown Source)


67381954 [qtp778720569-24] [ERROR] 2017-07-04 05:58:26 
(SolrException.java:log:148) - java.io.IOException: Error opening 
/configs/feedsInBoundCarrier/db-data-config.xml



         at 
org.apache.solr.cloud.ZkSolrResourceLoader.openResource(ZkSolrResourceLoader.java:111)



         at 
org.apache.solr.handler.dataimport.DataImporter.maybeReloadConfiguration(DataImporter.java:125)



         at 
org.apache.solr.handler.dataimport.DataImportHandler.handleRequestBody(DataImportHandler.java:176)



         at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:156)


         at org.apache.solr.core.SolrCore.execute(SolrCore.java:2036)


         at 
org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:657)


         at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:464)


         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)



         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)



         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)



         at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)



         at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)



         at 
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)



         at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)



         at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)



         at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)



         at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)



         at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)



         at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)



         at 
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)



         at 
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)



         at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)


         at org.eclipse.jetty.server.Server.handle(Server.java:518)


         at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)



         at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)



         at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)


         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)


         at 
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)



         at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)



         at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)



         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)



         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)


         at java.lang.Thread.run(Unknown Source)


Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: 
KeeperErrorCode = ConnectionLoss for 
/configs/feedsInBoundCarrier/db-data-config.xml



         at 
org.apache.zookeeper.KeeperException.create(KeeperException.java:99)



         at 
org.apache.zookeeper.KeeperException.create(KeeperException.java:51)


         at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)


         at 
org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:314)



         at 
org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:311)



         at 
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)



         at 
org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:311)



         at 
org.apache.solr.cloud.ZkController.pathExists(ZkController.java:796)



         at 
org.apache.solr.cloud.ZkSolrResourceLoader.openResource(ZkSolrResourceLoader.java:89)


         ... 32 more


67381955 [qtp778720569-24] [ERROR] 2017-07-04 05:58:26 
(SolrException.java:log:159) - null:java.io.IOException: Error opening 
/configs/feedsInBoundCarrier/db-data-config.xml



         at 
org.apache.solr.cloud.ZkSolrResourceLoader.openResource(ZkSolrResourceLoader.java:111)



         at 
org.apache.solr.handler.dataimport.DataImporter.maybeReloadConfiguration(DataImporter.java:125)



         at 
org.apache.solr.handler.dataimport.DataImportHandler.handleRequestBody(DataImportHandler.java:176)



         at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:156)


         at org.apache.solr.core.SolrCore.execute(SolrCore.java:2036)


         at 
org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:657)


         at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:464)


         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)



         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)



         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)



         at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)



         at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)



         at 
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)



         at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)



         at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)



         at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)



         at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)



         at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)



         at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)



         at 
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)



         at 
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)



         at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)


         at org.eclipse.jetty.server.Server.handle(Server.java:518)


         at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)



         at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)



         at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)


         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)


         at 
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)



         at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)



         at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)



         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)



         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)


         at java.lang.Thread.run(Unknown Source)


Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: 
KeeperErrorCode = ConnectionLoss for 
/configs/feedsInBoundCarrier/db-data-config.xml



         at 
org.apache.zookeeper.KeeperException.create(KeeperException.java:99)



         at 
org.apache.zookeeper.KeeperException.create(KeeperException.java:51)


         at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)


         at 
org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:314)



         at 
org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:311)



         at 
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)



         at 
org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:311)



         at 
org.apache.solr.cloud.ZkController.pathExists(ZkController.java:796)



         at 
org.apache.solr.cloud.ZkSolrResourceLoader.openResource(ZkSolrResourceLoader.java:89)


         ... 32 more


67393217 
[OverseerCollectionConfigSetProcessor-98247501534199809-prod-solr-node01:9080_solr-n_0000000158] 
[ERROR] 2017-07-04 05:58:37 (OverseerTaskProcessor.java:amILeader:392) -



org.apache.zookeeper.KeeperException$ConnectionLossException: 
KeeperErrorCode = ConnectionLoss for /overseer_elect/leader



         at 
org.apache.zookeeper.KeeperException.create(KeeperException.java:99)



         at 
org.apache.zookeeper.KeeperException.create(KeeperException.java:51)


         at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)


         at 
org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:348)



         at 
org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:345)



         at 
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)



         at 
org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:345)



         at 
org.apache.solr.cloud.OverseerTaskProcessor.amILeader(OverseerTaskProcessor.java:384)



         at 
org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:191)


         at java.lang.Thread.run(Unknown Source)


67420293 
[OverseerCollectionConfigSetProcessor-98247501534199809-prod-solr-node01:9080_solr-n_0000000158] 
[ERROR] 2017-07-04 05:59:04 (OverseerTaskProcessor.java:amILeader:392) -



org.apache.zookeeper.KeeperException$ConnectionLossException: 
KeeperErrorCode = ConnectionLoss for /overseer_elect/leader



         at 
org.apache.zookeeper.KeeperException.create(KeeperException.java:99)



         at 
org.apache.zookeeper.KeeperException.create(KeeperException.java:51)


         at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)


         at 
org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:348)



         at 
org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:345)



         at 
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)



         at 
org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:345)



         at 
org.apache.solr.cloud.OverseerTaskProcessor.amILeader(OverseerTaskProcessor.java:384)



         at 
org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:191)


         at java.lang.Thread.run(Unknown Source)


67437164 [qtp778720569-24] [ERROR] 2017-07-04 05:59:21 
(SolrException.java:log:148) - java.io.IOException: Error opening 
/configs/txnInBoundNonEx/db-data-config.xml



         at 
org.apache.solr.cloud.ZkSolrResourceLoader.openResource(ZkSolrResourceLoader.java:111)



         at 
org.apache.solr.handler.dataimport.DataImporter.maybeReloadConfiguration(DataImporter.java:125)



         at 
org.apache.solr.handler.dataimport.DataImportHandler.handleRequestBody(DataImportHandler.java:176)



         at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:156)


         at org.apache.solr.core.SolrCore.execute(SolrCore.java:2036)


         at 
org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:657)


         at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:464)


         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)



         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)



         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)



         at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)



         at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)



         at 
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)



         at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)



         at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)



         at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)



         at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)



         at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)



         at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)



         at 
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)



         at 
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)



         at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)


         at org.eclipse.jetty.server.Server.handle(Server.java:518)


         at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)



         at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)



         at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)


         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)


         at 
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)



         at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)



         at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)



         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)



         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)


         at java.lang.Thread.run(Unknown Source)


Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: 
KeeperErrorCode = ConnectionLoss for 
/configs/txnInBoundNonEx/db-data-config.xml



         at 
org.apache.zookeeper.KeeperException.create(KeeperException.java:99)



         at 
org.apache.zookeeper.KeeperException.create(KeeperException.java:51)


         at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)


         at 
org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:314)



         at 
org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:311)



         at 
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)



         at 
org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:311)



         at 
org.apache.solr.cloud.ZkController.pathExists(ZkController.java:796)



         at 
org.apache.solr.cloud.ZkSolrResourceLoader.openResource(ZkSolrResourceLoader.java:89)


         ... 32 more


67437165 [qtp778720569-24] [ERROR] 2017-07-04 05:59:21 
(SolrException.java:log:159) - null:java.io.IOException: Error opening 
/configs/txnInBoundNonEx/db-data-config.xml



         at 
org.apache.solr.cloud.ZkSolrResourceLoader.openResource(ZkSolrResourceLoader.java:111)



         at 
org.apache.solr.handler.dataimport.DataImporter.maybeReloadConfiguration(DataImporter.java:125)



         at 
org.apache.solr.handler.dataimport.DataImportHandler.handleRequestBody(DataImportHandler.java:176)



         at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:156)


         at org.apache.solr.core.SolrCore.execute(SolrCore.java:2036)


         at 
org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:657)


         at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:464)


         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)



         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)



         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)



         at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)



         at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)



         at 
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)



         at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)



         at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)



         at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)



         at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)



         at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)



         at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)



         at 
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)



         at 
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)



         at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)


         at org.eclipse.jetty.server.Server.handle(Server.java:518)


         at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)



         at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)



         at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)


         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)


         at 
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)



         at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)



         at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)



         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)



         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)


         at java.lang.Thread.run(Unknown Source)


Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: 
KeeperErrorCode = ConnectionLoss for 
/configs/txnInBoundNonEx/db-data-config.xml



         at 
org.apache.zookeeper.KeeperException.create(KeeperException.java:99)



         at 
org.apache.zookeeper.KeeperException.create(KeeperException.java:51)


         at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)


         at 
org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:314)



         at 
org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:311)



         at 
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)



         at 
org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:311)



         at 
org.apache.solr.cloud.ZkController.pathExists(ZkController.java:796)



         at 
org.apache.solr.cloud.ZkSolrResourceLoader.openResource(ZkSolrResourceLoader.java:89)


         ... 32 more

*From:* Bhalla, Rahat
*Sent:* Monday, July 03, 2017 8:38 PM
*To:* solr-user@lucene.apache.org

*Subject:* Solr Prod Issue | KeeperErrorCode = ConnectionLoss for 
/overseer_elect/leader

*Importance:* High

Hi Solr Users,


I hope this email finds you all in the best of spirits and in a mood 
where you’d be willing to help a young developer (me J ) with issues 
that I’m facing in regards with the Solr Cloud.



At my organization, we are running a Solr Cloud with 5 Nodes for Solr 
Instances with 13 collections spread across the 5 nodes and an ensemble 
of 3 zookeeper instances spread across three different nodes.



Over the last one week, our leader node seems to be going down every 
other day and while we restart the solr instances they still go down 
within the next 24 Hours or more.



We have tried rebooting the nodes that host the solr instances and that 
hasn’t helped. We plan to clear out the zookeeper logs and data folders 
before the restart of the zookeeper instances.



As of now, I’m the only one supporting Solr in my organization and any 
insight from you could help me a great deal to fix the issue. I’m 
copying the Exception stack trace from this morning. Any recommendations 
that you might have will be great appreciated.


*Below is a snapshot of one of the zoo nodes:*

*Exception Stacktrace*


138127149 
[OverseerCollectionConfigSetProcessor-98234161688412161-prod-solr-node01:9080_solr-n_0000000140] 
[ERROR] 2017-07-03 05:02:55 (OverseerTaskProcessor.java:amILeader:392) -



org.apache.zookeeper.KeeperException$ConnectionLossException: 
KeeperErrorCode = ConnectionLoss for /overseer_elect/leader



         at 
org.apache.zookeeper.KeeperException.create(KeeperException.java:99)



         at 
org.apache.zookeeper.KeeperException.create(KeeperException.java:51)


         at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)


         at 
org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:348)



         at 
org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:345)



         at 
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)



         at 
org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:345)



         at 
org.apache.solr.cloud.OverseerTaskProcessor.amILeader(OverseerTaskProcessor.java:384)



         at 
org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:191)


         at java.lang.Thread.run(Unknown Source)


138133409 [qtp778720569-10329] [ERROR] 2017-07-03 05:03:01 
(SolrException.java:log:148) - org.apache.solr.common.SolrException: 
Could not load collection from ZK: feedsOutBoundToExchange



         at 
org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1047)



         at 
org.apache.solr.common.cloud.ZkStateReader$LazyCollectionRef.get(ZkStateReader.java:610)



         at 
org.apache.solr.common.cloud.ClusterState.getCollectionsMap(ClusterState.java:248)



         at 
org.apache.solr.handler.admin.CollectionsHandler$CollectionOperation$20.call(CollectionsHandler.java:674)



         at 
org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:195)



         at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:156)



         at 
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:663)


         at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:445)


         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)



         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)



         at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)



         at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)



         at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)



         at 
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)



         at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)



         at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)



         at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)



         at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)



         at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)



         at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)



         at 
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)



         at 
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)



         at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)


         at org.eclipse.jetty.server.Server.handle(Server.java:518)


         at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)



         at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)



         at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)


         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)


         at 
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)



         at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)



         at 
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)



         at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)



         at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)


         at java.lang.Thread.run(Unknown Source)


Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: 
KeeperErrorCode = ConnectionLoss for 
/collections/feedsOutBoundToExchange/state.json



         at 
org.apache.zookeeper.KeeperException.create(KeeperException.java:99)



         at 
org.apache.zookeeper.KeeperException.create(KeeperException.java:51)


         at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1155)


         at 
org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:348)



         at 
org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:345)



         at 
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)



         at 
org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:345)



         at 
org.apache.solr.common.cloud.ZkStateReader.fetchCollectionState(ZkStateReader.java:1059)



         at 
org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1045)


         ... 33 more

*Thanks,*

**

*Rahat Bhalla*

*HealthPlan/Services/*
Phone: (813) 289-1000  EXT: 7002249**

rbha...@healthplan.com <mailto:rbha...@healthplan.com>

_www.healthplan.com <http://www.healthplan.com/> _


_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

CONFIDENTIALITY NOTICE: This email message, including any attachments, 
is for the sole use of the intended recipient(s) and may contain 
confidential and privileged information and/or Protected Health 
Information (PHI) subject to protection under the law, including the 
Health Insurance Portability and Accountability Act of 1996, as amended 
(HIPAA). If you are not the intended recipient or the person responsible 
for delivering the email to the intended recipient, be advised that you 
have received this email in error and that any use, disclosure, 
distribution, forwarding, printing, or copying of this email is strictly 
prohibited. If you have received this email in error, please notify the 
sender immediately and destroy all copies of the original message.


--
Ere Maijala
Kansalliskirjasto / The National Library of Finland























					Reply via email to