Hi,
I have a setup of 5 node SolrCloud (Lucene/Solr version 5.1.0) without
replicas. When I am executing complex and large queries with wild-cards
after some time I am getting following exceptions.
The index size on each of the node is around 170GB and the memory is set to
-Xms20g -Xmx24g on each node.
Empty shard!
org.apache.solr.common.SolrException log
SEVERE: org.apache.solr.common.SolrException: no servers hosting shard:
at
org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:214)
at
org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:184)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
There is no OutofMemory or any other major lead for me to understand what
had caused it. May be I am missing something. There are following other
exceptions:
SEVERE: null:org.apache.solr.common.SolrException:
org.apache.solr.client.solrj.SolrServerException: Timeout occurred while
waiting response from server at: http://<server>:8080/solr/collection
at
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:342)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1984)
at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:829)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:446)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:220)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:193)
at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
at
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:313)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
WARNING: listener throws error
org.apache.solr.common.SolrException:
org.apache.zookeeper.KeeperException$SessionExpiredException:
KeeperErrorCode = Session expired for /configs/collection/params.json
at
org.apache.solr.core.RequestParams.getFreshRequestParams(RequestParams.java:163)
at
org.apache.solr.core.SolrConfig.refreshRequestParams(SolrConfig.java:919)
at org.apache.solr.core.SolrCore$11.run(SolrCore.java:2500)
at org.apache.solr.cloud.ZkController$4.run(ZkController.java:2366)
Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException:
KeeperErrorCode = Session expired for /configs/collection/params.json
at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
at
org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:294)
at
org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:291)
at
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
at
org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:291)
at
org.apache.solr.core.RequestParams.getFreshRequestParams(RequestParams.java:153)
... 3 more
The Zookeeper session timeout is set to 30000. In the log file I can see
logs of the following pattern for all the queries I fired.
INFO: [collection] webapp=/solr path=/search_handler
params={sort=score+desc&start=0&q=(ft:search term)} status=0 QTime=<time>
If I am not wrong they are getting executed but somehow as the shard is
gone down which I can see in /clusterstate.json under the log, the search
is failing.
Kindly help me debug/solve this problem.
Regards,
Modassar
