Hi,
Any suggestion will be really helpful. Kindly provide your inputs.
Thanks,
Modassar
On Thu, Apr 16, 2015 at 4:27 PM, Modassar Ather <modather1...@gmail.com>
wrote:
> Hi,
>
> I have a setup of 5 node SolrCloud (Lucene/Solr version 5.1.0) without
> replicas. When I am executing complex and large queries with wild-cards
> after some time I am getting following exceptions.
> The index size on each of the node is around 170GB and the memory is set
> to -Xms20g -Xmx24g on each node.
>
> Empty shard!
> org.apache.solr.common.SolrException log
> SEVERE: org.apache.solr.common.SolrException: no servers hosting shard:
> at
> org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:214)
> at
> org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:184)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
>
> There is no OutofMemory or any other major lead for me to understand what
> had caused it. May be I am missing something. There are following other
> exceptions:
>
> SEVERE: null:org.apache.solr.common.SolrException:
> org.apache.solr.client.solrj.SolrServerException: Timeout occurred while
> waiting response from server at: http://<server>:8080/solr/collection
> at
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:342)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1984)
> at
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:829)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:446)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:220)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
> at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
> at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
> at
> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
> at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
> at
> org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:193)
> at
> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
> at
> org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:313)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
>
>
> WARNING: listener throws error
> org.apache.solr.common.SolrException:
> org.apache.zookeeper.KeeperException$SessionExpiredException:
> KeeperErrorCode = Session expired for /configs/collection/params.json
> at
> org.apache.solr.core.RequestParams.getFreshRequestParams(RequestParams.java:163)
> at
> org.apache.solr.core.SolrConfig.refreshRequestParams(SolrConfig.java:919)
> at org.apache.solr.core.SolrCore$11.run(SolrCore.java:2500)
> at org.apache.solr.cloud.ZkController$4.run(ZkController.java:2366)
> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException:
> KeeperErrorCode = Session expired for /configs/collection/params.json
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
> at
> org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:294)
> at
> org.apache.solr.common.cloud.SolrZkClient$4.execute(SolrZkClient.java:291)
> at
> org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:61)
> at
> org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:291)
> at
> org.apache.solr.core.RequestParams.getFreshRequestParams(RequestParams.java:153)
> ... 3 more
>
> The Zookeeper session timeout is set to 30000. In the log file I can see
> logs of the following pattern for all the queries I fired.
> INFO: [collection] webapp=/solr path=/search_handler
> params={sort=score+desc&start=0&q=(ft:search term)} status=0 QTime=<time>
> If I am not wrong they are getting executed but somehow as the shard is
> gone down which I can see in /clusterstate.json under the log, the search
> is failing.
>
> Kindly help me debug/solve this problem.
>
> Regards,
> Modassar
>
