Hi,
I have a setup of 4 shard Solr cluster with embedded zookeeper on one of
them. The zkClient time out is set to 30 seconds, -Xms is 20g and -Xms is
24g.
When executing a huge query with many wildcards inside it the server
crashes and becomes non-responsive. Even the dashboard does not responds
and shows connection lost error. This requires me to restart the servers.
I have set query *timeAllowed* to 5 minutes but it also seems to be not
getting honored and the query hangs around.
Kindly help me debug the issue and fix it or if there is a way the
timeAllowed can be made honored or a the query which is hanging for
sometime can be stopped.
*Following are few exceptions.*
*org.apache.zookeeper.server.NIOServerCnxn doIO*
WARNING: caught end of stream exception
EndOfStreamException: Unable to read additional data from client sessionid
<session id>, likely client has closed socket
at
org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
at
org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
at java.lang.Thread.run(Thread.java:745)
*org.apache.zookeeper.server.NIOServerCnxn sendBuffer*
SEVERE: Unexpected Exception:
java.nio.channels.CancelledKeyException
at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73)
at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77)
at
org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:151)
at
org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1081)
at
org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:404)
at
org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:200)
at
org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:131)
*org.apache.zookeeper.server.persistence.FileTxnLog commit*
WARNING: fsync-ing the write ahead log in SyncThread:0 took 28346ms which
will adversely effect operation latency. See the ZooKeeper troubleshooting
guide
org.apache.solr.common.SolrException log
SEVERE: org.apache.solr.common.SolrException: no servers hosting shard:
at
org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:149)
at
org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:119)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
*Caused by: java.lang.OutOfMemoryError: Java heap space*
at
org.apache.lucene.codecs.lucene41.Lucene41PostingsReader$BlockDocsAndPositionsEnum.<init>(Lucene41PostingsReader.java:640)
at
org.apache.lucene.codecs.lucene41.Lucene41PostingsReader.docsAndPositions(Lucene41PostingsReader.java:278)
at
org.apache.lucene.codecs.blocktree.SegmentTermsEnum.docsAndPositions(SegmentTermsEnum.java:1011)
at
org.apache.lucene.search.spans.SpanTermQuery.getSpans(SpanTermQuery.java:123)
at
org.apache.lucene.search.spans.SpanOrQuery$1.initSpanQueue(SpanOrQuery.java:180)
at
org.apache.lucene.search.spans.SpanOrQuery$1.next(SpanOrQuery.java:193)
at
org.apache.lucene.search.spans.SpanOrQuery$1.initSpanQueue(SpanOrQuery.java:182)
at
org.apache.lucene.search.spans.SpanOrQuery$1.next(SpanOrQuery.java:193)
at
org.apache.lucene.search.spans.NearSpansUnordered$SpansCell.next(NearSpansUnordered.java:88)
at
org.apache.lucene.search.spans.NearSpansUnordered.initList(NearSpansUnordered.java:295)
at
org.apache.lucene.search.spans.NearSpansUnordered.next(NearSpansUnordered.java:164)
at
org.apache.lucene.search.spans.SpanScorer.<init>(SpanScorer.java:46)
at
org.apache.lucene.search.spans.SpanWeight.scorer(SpanWeight.java:88)
at
org.apache.lucene.search.DisjunctionMaxQuery$DisjunctionMaxWeight.scorer(DisjunctionMaxQuery.java:160)
at
org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(BooleanQuery.java:356)
at
org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(BooleanQuery.java:356)
at
org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(BooleanQuery.java:356)
at
org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(BooleanQuery.java:356)
at
org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(BooleanQuery.java:356)
at
org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(BooleanQuery.java:356)
at
org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(BooleanQuery.java:356)
at
org.apache.lucene.queries.function.BoostedQuery$BoostedWeight.scorer(BoostedQuery.java:101)
at org.apache.lucene.search.Weight.bulkScorer(Weight.java:131)
at
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:618)
at
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:297)
at
org.apache.solr.search.grouping.CommandHandler.searchWithTimeLimiter(CommandHandler.java:219)
at
org.apache.solr.search.grouping.CommandHandler.execute(CommandHandler.java:154)
at
org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:362)
at
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:218)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1967)
at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:777)
Thanks,
Modassar
