Hi Since last 2 days we are facing extremely slow behaviour with solr slave servers. The threads are either in blocking or waiting state and require a restart of servers to keep the production running.
Following are the relevant details: There are 2 search servers in a virtualized VMware environment. Each has 2 instances of Solr running on separates ports in tomcat. Server 1: hosts 1 master(application 1), 1 slave (application 1) 6 GB RAM, JVM 2GB Server 2: hosts 1 master (application 2), 1 slave (application 1) 6 GB RAM, JVM 2GB Maximum number of threads each server can handle is 250. The number of requests is maxing out and the response time from solr is so slow that the threads are getting blocked. We have upgraded slaves to Solr 1.4 but master is still on Solr 1.3 as we do not want to play much around with the production systems. Yet the behavior is almost same. Solrj client which is used to query slave solr,gets timedout and there is high CPU usage/load avg.T he problem is reported on slaves for application 1. The SolrJ client which queries Solr over HTTP times out (10 sec is the timeout value) though in the Solr tomcat access log we find all requests have 200 response. During the tme, requests timeout the load avg. of the server goes extremely high (10-20). The issue gets resolved as soon as we optimize the slave index. In the solr admin, it shows only 4 requests/sec is handled with 400 ms response time. Blocked thread: 'http-9090-219' Id=248, BLOCKED on lock=org.apache.lucene.index.segmentreader$coreread...@1546c85, total cpu time=3930.0000ms user time=2830.0000ms at org.apache.lucene.index.SegmentReader$CoreReaders.getTermsReader(SegmentReader.java:161) at org.apache.lucene.index.SegmentTermDocs.seek(SegmentTermDocs.java:57) at org.apache.lucene.search.MultiTermQueryWrapperFilter$TermGenerator.generate(MultiTermQueryWrapperFilter.java:110) at org.apache.lucene.search.MultiTermQueryWrapperFilter.getDocIdSet(MultiTermQueryWrapperFilter.java:167) at org.apache.lucene.search.ConstantScoreQuery$ConstantScorer.<init>(ConstantScoreQuery.java:116) at org.apache.lucene.search.ConstantScoreQuery$ConstantWeight.scorer(ConstantScoreQuery.java:81) at org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(BooleanQuery.java:297) at org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(BooleanQuery.java:297) at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:246) at org.apache.lucene.search.Searcher.search(Searcher.java:171) at org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:988) at org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:884) at org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:341) at org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:182) at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:195) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316) at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128) at org.jstripe.tomcat.probe.Tomcat55AgentValve.invoke(Tomcat55AgentValve.java:20) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:567) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:849) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:454) at java.lang.Thread.run(Thread.java:595) Thanks Dipti
