The Solr instance can only handle so many queries per second. It should be possible to configure the web app container to limit the total number of active threads.
Also, have you watched the garbage collection statistics with 'jconsole' or another interactive tool? You might discover that it is spending all of its time doing garbage collection. On Thu, Mar 25, 2010 at 10:13 AM, dipti khullar <dipti.khul...@gmail.com> wrote: > Hi > > Since last 2 days we are facing extremely slow behaviour with solr slave > servers. The threads are either in blocking or waiting state and require a > restart of servers to keep the production running. > > Following are the relevant details: > > There are 2 search servers in a virtualized VMware environment. Each has 2 > instances of Solr running on separates ports in tomcat. > Server 1: hosts 1 master(application 1), 1 slave (application 1) 6 GB RAM, > JVM 2GB > Server 2: hosts 1 master (application 2), 1 slave (application 1) 6 GB RAM, > JVM 2GB > > Maximum number of threads each server can handle is 250. The number of > requests is maxing out and the response time from solr is so slow that the > threads are getting blocked. > > We have upgraded slaves to Solr 1.4 but master is still on Solr 1.3 as we do > not want to play much around with the production systems. Yet the behavior > is almost same. > > Solrj client which is used to query slave solr,gets timedout and there is > high CPU usage/load avg.T he problem is reported on slaves for application > 1. The SolrJ client which queries Solr over HTTP times out (10 sec is the > timeout value) though in the Solr tomcat access log we find all requests > have 200 response. > During the tme, requests timeout the load avg. of the server goes extremely > high (10-20). > The issue gets resolved as soon as we optimize the slave index. In the solr > admin, it shows only 4 requests/sec is handled with 400 ms response time. > > Blocked thread: > > 'http-9090-219' Id=248, BLOCKED on > lock=org.apache.lucene.index.segmentreader$coreread...@1546c85, total cpu > time=3930.0000ms user time=2830.0000ms > at > org.apache.lucene.index.SegmentReader$CoreReaders.getTermsReader(SegmentReader.java:161) > > at org.apache.lucene.index.SegmentTermDocs.seek(SegmentTermDocs.java:57) > at > org.apache.lucene.search.MultiTermQueryWrapperFilter$TermGenerator.generate(MultiTermQueryWrapperFilter.java:110) > > at > org.apache.lucene.search.MultiTermQueryWrapperFilter.getDocIdSet(MultiTermQueryWrapperFilter.java:167) > > at > org.apache.lucene.search.ConstantScoreQuery$ConstantScorer.<init>(ConstantScoreQuery.java:116) > > at > org.apache.lucene.search.ConstantScoreQuery$ConstantWeight.scorer(ConstantScoreQuery.java:81) > > at > org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(BooleanQuery.java:297) > > at > org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(BooleanQuery.java:297) > > at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:246) > at org.apache.lucene.search.Searcher.search(Searcher.java:171) > at > org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:988) > > at > org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:884) > > at > org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:341) > at > org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:182) > > at > org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:195) > > at > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131) > > at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316) > at > org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338) > > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241) > > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) > > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) > > at > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) > > at > org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) > > at > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128) > > at > org.jstripe.tomcat.probe.Tomcat55AgentValve.invoke(Tomcat55AgentValve.java:20) > > at > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) > > at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:567) > > at > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) > > at > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293) > at > org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:849) > at > org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583) > > at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:454) > at java.lang.Thread.run(Thread.java:595) > > Thanks > Dipti > -- Lance Norskog goks...@gmail.com
