Can you get a thread dump to see what is hanging? -Yonik http://www.lucidimagination.com
On Wed, Jun 29, 2011 at 11:45 AM, Bob Sandiford <bob.sandif...@sirsidynix.com> wrote: > Hi, all. > > I'm hoping someone has some thoughts here. > > We're running Solr 3.1 (with the patch for SolrQueryParser.java to not do the > getLuceneVersion() calls, but use luceneMatchVersion directly). > > We're running in a Tomcat instance, 64 bit Java. CATALINA_OPTS are: > -Xmx7168m -Xms7168m -XX:MaxPermSize=256M > > We're running 2 Solr cores, with the same schema. > > We use SolrJ to run our searches from a Java app running in JBoss. > > JBoss, Tomcat, and the Solr Index folders are all on the same server. > > In case it's relevant, we're using JMeter as a load test harness. > > We're running on Solaris, a 16 processor box with 48GB physical memory. > > I've run a successful load test at a 100 user load (at that rate there are > about 5-10 solr searches / second), and solr search responses were coming in > under 100ms. > > When I tried to ramp up, as far as I can tell, Solr is just hanging. (We > have some logging statements around the SolrJ calls - just before, we log how > long our query construction takes, then we run the SolrJ query and log the > search times. We're getting a number of the query construction logs, but no > corresponding search time logs). > > Symptoms: > The Tomcat and JBoss processes show as well under 1% CPU, and they are still > the top processes. CPU states show around 99% idle. RES usage for the two > Java processes around 3GB each. LWP under 120 for each. STATE just shows as > sleep. JBoss is still 'alive', as I can get into a piece of software that > talks to our JBoss app to get data. > > We set things up to use log4j logging for Solr - the log isn't showing any > errors or exceptions. > > We're not indexing - just searching. > > Back in January, we did load testing on a prototype, and had no problems > (though that was Solr 1.4 at the time). It ramped up beautifully - bottle > necks were our apps, not Solr. What I'm benchmarking now is a descendent of > that prototyping - a bit more complex on searches and more fields in the > schema, but same basic search logic as far as SolrJ usage. > > Any ideas? What else to look at? Ringing any bells? > > I can send more details if anyone wants specifics... > > Bob Sandiford | Lead Software Engineer | SirsiDynix > P: 800.288.8020 X6943 | bob.sandif...@sirsidynix.com > www.sirsidynix.com<http://www.sirsidynix.com/> > >
