On Tue, Apr 14, 2009 at 7:14 AM, vivek sar <vivex...@gmail.com> wrote: > Some more update. As I mentioned earlier we are using multi-core Solr > (up to 65 cores in one Solr instance with each core 10G). This was > opening around 3000 file descriptors (lsof). I removed some cores and > after some trial and error I found at 25 cores system seems to work > fine (around 1400 file descriptors). Tomcat is responsive even when > the indexing is happening at Solr (for 25 cores). But, as soon as it > goes to 26 cores the Tomcat becomes unresponsive again. The puzzling > thing is if I stop indexing I can search on even 65 cores, but while > indexing is happening it seems to support only up to 25 cores. > > 1) Is there a limit on number of cores a Solr instance can handle? > 2) Does Solr do anything to the existing cores while indexing? I'm > writing to only one core at a time. There is no hard limit (it is Integer.MAX_VALUE) . But inreality your mileage depends on your hardware and no:of file handles the OS can open > > We are struggling to find why Tomcat stops responding on high number > of cores while indexing is in-progress. Any help is very much > appreciated. > > Thanks, > -vivek > > On Mon, Apr 13, 2009 at 10:52 AM, vivek sar <vivex...@gmail.com> wrote: >> Here is some more information about my setup, >> >> Solr - v1.4 (nightly build 03/29/09) >> Servlet Container - Tomcat 6.0.18 >> JVM - 1.6.0 (64 bit) >> OS - Mac OS X Server 10.5.6 >> >> Hardware Overview: >> >> Processor Name: Quad-Core Intel Xeon >> Processor Speed: 3 GHz >> Number Of Processors: 2 >> Total Number Of Cores: 8 >> L2 Cache (per processor): 12 MB >> Memory: 20 GB >> Bus Speed: 1.6 GHz >> >> JVM Parameters (for Solr): >> >> export CATALINA_OPTS="-server -Xms6044m -Xmx6044m -DSOLR_APP >> -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Xloggc:gc.log >> -Dsun.rmi.dgc.client.gcInterval=3600000 >> -Dsun.rmi.dgc.server.gcInterval=3600000" >> >> Other: >> >> lsof|grep solr|wc -l >> 2493 >> >> ulimit -an >> open files (-n) 9000 >> >> Tomcat >> <Connector port="8080" protocol="HTTP/1.1" >> connectionTimeout="20000" >> maxThreads="100" /> >> >> Total Solr cores on same instance - 65 >> >> useCompoundFile - true >> >> The tests I ran, >> >> While Indexer is running >> 1) Go to "http://juum19.co.com:8080/solr"; - returns blank page (no >> error in the catalina.out) >> >> 2) Try "telnet juum19.co.com 8080" - returns with "Connection closed >> by foreign host" >> >> Stop the Indexer Program (Tomcat is still running with Solr) >> >> 3) Go to "http://juum19.co.com:8080/solr"; - works ok, shows the list >> of all the Solr cores >> >> 4) Try telnet - able to Telnet fine >> >> 5) Now comment out all the caches in solrconfig.xml. Try same tests, >> but the Tomcat still doesn't response. >> >> Is there a way to stop the auto-warmer. I commented out the caches in >> the solrconfig.xml but still see the following log, >> >> INFO: autowarming result for searc...@3aba3830 main >> fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0} >> >> INFO: Closing searc...@175dc1e2 >> main >> fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0} >> filterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0} >> queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0} >> documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0} >> >> >> 6) Change the Indexer frequency so it runs every 2 min (instead of all >> the time). I noticed once the commit is done, I'm able to run my >> searches. During commit and auto-warming period I just get blank page. >> >> 7) Changed from Solrj to XML update - I still get the blank page >> whenever update/commit is happening. >> >> Apr 13, 2009 6:46:18 PM >> org.apache.solr.update.processor.LogUpdateProcessor finish >> INFO: {add=[621094001, 621094002, 621094003, 621094004, 621094005, >> 621094006, 621094007, 621094008, ...(6992 more)]} 0 1948 >> Apr 13, 2009 6:46:18 PM org.apache.solr.core.SolrCore execute >> INFO: [20090413_12] webapp=/solr path=/update params={} status=0 QTime=1948 >> >> >> So, looks like it's not just StreamingUpdateSolrServer, but whenever >> the update/commit is happening I'm not able to search. I don't know if >> it's related to using multi-core. In this test I was using only single >> thread for update to a single core using only single Solr instance. >> >> So, it's clearly related to index process (update, commit and >> auto-warming). As soon as update/commit/auto-warming is completed I'm >> able to run my queries again. Is there anything that could stop >> searching while update process is in-progress - like any lock or >> something? >> >> Any other ideas? >> >> Thanks, >> -vivek >> >> On Mon, Apr 13, 2009 at 12:14 AM, Shalin Shekhar Mangar >> <shalinman...@gmail.com> wrote: >>> On Mon, Apr 13, 2009 at 12:36 PM, vivek sar <vivex...@gmail.com> wrote: >>> >>>> I index in 10K batches and commit after 5 index cyles (after 50K). Is >>>> there any limitation that I can't search during commit or >>>> auto-warming? I got 8 CPU cores and only 2 were showing busy (using >>>> top) - so it's unlikely that the CPU was pegged. >>>> >>>> >>> No, there is no such limitation. The old searcher will continue to serve >>> search requests until the new one is warmed and registered. >>> >>> So, CPU does not seem to be an issue. Does this happen only when you use >>> StreamingUpdateSolrServer? Which OS, file system? What JVM parameters are >>> you using? Which servlet container and version? >>> >>> -- >>> Regards, >>> Shalin Shekhar Mangar. >>> >> >
-- --Noble Paul
