Here is some more information about my setup,
Solr - v1.4 (nightly build 03/29/09)
Servlet Container - Tomcat 6.0.18
JVM - 1.6.0 (64 bit)
OS - Mac OS X Server 10.5.6
Hardware Overview:
Processor Name: Quad-Core Intel Xeon
Processor Speed: 3 GHz
Number Of Processors: 2
Total Number Of Cores: 8
L2 Cache (per processor): 12 MB
Memory: 20 GB
Bus Speed: 1.6 GHz
JVM Parameters (for Solr):
export CATALINA_OPTS="-server -Xms6044m -Xmx6044m -DSOLR_APP
-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Xloggc:gc.log
-Dsun.rmi.dgc.client.gcInterval=3600000
-Dsun.rmi.dgc.server.gcInterval=3600000"
Other:
lsof|grep solr|wc -l
2493
ulimit -an
open files (-n) 9000
Tomcat
<Connector port="8080" protocol="HTTP/1.1"
connectionTimeout="20000"
maxThreads="100" />
Total Solr cores on same instance - 65
useCompoundFile - true
The tests I ran,
While Indexer is running
1) Go to "http://juum19.co.com:8080/solr"; - returns blank page (no
error in the catalina.out)
2) Try "telnet juum19.co.com 8080" - returns with "Connection closed
by foreign host"
Stop the Indexer Program (Tomcat is still running with Solr)
3) Go to "http://juum19.co.com:8080/solr"; - works ok, shows the list
of all the Solr cores
4) Try telnet - able to Telnet fine
5) Now comment out all the caches in solrconfig.xml. Try same tests,
but the Tomcat still doesn't response.
Is there a way to stop the auto-warmer. I commented out the caches in
the solrconfig.xml but still see the following log,
INFO: autowarming result for searc...@3aba3830 main
fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
INFO: Closing searc...@175dc1e2
main
fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
filterCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
6) Change the Indexer frequency so it runs every 2 min (instead of all
the time). I noticed once the commit is done, I'm able to run my
searches. During commit and auto-warming period I just get blank page.
7) Changed from Solrj to XML update - I still get the blank page
whenever update/commit is happening.
Apr 13, 2009 6:46:18 PM
org.apache.solr.update.processor.LogUpdateProcessor finish
INFO: {add=[621094001, 621094002, 621094003, 621094004, 621094005,
621094006, 621094007, 621094008, ...(6992 more)]} 0 1948
Apr 13, 2009 6:46:18 PM org.apache.solr.core.SolrCore execute
INFO: [20090413_12] webapp=/solr path=/update params={} status=0 QTime=1948
So, looks like it's not just StreamingUpdateSolrServer, but whenever
the update/commit is happening I'm not able to search. I don't know if
it's related to using multi-core. In this test I was using only single
thread for update to a single core using only single Solr instance.
So, it's clearly related to index process (update, commit and
auto-warming). As soon as update/commit/auto-warming is completed I'm
able to run my queries again. Is there anything that could stop
searching while update process is in-progress - like any lock or
something?
Any other ideas?
Thanks,
-vivek
On Mon, Apr 13, 2009 at 12:14 AM, Shalin Shekhar Mangar
<shalinman...@gmail.com> wrote:
> On Mon, Apr 13, 2009 at 12:36 PM, vivek sar <vivex...@gmail.com> wrote:
>
>> I index in 10K batches and commit after 5 index cyles (after 50K). Is
>> there any limitation that I can't search during commit or
>> auto-warming? I got 8 CPU cores and only 2 were showing busy (using
>> top) - so it's unlikely that the CPU was pegged.
>>
>>
> No, there is no such limitation. The old searcher will continue to serve
> search requests until the new one is warmed and registered.
>
> So, CPU does not seem to be an issue. Does this happen only when you use
> StreamingUpdateSolrServer? Which OS, file system? What JVM parameters are
> you using? Which servlet container and version?
>
> --
> Regards,
> Shalin Shekhar Mangar.
>
