Naveen:
You should try NRT with Apache Solr 3.3 and RankingAlgorithm. You can
update 10,000 documents / sec while also concurrently searching. You can
set commit freq to about 15 mins or as desired. The 10,000 document
update performance is with the MBArtists index on a dual core Linux
system. So you may be able to see similar performance on your system.
You can get more details of the NRT implementation from here:
http://solr-ra.tgels.org/wiki/en/Near_Real_Time_Search_ver_3.x
You can download Apache Solr 3.3 with RankingAlgorithm from here:
http://solr-ra.tgels.org/
(There are no changes to your existing setup, everything should work as
earlier except for adding the <realtime> tag to your solrconfig.xml)
Regards
- Nagendra Nagarajayya
http://solr-ra.tgels.org
http://rankingalgorithm.tgels.org
On 8/13/2011 6:47 PM, Naveen Gupta wrote:
Hi,
Most of the settings are default.
We have single node (Memory 1 GB, Index Size 4GB)
We have a requirement where we are doing very fast commit. This is kind of
real time requirement where we are polling many threads from third party and
indexes into our system.
We want these results to be available soon.
We are committing for each user (may have 10k threads and inside that 1
thread may have 10 messages). So overall documents per user will be having
around .1 million (100000)
Earlier we were using commit Within as 10 milliseconds inside the document,
but that was slowing the indexing and we were not getting any error.
As we removed the commit Within, indexing became very fast. But after that
we started experiencing in the system
As i read many forums, everybody told that this is happening because of very
fast commit rate, but what is the solution for our problem?
We are using CURL to post the data and commit
Also till now we are using default solrconfig.
Aug 14, 2011 12:12:04 AM org.apache.solr.common.SolrException log
SEVERE: org.apache.solr.common.SolrException: Error opening new searcher.
exceeded limit of maxWarmingSearchers=2, try again later.
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1052)
at
org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:424)
at
org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:85)
at
org.apache.solr.handler.XMLLoader.processUpdate(XMLLoader.java:177)
at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:77)
at
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:55)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1360)
at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:356)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:252)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
at
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
at java.lang.Thread.run(Thread.java:662)
