I am using SOLR 4.0 , and using DIH to import and index the catalog data from
my MySQL database. The DIH took around 55minutes to complete the indexing,
but my max documents processed and actual documents which got indexed were
not matching. So when I checked the update handler statistics, it's showing
me roughly six hundred thousand documents as pending. Here is the update
handler stats
commits: 2130
autocommit maxTime: 15000ms
autocommits: 169
soft autocommit maxTime: 1000ms
soft autocommits: 1958
optimizes: 0
rollbacks: 0
expungeDeletes: 0
docsPending: 622009
adds: 0
deletesById: 0
deletesByQuery: 0
errors: 12
cumulative_adds: 12191849
cumulative_deletesById: 0
cumulative_deletesByQuery: 1
cumulative_errors: 0
Now when I have tried to commit it manually, i.e. /update?commit=true , it's
was throwing an out of memory error. It says writer hit out of memory error,
so cannot commit. Here is the stack
java.lang.IllegalStateException: this writer hit an OutOfMemoryError; cannot
commit
at
org.apache.lucene.index.IndexWriter.prepareCommit(IndexWriter.java:2717)
at
org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2875)
at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2855)
at
org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:531)
at
org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:87)
at
org.apache.solr.update.processor.UpdateRequestProcessor.processCommit(UpdateRequestProcessor.java:64)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.processCommit(DistributedUpdateProcessor.java:1007)
at
org.apache.solr.update.processor.LogUpdateProcessor.processCommit(LogUpdateProcessorFactory.java:157)
at
org.apache.solr.handler.RequestHandlerUtils.handleCommit(RequestHandlerUtils.java:69)
at
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1699)
at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:455)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:276)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:861)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:606)
at
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
at java.lang.Thread.run(Thread.java:662)
It will be really great help if you can tell me a way to commit those
pending documents. Shall I restart the tomcat ? Any help will be much
appreciated.
--
View this message in context:
http://lucene.472066.n3.nabble.com/Full-import-through-DIH-leaving-documents-as-uncommited-tp4035084.html
Sent from the Solr - User mailing list archive at Nabble.com.
