Hi,
I am currently running into merge-issues while optimizing an index.
To give you some informations:
We are running 4 SOLR Servers with identical OS, VM-Hardware, RAM etc.
Only one Server by now is having issues, the others are fine.
We are running SOLR 4.4.0 with Tomcat 6.0
It was running since October without any problems.
The problems first occur after doing a minor change in the synonyms.txt,
but I guess that was just a coincedence.
We added `ulimit -v unlimited` to our tomcat init-script years ago.
We have 4 Cores running on each SOLR Server, configuration, index-sizes
of all 4 servers are identical (we are distributing cfgs via git).
We did a rebuild of the index twice: First time without removing the old
index files, second time deleting the data dir and starting from scratch.
We are working with DIH, getting data from a MySQL DB.
After an initial complete index-run, the optimize is working. The
optimize fails one or two days later.
We are doing one optimize-run a day, the index contains ~10 millions
documents, the index size on disc is ~39GB while having 127G of free
disc space.
We have a mergeFactor of 3.
The solr.log says:
ERROR - 2014-01-12 22:47:11.062; org.apache.solr.common.SolrException;
java.io.IOException: background merge hit exception:
_dc8(4.4):C9876366/1327 _e8u(4.4):C4250/7 _f4a(4.4):C1553/13 _fj6(4.4
):C1903/15 _ep3(4.4):C1217/42 _fle(4.4):C256/7 _flf(4.4):C11 into _flg
[maxNumSegments=1]
at
org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1714)
at
org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1650)
at
org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:530)
at
org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:95)
at
org.apache.solr.update.processor.UpdateRequestProcessor.processCommit(UpdateRequestProcessor.java:64)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalCommit(DistributedUpdateProcessor.java:1235)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.processCommit(DistributedUpdateProcessor.java:1219)
at
org.apache.solr.update.processor.LogUpdateProcessor.processCommit(LogUpdateProcessorFactory.java:157)
at
org.apache.solr.handler.RequestHandlerUtils.handleCommit(RequestHandlerUtils.java:69)
at
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1904)
at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:659)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:362)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:158)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:857)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
at
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
at java.lang.Thread.run(Thread.java:735)
Caused by: java.lang.NullPointerException
at java.nio.ByteBuffer.get(ByteBuffer.java:661)
at java.nio.DirectByteBuffer.get(DirectByteBuffer.java:245)
at
org.apache.lucene.store.ByteBufferIndexInput.readBytes(ByteBufferIndexInput.java:107)
at
org.apache.lucene.codecs.lucene41.ForUtil.readBlock(ForUtil.java:197)
at
org.apache.lucene.codecs.lucene41.Lucene41PostingsReader$BlockDocsAndPositionsEnum.refillDocs(Lucene41PostingsReader.java:748)
at
org.apache.lucene.codecs.lucene41.Lucene41PostingsReader$BlockDocsAndPositionsEnum.nextDoc(Lucene41PostingsReader.java:813)
at
org.apache.lucene.codecs.MappingMultiDocsAndPositionsEnum.nextDoc(MappingMultiDocsAndPositionsEnum.java:104)
at
org.apache.lucene.codecs.PostingsConsumer.merge(PostingsConsumer.java:101)
at
org.apache.lucene.codecs.TermsConsumer.merge(TermsConsumer.java:164)
at
org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:72)
at
org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:365)
at
org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:98)
at
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3772)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3376)
at
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:405)
at
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:482)
I am currently running a checkIndex on that core/index - but that will
run for hours, but I just want to kick that thread off to get any
suggestions from you guys.
Any suggestions?
Thank you and best regards
Ralf
