Which version of Java are you using? That root cause exception is somewhat spooky: it's in the ByteBufferIndexCode that handles an "UnderflowException", ie when a small (maybe a few hundred bytes) read happens to span the 1 GB page boundary, and specifically the exception happens on the final read (curBuf.get(b, offset, len)). Such page-spanning reads are very rare.
The code looks fine to me though, and it's hard to explain how NPE (b = null) could happen: that byte array is allocated in the Lucene41PostingsReader.BlockDocsAndPositionsEnum class's ctor: encoded = new byte[MAX_ENCODED_SIZE]. Separately, you really should not have to optimize daily, if ever. Mike McCandless http://blog.mikemccandless.com On Mon, Jan 13, 2014 at 2:06 AM, Ralf Matulat <ralf.matu...@bundestag.de> wrote: > Hi, > I am currently running into merge-issues while optimizing an index. > > To give you some informations: > > We are running 4 SOLR Servers with identical OS, VM-Hardware, RAM etc. > Only one Server by now is having issues, the others are fine. > > We are running SOLR 4.4.0 with Tomcat 6.0 > It was running since October without any problems. > The problems first occur after doing a minor change in the synonyms.txt, but > I guess that was just a coincedence. > > We added `ulimit -v unlimited` to our tomcat init-script years ago. > > We have 4 Cores running on each SOLR Server, configuration, index-sizes of > all 4 servers are identical (we are distributing cfgs via git). > > We did a rebuild of the index twice: First time without removing the old > index files, second time deleting the data dir and starting from scratch. > > We are working with DIH, getting data from a MySQL DB. > After an initial complete index-run, the optimize is working. The optimize > fails one or two days later. > > We are doing one optimize-run a day, the index contains ~10 millions > documents, the index size on disc is ~39GB while having 127G of free disc > space. > > We have a mergeFactor of 3. > > The solr.log says: > > ERROR - 2014-01-12 22:47:11.062; org.apache.solr.common.SolrException; > java.io.IOException: background merge hit exception: _dc8(4.4):C9876366/1327 > _e8u(4.4):C4250/7 _f4a(4.4):C1553/13 _fj6(4.4 > ):C1903/15 _ep3(4.4):C1217/42 _fle(4.4):C256/7 _flf(4.4):C11 into _flg > [maxNumSegments=1] > at > org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1714) > at > org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1650) > at > org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:530) > at > org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:95) > at > org.apache.solr.update.processor.UpdateRequestProcessor.processCommit(UpdateRequestProcessor.java:64) > at > org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalCommit(DistributedUpdateProcessor.java:1235) > at > org.apache.solr.update.processor.DistributedUpdateProcessor.processCommit(DistributedUpdateProcessor.java:1219) > at > org.apache.solr.update.processor.LogUpdateProcessor.processCommit(LogUpdateProcessorFactory.java:157) > at > org.apache.solr.handler.RequestHandlerUtils.handleCommit(RequestHandlerUtils.java:69) > at > org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68) > at > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135) > at org.apache.solr.core.SolrCore.execute(SolrCore.java:1904) > at > org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:659) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:362) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:158) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) > at > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) > at > org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) > at > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) > at > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) > at > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) > at > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298) > at > org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:857) > at > org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588) > at > org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489) > at java.lang.Thread.run(Thread.java:735) > Caused by: java.lang.NullPointerException > at java.nio.ByteBuffer.get(ByteBuffer.java:661) > at java.nio.DirectByteBuffer.get(DirectByteBuffer.java:245) > at > org.apache.lucene.store.ByteBufferIndexInput.readBytes(ByteBufferIndexInput.java:107) > at > org.apache.lucene.codecs.lucene41.ForUtil.readBlock(ForUtil.java:197) > at > org.apache.lucene.codecs.lucene41.Lucene41PostingsReader$BlockDocsAndPositionsEnum.refillDocs(Lucene41PostingsReader.java:748) > at > org.apache.lucene.codecs.lucene41.Lucene41PostingsReader$BlockDocsAndPositionsEnum.nextDoc(Lucene41PostingsReader.java:813) > at > org.apache.lucene.codecs.MappingMultiDocsAndPositionsEnum.nextDoc(MappingMultiDocsAndPositionsEnum.java:104) > at > org.apache.lucene.codecs.PostingsConsumer.merge(PostingsConsumer.java:101) > at > org.apache.lucene.codecs.TermsConsumer.merge(TermsConsumer.java:164) > at > org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:72) > at > org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:365) > at > org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:98) > at > org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3772) > at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3376) > at > org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:405) > at > org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:482) > > I am currently running a checkIndex on that core/index - but that will run > for hours, but I just want to kick that thread off to get any suggestions > from you guys. > > Any suggestions? > Thank you and best regards > Ralf
