There are several of reasons this would "suddenly" start appearing.
1> Your disk went bad and some sector is no longer faithfully
recording the bits. In this case the checksum will be wrong
2> You ran out of disk space sometime and the index was corrupted.
This isn't really a hardware problem.
3> Your disk controller is going wonky and not reading reliably.
The "possible hardware issue" message is to alert you that this is
highly unusual and you should at leasts consider doing integrity
checks on your disk before assuming it's a Solr/Lucene problem
Best,
Erick
On Mon, Sep 24, 2018 at 9:26 AM Susheel Kumar <susheel2...@gmail.com> wrote:
>
> Hello,
>
> I am still trying to understand the corrupt index exception we saw in our
> logs. What does the hardware problem comment indicates here? Does that
> mean it caused most likely due to hardware issue?
>
> We never had this problem in last couple of months. The Solr is 6.6.2 and
> ZK: 3.4.10.
>
> Please share your thoughts.
>
> Thanks,
> Susheel
>
> Caused by: org.apache.lucene.index.CorruptIndexException: checksum
> failed *(hardware
> problem?)* : expected=db243d1a actual=7a00d3d2
> (resource=BufferedChecksumIndexInput(MMapIndexInput(path="/app/solr/data/COLL_shard1_replica1/data/index/_i27s.cfs")
> [slice=_i27s_Lucene50_0.tim])
>
> It suddenly started in the logs and before which there was no such error.
> Searches & ingestions all seems to be working prior to that.
>
> ----
>
> 2018-09-03 17:16:49.056 INFO (qtp834133664-519872) [c:COLL s:shard1
> r:core_node1 x:COLL_shard1_replica1]
> o.a.s.u.p.StatelessScriptUpdateProcessorFactory update-script#processAdd:
> newid=G31MXMRZESC0CYPR!A-G31MXMRZESC0CYPR.2552019802_1-2552008480_1-en_US
> 2018-09-03 17:16:49.057 ERROR (qtp834133664-519872) [c:COLL s:shard1
> r:core_node1 x:COLL_shard1_replica1] o.a.s.h.RequestHandlerBase
> org.apache.solr.common.SolrException: Exception writing document id
> G31MXMRZESC0CYPR!A-G31MXMRZESC0CYPR.2552019802_1-2552008480_1-en_US to the
> index; possible analysis error.
> at
> org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:206)
> at
> org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:67)
> at
> org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55)
> at
> org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:979)
> at
> org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1192)
> at
> org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:748)
> at
> org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55)
> at
> org.apache.solr.update.processor.StatelessScriptUpdateProcessorFactory$ScriptUpdateProcessor.processAdd(StatelessScriptUpdateProcessorFactory.java:380)
> at
> org.apache.solr.handler.loader.JavabinLoader$1.update(JavabinLoader.java:98)
> at
> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readOuterMostDocIterator(JavaBinUpdateRequestCodec.java:180)
> at
> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readIterator(JavaBinUpdateRequestCodec.java:136)
> at
> org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:306)
> at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:251)
> at
> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readNamedList(JavaBinUpdateRequestCodec.java:122)
> at
> org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:271)
> at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:251)
> at org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:173)
> at
> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.unmarshal(JavaBinUpdateRequestCodec.java:187)
> at
> org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoader.java:108)
> at org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoader.java:55)
> at
> org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:97)
> at
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:173)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2477)
> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:723)
> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:529)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> at
> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
> at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> at org.eclipse.jetty.server.Server.handle(Server.java:534)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
> at
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
> at
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
> at
> org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
> at
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
> at
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.lucene.store.AlreadyClosedException: this IndexWriter
> is closed
> at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:749)
> at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:763)
> at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1567)
> at
> org.apache.solr.update.DirectUpdateHandler2.updateDocument(DirectUpdateHandler2.java:924)
> at
> org.apache.solr.update.DirectUpdateHandler2.updateDocOrDocValues(DirectUpdateHandler2.java:913)
> at
> org.apache.solr.update.DirectUpdateHandler2.doNormalUpdate(DirectUpdateHandler2.java:302)
> at
> org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:239)
> at
> org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:194)
> ... 54 more
> Caused by: org.apache.lucene.index.CorruptIndexException: checksum failed
> (hardware problem?) : expected=db243d1a actual=7a00d3d2
> (resource=BufferedChecksumIndexInput(MMapIndexInput(path="/app/solr/data/COLL_shard1_replica1/data/index/_i27s.cfs")
> [slice=_i27s_Lucene50_0.tim]))
> at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:419)
> at org.apache.lucene.codecs.CodecUtil.checksumEntireFile(CodecUtil.java:526)
> at
> org.apache.lucene.codecs.blocktree.BlockTreeTermsReader.checkIntegrity(BlockTreeTermsReader.java:336)
> at
> org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.checkIntegrity(PerFieldPostingsFormat.java:348)
> at
> org.apache.lucene.codecs.perfield.PerFieldMergeState$FilterFieldsProducer.checkIntegrity(PerFieldMergeState.java:271)
> at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:96)
> at
> org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.merge(PerFieldPostingsFormat.java:164)
> at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:216)
> at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:101)
> at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4356)
> at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3931)
> at org.apache.solr.update.SolrIndexWriter.merge(SolrIndexWriter.java:188)
> at
> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:624)
> at
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:661)
>
> 2018-09-03 17:16:49.116 INFO (qtp834133664-519872) [c:COLL s:shard1
> r:core_node1 x:COLL_shard1_replica1] o.a.s.c.S.Request
> [COLL_shard1_replica1] webapp=/solr path=/update
> params={wt=javabin&version=2} status=400 QTime=69
