Hi Erick,
Thanks so much for your reply. I'll now look mostly into any possible
hardware issues than Solr/Lucene.
Thanks again.
On Mon, Sep 24, 2018 at 12:43 PM Erick Erickson <erickerick...@gmail.com>
wrote:
> There are several of reasons this would "suddenly" start appearing.
> 1> Your disk went bad and some sector is no longer faithfully
> recording the bits. In this case the checksum will be wrong
> 2> You ran out of disk space sometime and the index was corrupted.
> This isn't really a hardware problem.
> 3> Your disk controller is going wonky and not reading reliably.
>
> The "possible hardware issue" message is to alert you that this is
> highly unusual and you should at leasts consider doing integrity
> checks on your disk before assuming it's a Solr/Lucene problem
>
> Best,
> Erick
> On Mon, Sep 24, 2018 at 9:26 AM Susheel Kumar <susheel2...@gmail.com>
> wrote:
> >
> > Hello,
> >
> > I am still trying to understand the corrupt index exception we saw in our
> > logs. What does the hardware problem comment indicates here? Does that
> > mean it caused most likely due to hardware issue?
> >
> > We never had this problem in last couple of months. The Solr is 6.6.2 and
> > ZK: 3.4.10.
> >
> > Please share your thoughts.
> >
> > Thanks,
> > Susheel
> >
> > Caused by: org.apache.lucene.index.CorruptIndexException: checksum
> > failed *(hardware
> > problem?)* : expected=db243d1a actual=7a00d3d2
> >
> (resource=BufferedChecksumIndexInput(MMapIndexInput(path="/app/solr/data/COLL_shard1_replica1/data/index/_i27s.cfs")
> > [slice=_i27s_Lucene50_0.tim])
> >
> > It suddenly started in the logs and before which there was no such error.
> > Searches & ingestions all seems to be working prior to that.
> >
> > ----
> >
> > 2018-09-03 17:16:49.056 INFO (qtp834133664-519872) [c:COLL s:shard1
> > r:core_node1 x:COLL_shard1_replica1]
> > o.a.s.u.p.StatelessScriptUpdateProcessorFactory update-script#processAdd:
> > newid=G31MXMRZESC0CYPR!A-G31MXMRZESC0CYPR.2552019802_1-2552008480_1-en_US
> > 2018-09-03 17:16:49.057 ERROR (qtp834133664-519872) [c:COLL s:shard1
> > r:core_node1 x:COLL_shard1_replica1] o.a.s.h.RequestHandlerBase
> > org.apache.solr.common.SolrException: Exception writing document id
> > G31MXMRZESC0CYPR!A-G31MXMRZESC0CYPR.2552019802_1-2552008480_1-en_US to
> the
> > index; possible analysis error.
> > at
> >
> org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:206)
> > at
> >
> org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:67)
> > at
> >
> org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55)
> > at
> >
> org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:979)
> > at
> >
> org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1192)
> > at
> >
> org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:748)
> > at
> >
> org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55)
> > at
> >
> org.apache.solr.update.processor.StatelessScriptUpdateProcessorFactory$ScriptUpdateProcessor.processAdd(StatelessScriptUpdateProcessorFactory.java:380)
> > at
> >
> org.apache.solr.handler.loader.JavabinLoader$1.update(JavabinLoader.java:98)
> > at
> >
> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readOuterMostDocIterator(JavaBinUpdateRequestCodec.java:180)
> > at
> >
> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readIterator(JavaBinUpdateRequestCodec.java:136)
> > at
> >
> org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:306)
> > at
> org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:251)
> > at
> >
> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readNamedList(JavaBinUpdateRequestCodec.java:122)
> > at
> >
> org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:271)
> > at
> org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:251)
> > at
> org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:173)
> > at
> >
> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.unmarshal(JavaBinUpdateRequestCodec.java:187)
> > at
> >
> org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoader.java:108)
> > at
> org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoader.java:55)
> > at
> >
> org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:97)
> > at
> >
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)
> > at
> >
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:173)
> > at org.apache.solr.core.SolrCore.execute(SolrCore.java:2477)
> > at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:723)
> > at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:529)
> > at
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361)
> > at
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305)
> > at
> >
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
> > at
> >
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
> > at
> >
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> > at
> >
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> > at
> >
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
> > at
> >
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
> > at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
> > at
> >
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> > at
> >
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
> > at
> >
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> > at
> >
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
> > at
> >
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
> > at
> >
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> > at
> >
> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
> > at
> >
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> > at org.eclipse.jetty.server.Server.handle(Server.java:534)
> > at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
> > at
> >
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
> > at
> > org.eclipse.jetty.io
> .AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
> > at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
> > at
> > org.eclipse.jetty.io
> .SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
> > at
> >
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
> > at
> >
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
> > at
> >
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
> > at
> >
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
> > at
> >
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
> > at java.lang.Thread.run(Thread.java:748)
> > Caused by: org.apache.lucene.store.AlreadyClosedException: this
> IndexWriter
> > is closed
> > at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:749)
> > at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:763)
> > at
> org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1567)
> > at
> >
> org.apache.solr.update.DirectUpdateHandler2.updateDocument(DirectUpdateHandler2.java:924)
> > at
> >
> org.apache.solr.update.DirectUpdateHandler2.updateDocOrDocValues(DirectUpdateHandler2.java:913)
> > at
> >
> org.apache.solr.update.DirectUpdateHandler2.doNormalUpdate(DirectUpdateHandler2.java:302)
> > at
> >
> org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:239)
> > at
> >
> org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:194)
> > ... 54 more
> > Caused by: org.apache.lucene.index.CorruptIndexException: checksum failed
> > (hardware problem?) : expected=db243d1a actual=7a00d3d2
> >
> (resource=BufferedChecksumIndexInput(MMapIndexInput(path="/app/solr/data/COLL_shard1_replica1/data/index/_i27s.cfs")
> > [slice=_i27s_Lucene50_0.tim]))
> > at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:419)
> > at
> org.apache.lucene.codecs.CodecUtil.checksumEntireFile(CodecUtil.java:526)
> > at
> >
> org.apache.lucene.codecs.blocktree.BlockTreeTermsReader.checkIntegrity(BlockTreeTermsReader.java:336)
> > at
> >
> org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.checkIntegrity(PerFieldPostingsFormat.java:348)
> > at
> >
> org.apache.lucene.codecs.perfield.PerFieldMergeState$FilterFieldsProducer.checkIntegrity(PerFieldMergeState.java:271)
> > at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:96)
> > at
> >
> org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.merge(PerFieldPostingsFormat.java:164)
> > at
> org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:216)
> > at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:101)
> > at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4356)
> > at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3931)
> > at org.apache.solr.update.SolrIndexWriter.merge(SolrIndexWriter.java:188)
> > at
> >
> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:624)
> > at
> >
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:661)
> >
> > 2018-09-03 17:16:49.116 INFO (qtp834133664-519872) [c:COLL s:shard1
> > r:core_node1 x:COLL_shard1_replica1] o.a.s.c.S.Request
> > [COLL_shard1_replica1] webapp=/solr path=/update
> > params={wt=javabin&version=2} status=400 QTime=69
>
