Got it. I'll have first hardware folks check and if they don't see/find
anything suspicious then i'll return here.
Wondering if any body has seen similar error and if they were able to
confirm if it was hardware fault or so.
Thnx
On Mon, Sep 24, 2018 at 1:01 PM Erick Erickson <erickerick...@gmail.com>
wrote:
> Mind you it could _still_ be Solr/Lucene, but let's check the hardware
> first ;)
> On Mon, Sep 24, 2018 at 9:50 AM Susheel Kumar <susheel2...@gmail.com>
> wrote:
> >
> > Hi Erick,
> >
> > Thanks so much for your reply. I'll now look mostly into any possible
> > hardware issues than Solr/Lucene.
> >
> > Thanks again.
> >
> > On Mon, Sep 24, 2018 at 12:43 PM Erick Erickson <erickerick...@gmail.com
> >
> > wrote:
> >
> > > There are several of reasons this would "suddenly" start appearing.
> > > 1> Your disk went bad and some sector is no longer faithfully
> > > recording the bits. In this case the checksum will be wrong
> > > 2> You ran out of disk space sometime and the index was corrupted.
> > > This isn't really a hardware problem.
> > > 3> Your disk controller is going wonky and not reading reliably.
> > >
> > > The "possible hardware issue" message is to alert you that this is
> > > highly unusual and you should at leasts consider doing integrity
> > > checks on your disk before assuming it's a Solr/Lucene problem
> > >
> > > Best,
> > > Erick
> > > On Mon, Sep 24, 2018 at 9:26 AM Susheel Kumar <susheel2...@gmail.com>
> > > wrote:
> > > >
> > > > Hello,
> > > >
> > > > I am still trying to understand the corrupt index exception we saw
> in our
> > > > logs. What does the hardware problem comment indicates here? Does
> that
> > > > mean it caused most likely due to hardware issue?
> > > >
> > > > We never had this problem in last couple of months. The Solr is
> 6.6.2 and
> > > > ZK: 3.4.10.
> > > >
> > > > Please share your thoughts.
> > > >
> > > > Thanks,
> > > > Susheel
> > > >
> > > > Caused by: org.apache.lucene.index.CorruptIndexException: checksum
> > > > failed *(hardware
> > > > problem?)* : expected=db243d1a actual=7a00d3d2
> > > >
> > >
> (resource=BufferedChecksumIndexInput(MMapIndexInput(path="/app/solr/data/COLL_shard1_replica1/data/index/_i27s.cfs")
> > > > [slice=_i27s_Lucene50_0.tim])
> > > >
> > > > It suddenly started in the logs and before which there was no such
> error.
> > > > Searches & ingestions all seems to be working prior to that.
> > > >
> > > > ----
> > > >
> > > > 2018-09-03 17:16:49.056 INFO (qtp834133664-519872) [c:COLL s:shard1
> > > > r:core_node1 x:COLL_shard1_replica1]
> > > > o.a.s.u.p.StatelessScriptUpdateProcessorFactory
> update-script#processAdd:
> > > >
> newid=G31MXMRZESC0CYPR!A-G31MXMRZESC0CYPR.2552019802_1-2552008480_1-en_US
> > > > 2018-09-03 17:16:49.057 ERROR (qtp834133664-519872) [c:COLL s:shard1
> > > > r:core_node1 x:COLL_shard1_replica1] o.a.s.h.RequestHandlerBase
> > > > org.apache.solr.common.SolrException: Exception writing document id
> > > > G31MXMRZESC0CYPR!A-G31MXMRZESC0CYPR.2552019802_1-2552008480_1-en_US
> to
> > > the
> > > > index; possible analysis error.
> > > > at
> > > >
> > >
> org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:206)
> > > > at
> > > >
> > >
> org.apache.solr.update.processor.RunUpdateProcessor.processAdd(RunUpdateProcessorFactory.java:67)
> > > > at
> > > >
> > >
> org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55)
> > > > at
> > > >
> > >
> org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalAdd(DistributedUpdateProcessor.java:979)
> > > > at
> > > >
> > >
> org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:1192)
> > > > at
> > > >
> > >
> org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:748)
> > > > at
> > > >
> > >
> org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55)
> > > > at
> > > >
> > >
> org.apache.solr.update.processor.StatelessScriptUpdateProcessorFactory$ScriptUpdateProcessor.processAdd(StatelessScriptUpdateProcessorFactory.java:380)
> > > > at
> > > >
> > >
> org.apache.solr.handler.loader.JavabinLoader$1.update(JavabinLoader.java:98)
> > > > at
> > > >
> > >
> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readOuterMostDocIterator(JavaBinUpdateRequestCodec.java:180)
> > > > at
> > > >
> > >
> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readIterator(JavaBinUpdateRequestCodec.java:136)
> > > > at
> > > >
> > >
> org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:306)
> > > > at
> > > org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:251)
> > > > at
> > > >
> > >
> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$1.readNamedList(JavaBinUpdateRequestCodec.java:122)
> > > > at
> > > >
> > >
> org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:271)
> > > > at
> > > org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:251)
> > > > at
> > >
> org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:173)
> > > > at
> > > >
> > >
> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.unmarshal(JavaBinUpdateRequestCodec.java:187)
> > > > at
> > > >
> > >
> org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoader.java:108)
> > > > at
> > >
> org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoader.java:55)
> > > > at
> > > >
> > >
> org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:97)
> > > > at
> > > >
> > >
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)
> > > > at
> > > >
> > >
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:173)
> > > > at org.apache.solr.core.SolrCore.execute(SolrCore.java:2477)
> > > > at
> org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:723)
> > > > at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:529)
> > > > at
> > > >
> > >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:361)
> > > > at
> > > >
> > >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:305)
> > > > at
> > > >
> > >
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
> > > > at
> > > >
> > >
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
> > > > at
> > > >
> > >
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> > > > at
> > > >
> > >
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> > > > at
> > > >
> > >
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
> > > > at
> > > >
> > >
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
> > > > at
> > >
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
> > > > at
> > > >
> > >
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> > > > at
> > > >
> > >
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
> > > > at
> > > >
> > >
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> > > > at
> > > >
> > >
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
> > > > at
> > > >
> > >
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
> > > > at
> > > >
> > >
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> > > > at
> > > >
> > >
> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
> > > > at
> > > >
> > >
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> > > > at org.eclipse.jetty.server.Server.handle(Server.java:534)
> > > > at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
> > > > at
> > > >
> > >
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
> > > > at
> > > > org.eclipse.jetty.io
> > > .AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
> > > > at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
> > > > at
> > > > org.eclipse.jetty.io
> > > .SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
> > > > at
> > > >
> > >
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
> > > > at
> > > >
> > >
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
> > > > at
> > > >
> > >
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
> > > > at
> > > >
> > >
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
> > > > at
> > > >
> > >
> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
> > > > at java.lang.Thread.run(Thread.java:748)
> > > > Caused by: org.apache.lucene.store.AlreadyClosedException: this
> > > IndexWriter
> > > > is closed
> > > > at
> org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:749)
> > > > at
> org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:763)
> > > > at
> > >
> org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1567)
> > > > at
> > > >
> > >
> org.apache.solr.update.DirectUpdateHandler2.updateDocument(DirectUpdateHandler2.java:924)
> > > > at
> > > >
> > >
> org.apache.solr.update.DirectUpdateHandler2.updateDocOrDocValues(DirectUpdateHandler2.java:913)
> > > > at
> > > >
> > >
> org.apache.solr.update.DirectUpdateHandler2.doNormalUpdate(DirectUpdateHandler2.java:302)
> > > > at
> > > >
> > >
> org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUpdateHandler2.java:239)
> > > > at
> > > >
> > >
> org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpdateHandler2.java:194)
> > > > ... 54 more
> > > > Caused by: org.apache.lucene.index.CorruptIndexException: checksum
> failed
> > > > (hardware problem?) : expected=db243d1a actual=7a00d3d2
> > > >
> > >
> (resource=BufferedChecksumIndexInput(MMapIndexInput(path="/app/solr/data/COLL_shard1_replica1/data/index/_i27s.cfs")
> > > > [slice=_i27s_Lucene50_0.tim]))
> > > > at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:419)
> > > > at
> > >
> org.apache.lucene.codecs.CodecUtil.checksumEntireFile(CodecUtil.java:526)
> > > > at
> > > >
> > >
> org.apache.lucene.codecs.blocktree.BlockTreeTermsReader.checkIntegrity(BlockTreeTermsReader.java:336)
> > > > at
> > > >
> > >
> org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.checkIntegrity(PerFieldPostingsFormat.java:348)
> > > > at
> > > >
> > >
> org.apache.lucene.codecs.perfield.PerFieldMergeState$FilterFieldsProducer.checkIntegrity(PerFieldMergeState.java:271)
> > > > at
> org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:96)
> > > > at
> > > >
> > >
> org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.merge(PerFieldPostingsFormat.java:164)
> > > > at
> > >
> org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:216)
> > > > at
> org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:101)
> > > > at
> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4356)
> > > > at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3931)
> > > > at
> org.apache.solr.update.SolrIndexWriter.merge(SolrIndexWriter.java:188)
> > > > at
> > > >
> > >
> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:624)
> > > > at
> > > >
> > >
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:661)
> > > >
> > > > 2018-09-03 17:16:49.116 INFO (qtp834133664-519872) [c:COLL s:shard1
> > > > r:core_node1 x:COLL_shard1_replica1] o.a.s.c.S.Request
> > > > [COLL_shard1_replica1] webapp=/solr path=/update
> > > > params={wt=javabin&version=2} status=400 QTime=69
> > >
>
