To be more concrete: Is the definitive test of whether or not a core's
index is corrupt to copy it onto a new set of hardware and attempt to write
to it? If this is a definitive test, we can run the experiment and update
the report so you have a sense of how often this happens.
Since this is a SOLR cloud node, which is already removed but whose data
dir was preserved, I believe I can just copy the data directory to a fresh
machine and start a regular non-cloud solr node hosting this core. Can you
please confirm that this will be a definitive test, or whether there is
some aspect needed to make it definitive?
Thanks!
On Wed, Oct 3, 2018 at 2:10 AM Stephen Bianamara <sbianam...@panopto.com>
wrote:
> Hello All --
>
> As it would happen, we've seen this error on version 6.6.2 very recently.
> This is also on an AWS instance, like Simon's report. The drive doesn't
> show any sign of being unhealthy, either from cursory investigation. FWIW,
> this occurred during a collection backup.
>
> Erick, is there some diagnostic data we can find to help pin this down?
>
> Thanks!
> Stephen
>
> On Sun, Sep 30, 2018 at 12:48 PM Susheel Kumar <susheel2...@gmail.com>
> wrote:
>
>> Thank you, Simon. Which basically points that something related to env and
>> was causing the checksum failures than any lucene/solr issue.
>>
>> Eric - I did check with hardware folks and they are aware of some VMware
>> issue where the VM hosted in HCI environment is coming into some halt
>> state
>> for minute or so and may be loosing connections to disk/network. So that
>> probably may be the reason of index corruption though they have not been
>> able to find anything specific from logs during the time Solr run into
>> issue
>>
>> Also I had again issue where Solr is loosing the connection with zookeeper
>> (Client session timed out, have not heard from server in 8367ms for
>> sessionid 0x0) Does that points to similar hardware issue, Any
>> suggestions?
>>
>> Thanks,
>> Susheel
>>
>> 2018-09-29 17:30:44.070 INFO
>> (searcherExecutor-7-thread-1-processing-n:server54:8080_solr
>> x:COLL_shard4_replica2 s:shard4 c:COLL r:core_node8) [c:COLL s:shard4
>> r:core_node8 x:COLL_shard4_replica2] o.a.s.c.SolrCore
>> [COLL_shard4_replica2] Registered new searcher
>> Searcher@7a4465b1[COLL_shard4_replica2]
>>
>> main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_7x3f(6.6.2):C826923/317917:delGen=2523)
>> Uninverting(_83pb(6.6.2):C805451/172968:delGen=2957)
>> Uninverting(_3ywj(6.6.2):C727978/334529:delGen=2962)
>> Uninverting(_7vsw(6.6.2):C872110/385178:delGen=2020)
>> Uninverting(_8n89(6.6.2):C741293/109260:delGen=3863)
>> Uninverting(_7zkq(6.6.2):C720666/101205:delGen=3151)
>> Uninverting(_825d(6.6.2):C707731/112410:delGen=3168)
>> Uninverting(_dgwu(6.6.2):C760421/295964:delGen=4624)
>> Uninverting(_gs5x(6.6.2):C540942/138952:delGen=1623)
>> Uninverting(_gu6a(6.6.2):c75213/35640:delGen=1110)
>> Uninverting(_h33i(6.6.2):c131276/40356:delGen=706)
>> Uninverting(_h5tc(6.6.2):c44320/11080:delGen=380)
>> Uninverting(_h9d9(6.6.2):c35088/3188:delGen=104)
>> Uninverting(_h80h(6.6.2):c11927/3412:delGen=153)
>> Uninverting(_h7ll(6.6.2):c11284/1368:delGen=205)
>> Uninverting(_h8bs(6.6.2):c11518/2103:delGen=149)
>> Uninverting(_h9r3(6.6.2):c16439/1018:delGen=52)
>> Uninverting(_h9z1(6.6.2):c9428/823:delGen=27)
>> Uninverting(_h9v2(6.6.2):c933/33:delGen=12)
>> Uninverting(_ha1c(6.6.2):c1056/1:delGen=1)
>> Uninverting(_ha6i(6.6.2):c1883/124:delGen=8)
>> Uninverting(_ha3x(6.6.2):c807/14:delGen=3)
>> Uninverting(_ha47(6.6.2):c1229/133:delGen=6)
>> Uninverting(_hapk(6.6.2):c523) Uninverting(_haoq(6.6.2):c279)
>> Uninverting(_hamr(6.6.2):c311) Uninverting(_hap0(6.6.2):c338)
>> Uninverting(_hapu(6.6.2):c275) Uninverting(_hapv(6.6.2):C4/2:delGen=1)
>> Uninverting(_hapw(6.6.2):C5/2:delGen=1)
>> Uninverting(_hapx(6.6.2):C2/1:delGen=1)
>> Uninverting(_hapy(6.6.2):C2/1:delGen=1)
>> Uninverting(_hapz(6.6.2):C3/1:delGen=1)
>> Uninverting(_haq0(6.6.2):C6/3:delGen=1)
>> Uninverting(_haq1(6.6.2):C1)))}
>> 2018-09-29 17:30:52.390 WARN
>>
>> (zkCallback-5-thread-91-processing-n:server54:8080_solr-SendThread(server117:2182))
>> [ ] o.a.z.ClientCnxn Client session timed out, have not heard from
>> server in 8367ms for sessionid 0x0
>> 2018-09-29 17:31:01.302 WARN
>>
>> (zkCallback-5-thread-91-processing-n:server54:8080_solr-SendThread(server120:2182))
>> [ ] o.a.z.ClientCnxn Client session timed out, have not heard from
>> server in 8812ms for sessionid 0x0
>> 2018-09-29 17:31:14.049 INFO
>> (zkCallback-5-thread-91-processing-n:server54:8080_solr-EventThread) [
>> ] o.a.s.c.c.ConnectionManager Connection with ZooKeeper
>> reestablished.
>> 2018-09-29 17:31:14.049 INFO
>> (zkCallback-5-thread-91-processing-n:server54:8080_solr-EventThread) [
>> ] o.a.s.c.ZkController ZooKeeper session re-connected ... refreshing
>> core states after session expiration.
>> 2018-09-29 17:31:14.051 INFO
>> (zkCallback-5-thread-91-processing-n:server54:8080_solr-EventThread) [
>> ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (16)
>> -> (15)
>> 2018-09-29 17:31:14.144 INFO (qtp834133664-520378) [c:COLL s:shard4
>> r:core_node8 x:COLL_shard4_replica2] o.a.s.c.S.Request
>> [COLL_shard4_replica2] webapp=/solr path=/admin/ping
>>
>> params={distrib=false&df=wordTokens&_stateVer_=COLL:1246&preferLocalShards=false&qt=/admin/ping&fl=id&fl=score&shards.purpose=4&start=0&fsv=true&shard.url=
>> http://server54:8080/solr/COLL_shard4_replica2/|http://server53:8080/solr/COLL_shard4_replica1/&rows=10&version=2&q={!lucene}*:*&NOW=1538242274139&isShard=true&wt=javabin
>> }
>> webapp=/solr path=/admin/ping
>>
>> params={distrib=false&df=wordTokens&_stateVer_=COLL:1246&preferLocalShards=false&qt=/admin/ping&fl=id&fl=score&shards.purpose=4&start=0&fsv=true&shard.url=
>> http://server54:8080/solr/COLL_shard4_replica2/|http://server53:8080/solr/COLL_shard4_replica1/&rows=10&version=2&q={!lucene}*:*&NOW=1538242274139&isShard=true&wt=javabin
>> }
>> hits=4989979 status=0 QTime=0
>>
>>
>>
>>
>> On Wed, Sep 26, 2018 at 9:44 AM simon <mtnes...@gmail.com> wrote:
>>
>> > I saw something like this a year ago which i reported as a possible
>> bug (
>> > https://issues.apache.org/jira/browse/SOLR-10840, which has a full
>> > description and stack traces)
>> >
>> > This occurred very randomly on an AWS instance; moving the index
>> directory
>> > to a different file system did not fix the problem Eventually I cloned
>> our
>> > environment to a new AWS instance, which proved to be the solution.
>> Why, I
>> > have no idea...
>> >
>> > -Simon
>> >
>> > On Mon, Sep 24, 2018 at 1:13 PM, Susheel Kumar <susheel2...@gmail.com>
>> > wrote:
>> >
>> > > Got it. I'll have first hardware folks check and if they don't
>> see/find
>> > > anything suspicious then i'll return here.
>> > >
>> > > Wondering if any body has seen similar error and if they were able to
>> > > confirm if it was hardware fault or so.
>> > >
>> > > Thnx
>> > >
>> > > On Mon, Sep 24, 2018 at 1:01 PM Erick Erickson <
>> erickerick...@gmail.com>
>> > > wrote:
>> > >
>> > > > Mind you it could _still_ be Solr/Lucene, but let's check the
>> hardware
>> > > > first ;)
>> > > > On Mon, Sep 24, 2018 at 9:50 AM Susheel Kumar <
>> susheel2...@gmail.com>
>> > > > wrote:
>> > > > >
>> > > > > Hi Erick,
>> > > > >
>> > > > > Thanks so much for your reply. I'll now look mostly into any
>> > possible
>> > > > > hardware issues than Solr/Lucene.
>> > > > >
>> > > > > Thanks again.
>> > > > >
>> > > > > On Mon, Sep 24, 2018 at 12:43 PM Erick Erickson <
>> > > erickerick...@gmail.com
>> > > > >
>> > > > > wrote:
>> > > > >
>> > > > > > There are several of reasons this would "suddenly" start
>> appearing.
>> > > > > > 1> Your disk went bad and some sector is no longer faithfully
>> > > > > > recording the bits. In this case the checksum will be wrong
>> > > > > > 2> You ran out of disk space sometime and the index was
>> corrupted.
>> > > > > > This isn't really a hardware problem.
>> > > > > > 3> Your disk controller is going wonky and not reading reliably.
>> > > > > >
>> > > > > > The "possible hardware issue" message is to alert you that this
>> is
>> > > > > > highly unusual and you should at leasts consider doing integrity
>> > > > > > checks on your disk before assuming it's a Solr/Lucene problem
>> > > > > >
>> > > > > > Best,
>> > > > > > Erick
>> > > > > > On Mon, Sep 24, 2018 at 9:26 AM Susheel Kumar <
>> > susheel2...@gmail.com
>> > > >
>> > > > > > wrote:
>> > > > > > >
>> > > > > > > Hello,
>> > > > > > >
>> > > > > > > I am still trying to understand the corrupt index exception we
>> > saw
>> > > > in our
>> > > > > > > logs. What does the hardware problem comment indicates here?
>> > Does
>> > > > that
>> > > > > > > mean it caused most likely due to hardware issue?
>> > > > > > >
>> > > > > > > We never had this problem in last couple of months. The Solr
>> is
>> > > > 6.6.2 and
>> > > > > > > ZK: 3.4.10.
>> > > > > > >
>> > > > > > > Please share your thoughts.
>> > > > > > >
>> > > > > > > Thanks,
>> > > > > > > Susheel
>> > > > > > >
>> > > > > > > Caused by: org.apache.lucene.index.CorruptIndexException:
>> > checksum
>> > > > > > > failed *(hardware
>> > > > > > > problem?)* : expected=db243d1a actual=7a00d3d2
>> > > > > > >
>> > > > > >
>> > > > (resource=BufferedChecksumIndexInput(MMapIndexInput(path="/
>> > > app/solr/data/COLL_shard1_replica1/data/index/_i27s.cfs")
>> > > > > > > [slice=_i27s_Lucene50_0.tim])
>> > > > > > >
>> > > > > > > It suddenly started in the logs and before which there was no
>> > such
>> > > > error.
>> > > > > > > Searches & ingestions all seems to be working prior to that.
>> > > > > > >
>> > > > > > > ----
>> > > > > > >
>> > > > > > > 2018-09-03 17:16:49.056 INFO (qtp834133664-519872) [c:COLL
>> > > s:shard1
>> > > > > > > r:core_node1 x:COLL_shard1_replica1]
>> > > > > > > o.a.s.u.p.StatelessScriptUpdateProcessorFactory
>> > > > update-script#processAdd:
>> > > > > > >
>> > > > newid=G31MXMRZESC0CYPR!A-G31MXMRZESC0CYPR.2552019802_1-25520
>> > > 08480_1-en_US
>> > > > > > > 2018-09-03 17:16:49.057 ERROR (qtp834133664-519872) [c:COLL
>> > > s:shard1
>> > > > > > > r:core_node1 x:COLL_shard1_replica1]
>> o.a.s.h.RequestHandlerBase
>> > > > > > > org.apache.solr.common.SolrException: Exception writing
>> document
>> > > id
>> > > > > > > G31MXMRZESC0CYPR!A-G31MXMRZESC0CYPR.2552019802_1-2552008480_
>> > > 1-en_US
>> > > > to
>> > > > > > the
>> > > > > > > index; possible analysis error.
>> > > > > > > at
>> > > > > > >
>> > > > > >
>> > > > org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpd
>> > > ateHandler2.java:206)
>> > > > > > > at
>> > > > > > >
>> > > > > >
>> > > > org.apache.solr.update.processor.RunUpdateProcessor.processA
>> > > dd(RunUpdateProcessorFactory.java:67)
>> > > > > > > at
>> > > > > > >
>> > > > > >
>> > > > org.apache.solr.update.processor.UpdateRequestProcessor.proc
>> > > essAdd(UpdateRequestProcessor.java:55)
>> > > > > > > at
>> > > > > > >
>> > > > > >
>> > > > org.apache.solr.update.processor.DistributedUpdateProcessor.
>> > > doLocalAdd(DistributedUpdateProcessor.java:979)
>> > > > > > > at
>> > > > > > >
>> > > > > >
>> > > > org.apache.solr.update.processor.DistributedUpdateProcessor.
>> > > versionAdd(DistributedUpdateProcessor.java:1192)
>> > > > > > > at
>> > > > > > >
>> > > > > >
>> > > > org.apache.solr.update.processor.DistributedUpdateProcessor.
>> > > processAdd(DistributedUpdateProcessor.java:748)
>> > > > > > > at
>> > > > > > >
>> > > > > >
>> > > > org.apache.solr.update.processor.UpdateRequestProcessor.proc
>> > > essAdd(UpdateRequestProcessor.java:55)
>> > > > > > > at
>> > > > > > >
>> > > > > >
>> > > >
>> org.apache.solr.update.processor.StatelessScriptUpdateProcessorFactory$
>> > > ScriptUpdateProcessor.processAdd(StatelessScriptUpdateProces
>> > > sorFactory.java:380)
>> > > > > > > at
>> > > > > > >
>> > > > > >
>> > > > org.apache.solr.handler.loader.JavabinLoader$1.update(Javabi
>> > > nLoader.java:98)
>> > > > > > > at
>> > > > > > >
>> > > > > >
>> > > > org.apache.solr.client.solrj.request.JavaBinUpdateRequestCod
>> > > ec$1.readOuterMostDocIterator(JavaBinUpdateRequestCodec.java:180)
>> > > > > > > at
>> > > > > > >
>> > > > > >
>> > > > org.apache.solr.client.solrj.request.JavaBinUpdateRequestCod
>> > > ec$1.readIterator(JavaBinUpdateRequestCodec.java:136)
>> > > > > > > at
>> > > > > > >
>> > > > > >
>> > > > org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinC
>> > > odec.java:306)
>> > > > > > > at
>> > > > > > org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCode
>> > > c.java:251)
>> > > > > > > at
>> > > > > > >
>> > > > > >
>> > > > org.apache.solr.client.solrj.request.JavaBinUpdateRequestCod
>> > > ec$1.readNamedList(JavaBinUpdateRequestCodec.java:122)
>> > > > > > > at
>> > > > > > >
>> > > > > >
>> > > > org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinC
>> > > odec.java:271)
>> > > > > > > at
>> > > > > > org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCode
>> > > c.java:251)
>> > > > > > > at
>> > > > > >
>> > > > org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCo
>> > > dec.java:173)
>> > > > > > > at
>> > > > > > >
>> > > > > >
>> > > > org.apache.solr.client.solrj.request.JavaBinUpdateRequestCod
>> > > ec.unmarshal(JavaBinUpdateRequestCodec.java:187)
>> > > > > > > at
>> > > > > > >
>> > > > > >
>> > > > org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDoc
>> > > s(JavabinLoader.java:108)
>> > > > > > > at
>> > > > > >
>> > > >
>> > org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoader.java:55)
>> > > > > > > at
>> > > > > > >
>> > > > > >
>> > > > org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRe
>> > > questHandler.java:97)
>> > > > > > > at
>> > > > > > >
>> > > > > >
>> > > > org.apache.solr.handler.ContentStreamHandlerBase.handleReque
>> > > stBody(ContentStreamHandlerBase.java:68)
>> > > > > > > at
>> > > > > > >
>> > > > > >
>> > > > org.apache.solr.handler.RequestHandlerBase.handleRequest(Req
>> > > uestHandlerBase.java:173)
>> > > > > > > at org.apache.solr.core.SolrCore.execute(SolrCore.java:2477)
>> > > > > > > at
>> > > > org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:723)
>> > > > > > > at
>> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:
>> > > 529)
>> > > > > > > at
>> > > > > > >
>> > > > > >
>> > > > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDisp
>> > > atchFilter.java:361)
>> > > > > > > at
>> > > > > > >
>> > > > > >
>> > > > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDisp
>> > > atchFilter.java:305)
>> > > > > > > at
>> > > > > > >
>> > > > > >
>> > > > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilte
>> > > r(ServletHandler.java:1691)
>> > > > > > > at
>> > > > > > >
>> > > > > >
>> > > > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHan
>> > > dler.java:582)
>> > > > > > > at
>> > > > > > >
>> > > > > >
>> > > > org.eclipse.jetty.server.handler.ScopedHandler.handle(Scoped
>> > > Handler.java:143)
>> > > > > > > at
>> > > > > > >
>> > > > > >
>> > > > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHa
>> > > ndler.java:548)
>> > > > > > > at
>> > > > > > >
>> > > > > >
>> > > > org.eclipse.jetty.server.session.SessionHandler.doHandle(
>> > > SessionHandler.java:226)
>> > > > > > > at
>> > > > > > >
>> > > > > >
>> > > > org.eclipse.jetty.server.handler.ContextHandler.doHandle(
>> > > ContextHandler.java:1180)
>> > > > > > > at
>> > > > > >
>> > > > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHand
>> > > ler.java:512)
>> > > > > > > at
>> > > > > > >
>> > > > > >
>> > > > org.eclipse.jetty.server.session.SessionHandler.doScope(
>> > > SessionHandler.java:185)
>> > > > > > > at
>> > > > > > >
>> > > > > >
>> > > > org.eclipse.jetty.server.handler.ContextHandler.doScope(
>> > > ContextHandler.java:1112)
>> > > > > > > at
>> > > > > > >
>> > > > > >
>> > > > org.eclipse.jetty.server.handler.ScopedHandler.handle(Scoped
>> > > Handler.java:141)
>> > > > > > > at
>> > > > > > >
>> > > > > >
>> > > > org.eclipse.jetty.server.handler.ContextHandlerCollection.ha
>> > > ndle(ContextHandlerCollection.java:213)
>> > > > > > > at
>> > > > > > >
>> > > > > >
>> > > > org.eclipse.jetty.server.handler.HandlerCollection.handle(
>> > > HandlerCollection.java:119)
>> > > > > > > at
>> > > > > > >
>> > > > > >
>> > > > org.eclipse.jetty.server.handler.HandlerWrapper.handle(Handl
>> > > erWrapper.java:134)
>> > > > > > > at
>> > > > > > >
>> > > > > >
>> > > > org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(Rewr
>> > > iteHandler.java:335)
>> > > > > > > at
>> > > > > > >
>> > > > > >
>> > > > org.eclipse.jetty.server.handler.HandlerWrapper.handle(Handl
>> > > erWrapper.java:134)
>> > > > > > > at org.eclipse.jetty.server.Server.handle(Server.java:534)
>> > > > > > > at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.
>> > > java:320)
>> > > > > > > at
>> > > > > > >
>> > > > > >
>> > > > org.eclipse.jetty.server.HttpConnection.onFillable(HttpConne
>> > > ction.java:251)
>> > > > > > > at
>> > > > > > > org.eclipse.jetty.io
>> > > > > > .AbstractConnection$ReadCallback.succeeded(AbstractConnectio
>> > > n.java:273)
>> > > > > > > at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.
>> > > java:95)
>> > > > > > > at
>> > > > > > > org.eclipse.jetty.io
>> > > > > > .SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
>> > > > > > > at
>> > > > > > >
>> > > > > >
>> > > > org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume
>> > > .executeProduceConsume(ExecuteProduceConsume.java:303)
>> > > > > > > at
>> > > > > > >
>> > > > > >
>> > > > org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume
>> > > .produceConsume(ExecuteProduceConsume.java:148)
>> > > > > > > at
>> > > > > > >
>> > > > > >
>> > > > org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume
>> > > .run(ExecuteProduceConsume.java:136)
>> > > > > > > at
>> > > > > > >
>> > > > > >
>> > > > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(Queued
>> > > ThreadPool.java:671)
>> > > > > > > at
>> > > > > > >
>> > > > > >
>> > > > org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedT
>> > > hreadPool.java:589)
>> > > > > > > at java.lang.Thread.run(Thread.java:748)
>> > > > > > > Caused by: org.apache.lucene.store.AlreadyClosedException:
>> this
>> > > > > > IndexWriter
>> > > > > > > is closed
>> > > > > > > at
>> > > > org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:749)
>> > > > > > > at
>> > > > org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:763)
>> > > > > > > at
>> > > > > >
>> > > > org.apache.lucene.index.IndexWriter.updateDocument(IndexWrit
>> > > er.java:1567)
>> > > > > > > at
>> > > > > > >
>> > > > > >
>> > > > org.apache.solr.update.DirectUpdateHandler2.updateDocument(D
>> > > irectUpdateHandler2.java:924)
>> > > > > > > at
>> > > > > > >
>> > > > > >
>> > > > org.apache.solr.update.DirectUpdateHandler2.updateDocOrDocVa
>> > > lues(DirectUpdateHandler2.java:913)
>> > > > > > > at
>> > > > > > >
>> > > > > >
>> > > > org.apache.solr.update.DirectUpdateHandler2.doNormalUpdate(D
>> > > irectUpdateHandler2.java:302)
>> > > > > > > at
>> > > > > > >
>> > > > > >
>> > > > org.apache.solr.update.DirectUpdateHandler2.addDoc0(DirectUp
>> > > dateHandler2.java:239)
>> > > > > > > at
>> > > > > > >
>> > > > > >
>> > > > org.apache.solr.update.DirectUpdateHandler2.addDoc(DirectUpd
>> > > ateHandler2.java:194)
>> > > > > > > ... 54 more
>> > > > > > > Caused by: org.apache.lucene.index.CorruptIndexException:
>> > checksum
>> > > > failed
>> > > > > > > (hardware problem?) : expected=db243d1a actual=7a00d3d2
>> > > > > > >
>> > > > > >
>> > > > (resource=BufferedChecksumIndexInput(MMapIndexInput(path="/
>> > > app/solr/data/COLL_shard1_replica1/data/index/_i27s.cfs")
>> > > > > > > [slice=_i27s_Lucene50_0.tim]))
>> > > > > > > at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.
>> > > java:419)
>> > > > > > > at
>> > > > > >
>> > > > org.apache.lucene.codecs.CodecUtil.checksumEntireFile(CodecU
>> > > til.java:526)
>> > > > > > > at
>> > > > > > >
>> > > > > >
>> > > > org.apache.lucene.codecs.blocktree.BlockTreeTermsReader.chec
>> > > kIntegrity(BlockTreeTermsReader.java:336)
>> > > > > > > at
>> > > > > > >
>> > > > > >
>> > > > org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$Fie
>> > > ldsReader.checkIntegrity(PerFieldPostingsFormat.java:348)
>> > > > > > > at
>> > > > > > >
>> > > > > >
>> > > > org.apache.lucene.codecs.perfield.PerFieldMergeState$FilterF
>> > > ieldsProducer.checkIntegrity(PerFieldMergeState.java:271)
>> > > > > > > at
>> > > >
>> org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:96)
>> > > > > > > at
>> > > > > > >
>> > > > > >
>> > > > org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$Fie
>> > > ldsWriter.merge(PerFieldPostingsFormat.java:164)
>> > > > > > > at
>> > > > > >
>> > > >
>> > org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:216)
>> > > > > > > at
>> > > > org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:101)
>> > > > > > > at
>> > > >
>> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4356)
>> > > > > > > at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:
>> > > 3931)
>> > > > > > > at
>> > > >
>> org.apache.solr.update.SolrIndexWriter.merge(SolrIndexWriter.java:188)
>> > > > > > > at
>> > > > > > >
>> > > > > >
>> > > > org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(Con
>> > > currentMergeScheduler.java:624)
>> > > > > > > at
>> > > > > > >
>> > > > > >
>> > > > org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread
>> > > .run(ConcurrentMergeScheduler.java:661)
>> > > > > > >
>> > > > > > > 2018-09-03 17:16:49.116 INFO (qtp834133664-519872) [c:COLL
>> > > s:shard1
>> > > > > > > r:core_node1 x:COLL_shard1_replica1] o.a.s.c.S.Request
>> > > > > > > [COLL_shard1_replica1] webapp=/solr path=/update
>> > > > > > > params={wt=javabin&version=2} status=400 QTime=69
>> > > > > >
>> > > >
>> > >
>> >
>>
>
