There is some analysis error also. I would suggest to test the indexer on
just one shard setup first, then test for a replica (1 shard and 1 replica)
and then test for 2 shards and 2 replica. This would confirm if there is
basic issue with indexing / cluster setup.
On Mon, Jul 17, 2017 at 9:04 AM, Joe Obernberger <
joseph.obernber...@gmail.com> wrote:
> Some more info:
>
> When I stop all the indexers, in about 5-10 minutes the cluster goes all
> green. When I start just one indexer, several nodes immediately go down
> with the 'Error adding log' message.
>
> I'm using CloudSolrClient.add(List<SolrInputDocument>) to do the
> indexing. Is this correct for SolrCloud?
>
> Thank you!
>
> -Joe
>
>
>
> On 7/17/2017 8:36 AM, Joe Obernberger wrote:
>
>> We've been indexing data on a 45 node cluster with 100 shards and 3
>> replicas, but our indexing processes have been stopping due to errors. On
>> the server side the error is "Error logging add". Stack trace:
>>
>> 2017-07-17 12:29:24.057 INFO (qtp985934102-5161548) [c:UNCLASS s:shard58
>> r:core_node290 x:UNCLASS_shard58_replica1]
>> o.a.s.u.p.LogUpdateProcessorFactory
>> [UNCLASS_shard58_replica1] webapp=/solr path=/update
>> params={update.distrib=FROMLEADER&update.chain=add-unknown-
>> fields-to-the-schema&distrib.from=http://tarvos:9100/solr/
>> UNCLASS_shard58_replica2/&wt=javabin&version=2}{add=[
>> COLLECT20003218348784 (1573172872544780288), COLLECT20003218351447
>> (1573172872620277760), COLLECT20003218353085 (1573172872625520640),
>> COLLECT20003218357937 (1573172872627617792), COLLECT20003218361860
>> (1573172872629714944), COLLECT20003218362535 (1573172872631812096)]} 0 171
>> 2017-07-17 12:29:24.160 INFO (qtp985934102-5160762) [c:UNCLASS s:shard13
>> r:core_node81 x:UNCLASS_shard13_replica1] o.a.s.u.p.LogUpdateProcessorFactory
>> [UNCLASS_shard13_replica1] webapp=/solr path=/update
>> params={update.distrib=FROMLEADER&update.chain=add-unknown-
>> fields-to-the-schema&distrib.from=http://tarvos:9100/solr/
>> UNCLASS_shard13_replica2/&wt=javabin&version=2}{add=[
>> COLLECT20003218344436 (1573172872538488832), COLLECT20003218347497
>> (1573172872620277760), COLLECT20003218351645 (1573172872625520640),
>> COLLECT20003218356965 (1573172872629714944), COLLECT20003218357775
>> (1573172872632860672), COLLECT20003218358017 (1573172872646492160),
>> COLLECT20003218358152 (1573172872650686464), COLLECT20003218359395
>> (1573172872651735040), COLLECT20003218362571 (1573172872652783616)]} 0 274
>> 2017-07-17 12:29:24.163 INFO (qtp985934102-5161057) [c:UNCLASS s:shard43
>> r:core_node108 x:UNCLASS_shard43_replica1]
>> o.a.s.u.p.LogUpdateProcessorFactory
>> [UNCLASS_shard43_replica1] webapp=/solr path=/update
>> params={update.distrib=FROMLEADER&update.chain=add-unknown-
>> fields-to-the-schema&distrib.from=http://tarvos:9100/solr/
>> UNCLASS_shard43_replica2/&wt=javabin&version=2}{} 0 0
>> 2017-07-17 12:29:24.164 ERROR (qtp985934102-5161057) [c:UNCLASS s:shard43
>> r:core_node108 x:UNCLASS_shard43_replica1] o.a.s.h.RequestHandlerBase
>> org.apache.solr.common.SolrException: Error logging add
>> at org.apache.solr.update.TransactionLog.write(TransactionLog.
>> java:418)
>> at org.apache.solr.update.UpdateLog.add(UpdateLog.java:532)
>> at org.apache.solr.update.UpdateLog.add(UpdateLog.java:516)
>> at org.apache.solr.update.processor.DistributedUpdateProcessor.
>> versionAdd(DistributedUpdateProcessor.java:1113)
>> at org.apache.solr.update.processor.DistributedUpdateProcessor.
>> processAdd(DistributedUpdateProcessor.java:748)
>> at org.apache.solr.update.processor.LogUpdateProcessorFactory$L
>> ogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103)
>> at org.apache.solr.handler.loader.JavabinLoader$1.update(Javabi
>> nLoader.java:98)
>> at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCod
>> ec$1.readOuterMostDocIterator(JavaBinUpdateRequestCodec.java:180)
>> at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCod
>> ec$1.readIterator(JavaBinUpdateRequestCodec.java:136)
>> at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinC
>> odec.java:306)
>> at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCode
>> c.java:251)
>> at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCod
>> ec$1.readNamedList(JavaBinUpdateRequestCodec.java:122)
>> at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinC
>> odec.java:271)
>> at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCode
>> c.java:251)
>> at org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCo
>> dec.java:173)
>> at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCod
>> ec.unmarshal(JavaBinUpdateRequestCodec.java:187)
>> at org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDoc
>> s(JavabinLoader.java:108)
>> at org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoa
>> der.java:55)
>> at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRe
>> questHandler.java:97)
>> at org.apache.solr.handler.ContentStreamHandlerBase.handleReque
>> stBody(ContentStreamHandlerBase.java:68)
>> at org.apache.solr.handler.RequestHandlerBase.handleRequest(Req
>> uestHandlerBase.java:173)
>> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2477)
>> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.
>> java:723)
>> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:
>> 529)
>> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDisp
>> atchFilter.java:361)
>> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDisp
>> atchFilter.java:305)
>> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilte
>> r(ServletHandler.java:1691)
>> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHan
>> dler.java:582)
>> at org.eclipse.jetty.server.handler.ScopedHandler.handle(Scoped
>> Handler.java:143)
>> at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHa
>> ndler.java:548)
>> at org.eclipse.jetty.server.session.SessionHandler.doHandle(
>> SessionHandler.java:226)
>> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(
>> ContextHandler.java:1180)
>> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHand
>> ler.java:512)
>> at org.eclipse.jetty.server.session.SessionHandler.doScope(
>> SessionHandler.java:185)
>> at org.eclipse.jetty.server.handler.ContextHandler.doScope(
>> ContextHandler.java:1112)
>> at org.eclipse.jetty.server.handler.ScopedHandler.handle(Scoped
>> Handler.java:141)
>> at org.eclipse.jetty.server.handler.ContextHandlerCollection.ha
>> ndle(ContextHandlerCollection.java:213)
>> at org.eclipse.jetty.server.handler.HandlerCollection.handle(
>> HandlerCollection.java:119)
>> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(Handl
>> erWrapper.java:134)
>> at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(Rewr
>> iteHandler.java:335)
>> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(Handl
>> erWrapper.java:134)
>> at org.eclipse.jetty.server.Server.handle(Server.java:534)
>> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.
>> java:320)
>> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConne
>> ction.java:251)
>> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.
>> succeeded(AbstractConnection.java:273)
>> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.
>> java:95)
>> at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChann
>> elEndPoint.java:93)
>> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume
>> .executeProduceConsume(ExecuteProduceConsume.java:303)
>> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume
>> .produceConsume(ExecuteProduceConsume.java:148)
>> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume
>> .run(ExecuteProduceConsume.java:136)
>> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(Queued
>> ThreadPool.java:671)
>> at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedT
>> hreadPool.java:589)
>> at java.lang.Thread.run(Thread.java:748)
>> Caused by: org.apache.hadoop.ipc.RemoteException(java.io.IOException):
>> File /solr6.6.0/UNCLASS/core_node108/data/tlog/tlog.0000000000000006211
>> could only be replicated to 0 nodes instead of minReplication (=1). There
>> are 40 datanode(s) running and no node(s) are excluded in this operation.
>> at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.c
>> hooseTarget4NewBlock(BlockManager.java:1622)
>> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAddit
>> ionalBlock(FSNamesystem.java:3351)
>> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.add
>> Block(NameNodeRpcServer.java:683)
>> at org.apache.hadoop.hdfs.server.namenode.AuthorizationProvider
>> ProxyClientProtocol.addBlock(AuthorizationProviderProxyClien
>> tProtocol.java:214)
>> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServ
>> erSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSide
>> TranslatorPB.java:495)
>> at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocol
>> Protos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNam
>> enodeProtocolProtos.java)
>> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcIn
>> voker.call(ProtobufRpcEngine.java:617)
>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2212)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:422)
>> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGro
>> upInformation.java:1920)
>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2210)
>>
>> at org.apache.hadoop.ipc.Client.call(Client.java:1475)
>> at org.apache.hadoop.ipc.Client.call(Client.java:1412)
>> at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(
>> ProtobufRpcEngine.java:229)
>> at com.sun.proxy.$Proxy11.addBlock(Unknown Source)
>> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTran
>> slatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:418)
>> at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe
>> thodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:498)
>> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMeth
>> od(RetryInvocationHandler.java:191)
>> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(Ret
>> ryInvocationHandler.java:102)
>> at com.sun.proxy.$Proxy12.addBlock(Unknown Source)
>> at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFo
>> llowingBlock(DFSOutputStream.java:1459)
>> at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBloc
>> kOutputStream(DFSOutputStream.java:1255)
>> at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(
>> DFSOutputStream.java:449)
>>
>> 2017-07-17 12:29:24.164 ERROR (qtp985934102-5161057) [c:UNCLASS s:shard43
>> r:core_node108 x:UNCLASS_shard43_replica1] o.a.s.s.HttpSolrCall null:
>> org.apache.solr.common.SolrException: Error logging add
>> at org.apache.solr.update.TransactionLog.write(TransactionLog.
>> java:418)
>> at org.apache.solr.update.UpdateLog.add(UpdateLog.java:532)
>> at org.apache.solr.update.UpdateLog.add(UpdateLog.java:516)
>> at org.apache.solr.update.processor.DistributedUpdateProcessor.
>> versionAdd(DistributedUpdateProcessor.java:1113)
>> at org.apache.solr.update.processor.DistributedUpdateProcessor.
>> processAdd(DistributedUpdateProcessor.java:748)
>> at org.apache.solr.update.processor.LogUpdateProcessorFactory$L
>> ogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103)
>> at org.apache.solr.handler.loader.JavabinLoader$1.update(Javabi
>> nLoader.java:98)
>> at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCod
>> ec$1.readOuterMostDocIterator(JavaBinUpdateRequestCodec.java:180)
>> at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCod
>> ec$1.readIterator(JavaBinUpdateRequestCodec.java:136)
>> at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinC
>> odec.java:306)
>> at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCode
>> c.java:251)
>> at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCod
>> ec$1.readNamedList(JavaBinUpdateRequestCodec.java:122)
>> at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinC
>> odec.java:271)
>> at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCode
>> c.java:251)
>> at org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCo
>> dec.java:173)
>> at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCod
>> ec.unmarshal(JavaBinUpdateRequestCodec.java:187)
>> at org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDoc
>> s(JavabinLoader.java:108)
>> at org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoa
>> der.java:55)
>> at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRe
>> questHandler.java:97)
>> at org.apache.solr.handler.ContentStreamHandlerBase.handleReque
>> stBody(ContentStreamHandlerBase.java:68)
>> at org.apache.solr.handler.RequestHandlerBase.handleRequest(Req
>> uestHandlerBase.java:173)
>> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2477)
>> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.
>> java:723)
>> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:
>> 529)
>> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDisp
>> atchFilter.java:361)
>> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDisp
>> atchFilter.java:305)
>> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilte
>> r(ServletHandler.java:1691)
>> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHan
>> dler.java:582)
>> at org.eclipse.jetty.server.handler.ScopedHandler.handle(Scoped
>> Handler.java:143)
>> at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHa
>> ndler.java:548)
>> at org.eclipse.jetty.server.session.SessionHandler.doHandle(
>> SessionHandler.java:226)
>> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(
>> ContextHandler.java:1180)
>> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHand
>> ler.java:512)
>> at org.eclipse.jetty.server.session.SessionHandler.doScope(
>> SessionHandler.java:185)
>> at org.eclipse.jetty.server.handler.ContextHandler.doScope(
>> ContextHandler.java:1112)
>> at org.eclipse.jetty.server.handler.ScopedHandler.handle(Scoped
>> Handler.java:141)
>> at org.eclipse.jetty.server.handler.ContextHandlerCollection.ha
>> ndle(ContextHandlerCollection.java:213)
>> at org.eclipse.jetty.server.handler.HandlerCollection.handle(
>> HandlerCollection.java:119)
>> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(Handl
>> erWrapper.java:134)
>> at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(Rewr
>> iteHandler.java:335)
>> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(Handl
>> erWrapper.java:134)
>> at org.eclipse.jetty.server.Server.handle(Server.java:534)
>> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.
>> java:320)
>> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConne
>> ction.java:251)
>> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.
>> succeeded(AbstractConnection.java:273)
>> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.
>> java:95)
>> at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChann
>> elEndPoint.java:93)
>> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume
>> .executeProduceConsume(ExecuteProduceConsume.java:303)
>> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume
>> .produceConsume(ExecuteProduceConsume.java:148)
>> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume
>> .run(ExecuteProduceConsume.java:136)
>> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(Queued
>> ThreadPool.java:671)
>> at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedT
>> hreadPool.java:589)
>> at java.lang.Thread.run(Thread.java:748)
>> Caused by: org.apache.hadoop.ipc.RemoteException(java.io.IOException):
>> File /solr6.6.0/UNCLASS/core_node108/data/tlog/tlog.0000000000000006211
>> could only be replicated to 0 nodes instead of minReplication (=1). There
>> are 40 datanode(s) running and no node(s) are excluded in this operation.
>> at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.c
>> hooseTarget4NewBlock(BlockManager.java:1622)
>> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAddit
>> ionalBlock(FSNamesystem.java:3351)
>> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.add
>> Block(NameNodeRpcServer.java:683)
>> at org.apache.hadoop.hdfs.server.namenode.AuthorizationProvider
>> ProxyClientProtocol.addBlock(AuthorizationProviderProxyClien
>> tProtocol.java:214)
>> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServ
>> erSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSide
>> TranslatorPB.java:495)
>> at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocol
>> Protos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNam
>> enodeProtocolProtos.java)
>> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcIn
>> voker.call(ProtobufRpcEngine.java:617)
>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216)
>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2212)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:422)
>> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGro
>> upInformation.java:1920)
>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2210)
>>
>> at org.apache.hadoop.ipc.Client.call(Client.java:1475)
>> at org.apache.hadoop.ipc.Client.call(Client.java:1412)
>> at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(
>> ProtobufRpcEngine.java:229)
>> at com.sun.proxy.$Proxy11.addBlock(Unknown Source)
>> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTran
>> slatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:418)
>> at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe
>> thodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:498)
>> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMeth
>> od(RetryInvocationHandler.java:191)
>> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(Ret
>> ryInvocationHandler.java:102)
>> at com.sun.proxy.$Proxy12.addBlock(Unknown Source)
>> at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFo
>> llowingBlock(DFSOutputStream.java:1459)
>> at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBloc
>> kOutputStream(DFSOutputStream.java:1255)
>> at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(
>> DFSOutputStream.java:449)
>>
>> 2017-07-17 12:29:24.187 INFO
>> (zkCallback-5-thread-144-processing-n:juliet:9100_solr)
>> [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
>> state:SyncConnected type:NodeDataChanged
>> path:/collections/UNCLASS/state.json]
>> for collection [UNCLASS] has occurred - updating... (live nodes size: [45])
>>
>> On the client side, the error looks like:
>> 2017-07-16 19:03:16,118 WARN [com.ngc.bigdata.ie_solrindexer.IndexDocument]
>> Indexing error: org.apache.solr.client.solrj.i
>> mpl.CloudSolrClient$RouteException: Error from server at
>> http://leda:9100/solr/UNCLASS_shard44_replica3: Exception writing
>> document id COLLECT10086453202 to the index; possible analysis error. for
>> collection: UNCLASS
>> org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: Error
>> from server at http://leda:9100/solr/UNCLASS_shard44_replica3: Exception
>> writing document id COLLECT10086453202 to the index; possible analysis
>> error.
>> at org.apache.solr.client.solrj.impl.CloudSolrClient.directUpda
>> te(CloudSolrClient.java:819)
>> at org.apache.solr.client.solrj.impl.CloudSolrClient.sendReques
>> t(CloudSolrClient.java:1263)
>> at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWit
>> hRetryOnStaleState(CloudSolrClient.java:1134)
>> at org.apache.solr.client.solrj.impl.CloudSolrClient.request(Cl
>> oudSolrClient.java:1073)
>> at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest
>> .java:160)
>> at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:
>> 106)
>> at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:
>> 71)
>> at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:
>> 85)
>> at com.ngc.bigdata.ie_solrindexer.IndexDocument.indexSolrDocs(
>> IndexDocument.java:959)
>> at com.ngc.bigdata.ie_solrindexer.IndexDocument.index(
>> IndexDocument.java:236)
>> at com.ngc.bigdata.ie_solrindexer.SolrIndexerProcessor.doWork(S
>> olrIndexerProcessor.java:63)
>> at com.ngc.intelenterprise.intelentutil.utils.Processor.run(
>> Processor.java:140)
>> at com.ngc.intelenterprise.intelentutil.jms.IntelEntQueueProc.
>> process(IntelEntQueueProc.java:208)
>> at org.apache.camel.processor.DelegateSyncProcessor.process(Del
>> egateSyncProcessor.java:63)
>> at org.apache.camel.management.InstrumentationProcessor.process
>> (InstrumentationProcessor.java:77)
>> at org.apache.camel.processor.RedeliveryErrorHandler.process(Re
>> deliveryErrorHandler.java:460)
>> at org.apache.camel.processor.CamelInternalProcessor.process(Ca
>> melInternalProcessor.java:190)
>> at org.apache.camel.processor.CamelInternalProcessor.process(Ca
>> melInternalProcessor.java:190)
>> at org.apache.camel.component.seda.SedaConsumer.sendToConsumers
>> (SedaConsumer.java:298)
>> at org.apache.camel.component.seda.SedaConsumer.doRun(SedaConsu
>> mer.java:207)
>> at org.apache.camel.component.seda.SedaConsumer.run(SedaConsume
>> r.java:154)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
>> Executor.java:1142)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
>> lExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:748)
>> Caused by:
>> org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:
>> Error from server at http://leda:9100/solr/UNCLASS_shard44_replica3:
>> Exception writing document id COLLECT10086453202 to the index; possible
>> analysis error.
>> at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMeth
>> od(HttpSolrClient.java:610)
>> at org.apache.solr.client.solrj.impl.HttpSolrClient.request(Htt
>> pSolrClient.java:279)
>> at org.apache.solr.client.solrj.impl.HttpSolrClient.request(Htt
>> pSolrClient.java:268)
>> at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest
>> (LBHttpSolrClient.java:447)
>> at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(L
>> BHttpSolrClient.java:388)
>> at org.apache.solr.client.solrj.impl.CloudSolrClient.lambda$dir
>> ectUpdate$0(CloudSolrClient.java:796)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolE
>> xecutor.lambda$execute$0(ExecutorUtil.java:229)
>> ... 3 more
>> 2017-07-16 19:03:16,134 ERROR [com.ngc.bigdata.ie_solrindexer.IndexDocument]
>> Error indexing: org.apache.solr.client.solrj.i
>> mpl.CloudSolrClient$RouteException: Error from server at
>> http://leda:9100/solr/UNCLASS_shard44_replica3: Exception writing
>> document id COLLECT10086453202 to the index; possible analysis error. for
>> collection: UNCLASS.
>> 2017-07-16 19:03:16,135 ERROR [com.ngc.bigdata.ie_solrindexer.IndexDocument]
>> Exception during indexing: org.apache.solr.client.solrj.i
>> mpl.CloudSolrClient$RouteException: Error from server at
>> http://leda:9100/solr/UNCLASS_shard44_replica3: Exception writing
>> document id COLLECT10086453202 to the index; possible analysis error.
>>
>> I can fire them back up, but they only run for a short time before
>> getting more indexing errors. Several of the nodes show as down in the
>> cloud view. Any help would be appreciated! Thank you!
>>
>>
>> -Joe
>>
>>
>
