Joe:
I agree that 46 million docs later you'd expect things to have settled
out. However, I do note that you have
"add-unknown-fields-to-the-schema" in your error stack which means
you're using "field guessing", sometimes called data_driven. I would
recommend you do _not_ use this for production as, while it does the
best job it can it has to make assumptions about what the data looks
like based on the first document it sees which may later be violated.
Getting "possible analysis error" is one of the messages that happens
when this occurs.
The simple example is that if the first time data_driven sees "1"
it'll guess integer. If sometime later there's a doc with "1.0" it'll
generate a parse error.
I totally agree that 46 million docs later you'd expect all of this
kind of thing to have flushed out, but the "possible analysis error"
seems to be pointing that direction. If this is, indeed, the problem
you'll see better evidence on the Solr instance that's actually having
the problem. Unfortunately you'll just to look at one Solr log from
each shard to see whether this is an issue.
Best,
Erick
On Mon, Jul 17, 2017 at 7:23 AM, Joe Obernberger
<joseph.obernber...@gmail.com> wrote:
> So far we've indexed about 46 million documents, but over the weekend, these
> errors started coming up. I would expect that if there was a basic issue,
> it would have started right away? We ran a test cluster with just a few
> shards/replicas prior and didn't see any issues using the same indexing
> code, but we're running a lot more indexers simultaneously with the larger
> cluster; perhaps we're just overloading HDFS? The same nodes that run Solr
> also run HDFS datanodes, but they are pretty beefy machines; we're not
> swapping.
>
> As Shawn pointed out, I will be checking the HDFS version (we're using
> Cloudera CDH 5.10.2), and the HDFS logs.
>
> -Joe
>
>
>
> On 7/17/2017 10:16 AM, Susheel Kumar wrote:
>>
>> There is some analysis error also. I would suggest to test the indexer on
>> just one shard setup first, then test for a replica (1 shard and 1
>> replica)
>> and then test for 2 shards and 2 replica. This would confirm if there is
>> basic issue with indexing / cluster setup.
>>
>> On Mon, Jul 17, 2017 at 9:04 AM, Joe Obernberger <
>> joseph.obernber...@gmail.com> wrote:
>>
>>> Some more info:
>>>
>>> When I stop all the indexers, in about 5-10 minutes the cluster goes all
>>> green. When I start just one indexer, several nodes immediately go down
>>> with the 'Error adding log' message.
>>>
>>> I'm using CloudSolrClient.add(List<SolrInputDocument>) to do the
>>> indexing. Is this correct for SolrCloud?
>>>
>>> Thank you!
>>>
>>> -Joe
>>>
>>>
>>>
>>> On 7/17/2017 8:36 AM, Joe Obernberger wrote:
>>>
>>>> We've been indexing data on a 45 node cluster with 100 shards and 3
>>>> replicas, but our indexing processes have been stopping due to errors.
>>>> On
>>>> the server side the error is "Error logging add". Stack trace:
>>>>
>>>> 2017-07-17 12:29:24.057 INFO (qtp985934102-5161548) [c:UNCLASS
>>>> s:shard58
>>>> r:core_node290 x:UNCLASS_shard58_replica1]
>>>> o.a.s.u.p.LogUpdateProcessorFactory
>>>> [UNCLASS_shard58_replica1] webapp=/solr path=/update
>>>> params={update.distrib=FROMLEADER&update.chain=add-unknown-
>>>> fields-to-the-schema&distrib.from=http://tarvos:9100/solr/
>>>> UNCLASS_shard58_replica2/&wt=javabin&version=2}{add=[
>>>> COLLECT20003218348784 (1573172872544780288), COLLECT20003218351447
>>>> (1573172872620277760), COLLECT20003218353085 (1573172872625520640),
>>>> COLLECT20003218357937 (1573172872627617792), COLLECT20003218361860
>>>> (1573172872629714944), COLLECT20003218362535 (1573172872631812096)]} 0
>>>> 171
>>>> 2017-07-17 12:29:24.160 INFO (qtp985934102-5160762) [c:UNCLASS
>>>> s:shard13
>>>> r:core_node81 x:UNCLASS_shard13_replica1]
>>>> o.a.s.u.p.LogUpdateProcessorFactory
>>>> [UNCLASS_shard13_replica1] webapp=/solr path=/update
>>>> params={update.distrib=FROMLEADER&update.chain=add-unknown-
>>>> fields-to-the-schema&distrib.from=http://tarvos:9100/solr/
>>>> UNCLASS_shard13_replica2/&wt=javabin&version=2}{add=[
>>>> COLLECT20003218344436 (1573172872538488832), COLLECT20003218347497
>>>> (1573172872620277760), COLLECT20003218351645 (1573172872625520640),
>>>> COLLECT20003218356965 (1573172872629714944), COLLECT20003218357775
>>>> (1573172872632860672), COLLECT20003218358017 (1573172872646492160),
>>>> COLLECT20003218358152 (1573172872650686464), COLLECT20003218359395
>>>> (1573172872651735040), COLLECT20003218362571 (1573172872652783616)]} 0
>>>> 274
>>>> 2017-07-17 12:29:24.163 INFO (qtp985934102-5161057) [c:UNCLASS
>>>> s:shard43
>>>> r:core_node108 x:UNCLASS_shard43_replica1]
>>>> o.a.s.u.p.LogUpdateProcessorFactory
>>>> [UNCLASS_shard43_replica1] webapp=/solr path=/update
>>>> params={update.distrib=FROMLEADER&update.chain=add-unknown-
>>>> fields-to-the-schema&distrib.from=http://tarvos:9100/solr/
>>>> UNCLASS_shard43_replica2/&wt=javabin&version=2}{} 0 0
>>>> 2017-07-17 12:29:24.164 ERROR (qtp985934102-5161057) [c:UNCLASS
>>>> s:shard43
>>>> r:core_node108 x:UNCLASS_shard43_replica1] o.a.s.h.RequestHandlerBase
>>>> org.apache.solr.common.SolrException: Error logging add
>>>> at org.apache.solr.update.TransactionLog.write(TransactionLog.
>>>> java:418)
>>>> at org.apache.solr.update.UpdateLog.add(UpdateLog.java:532)
>>>> at org.apache.solr.update.UpdateLog.add(UpdateLog.java:516)
>>>> at org.apache.solr.update.processor.DistributedUpdateProcessor.
>>>> versionAdd(DistributedUpdateProcessor.java:1113)
>>>> at org.apache.solr.update.processor.DistributedUpdateProcessor.
>>>> processAdd(DistributedUpdateProcessor.java:748)
>>>> at org.apache.solr.update.processor.LogUpdateProcessorFactory$L
>>>> ogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103)
>>>> at org.apache.solr.handler.loader.JavabinLoader$1.update(Javabi
>>>> nLoader.java:98)
>>>> at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCod
>>>> ec$1.readOuterMostDocIterator(JavaBinUpdateRequestCodec.java:180)
>>>> at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCod
>>>> ec$1.readIterator(JavaBinUpdateRequestCodec.java:136)
>>>> at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinC
>>>> odec.java:306)
>>>> at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCode
>>>> c.java:251)
>>>> at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCod
>>>> ec$1.readNamedList(JavaBinUpdateRequestCodec.java:122)
>>>> at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinC
>>>> odec.java:271)
>>>> at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCode
>>>> c.java:251)
>>>> at org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCo
>>>> dec.java:173)
>>>> at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCod
>>>> ec.unmarshal(JavaBinUpdateRequestCodec.java:187)
>>>> at org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDoc
>>>> s(JavabinLoader.java:108)
>>>> at org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoa
>>>> der.java:55)
>>>> at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRe
>>>> questHandler.java:97)
>>>> at org.apache.solr.handler.ContentStreamHandlerBase.handleReque
>>>> stBody(ContentStreamHandlerBase.java:68)
>>>> at org.apache.solr.handler.RequestHandlerBase.handleRequest(Req
>>>> uestHandlerBase.java:173)
>>>> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2477)
>>>> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.
>>>> java:723)
>>>> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:
>>>> 529)
>>>> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDisp
>>>> atchFilter.java:361)
>>>> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDisp
>>>> atchFilter.java:305)
>>>> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilte
>>>> r(ServletHandler.java:1691)
>>>> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHan
>>>> dler.java:582)
>>>> at org.eclipse.jetty.server.handler.ScopedHandler.handle(Scoped
>>>> Handler.java:143)
>>>> at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHa
>>>> ndler.java:548)
>>>> at org.eclipse.jetty.server.session.SessionHandler.doHandle(
>>>> SessionHandler.java:226)
>>>> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(
>>>> ContextHandler.java:1180)
>>>> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHand
>>>> ler.java:512)
>>>> at org.eclipse.jetty.server.session.SessionHandler.doScope(
>>>> SessionHandler.java:185)
>>>> at org.eclipse.jetty.server.handler.ContextHandler.doScope(
>>>> ContextHandler.java:1112)
>>>> at org.eclipse.jetty.server.handler.ScopedHandler.handle(Scoped
>>>> Handler.java:141)
>>>> at org.eclipse.jetty.server.handler.ContextHandlerCollection.ha
>>>> ndle(ContextHandlerCollection.java:213)
>>>> at org.eclipse.jetty.server.handler.HandlerCollection.handle(
>>>> HandlerCollection.java:119)
>>>> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(Handl
>>>> erWrapper.java:134)
>>>> at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(Rewr
>>>> iteHandler.java:335)
>>>> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(Handl
>>>> erWrapper.java:134)
>>>> at org.eclipse.jetty.server.Server.handle(Server.java:534)
>>>> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.
>>>> java:320)
>>>> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConne
>>>> ction.java:251)
>>>> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.
>>>> succeeded(AbstractConnection.java:273)
>>>> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.
>>>> java:95)
>>>> at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChann
>>>> elEndPoint.java:93)
>>>> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume
>>>> .executeProduceConsume(ExecuteProduceConsume.java:303)
>>>> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume
>>>> .produceConsume(ExecuteProduceConsume.java:148)
>>>> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume
>>>> .run(ExecuteProduceConsume.java:136)
>>>> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(Queued
>>>> ThreadPool.java:671)
>>>> at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedT
>>>> hreadPool.java:589)
>>>> at java.lang.Thread.run(Thread.java:748)
>>>> Caused by: org.apache.hadoop.ipc.RemoteException(java.io.IOException):
>>>> File /solr6.6.0/UNCLASS/core_node108/data/tlog/tlog.0000000000000006211
>>>> could only be replicated to 0 nodes instead of minReplication (=1).
>>>> There
>>>> are 40 datanode(s) running and no node(s) are excluded in this
>>>> operation.
>>>> at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.c
>>>> hooseTarget4NewBlock(BlockManager.java:1622)
>>>> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAddit
>>>> ionalBlock(FSNamesystem.java:3351)
>>>> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.add
>>>> Block(NameNodeRpcServer.java:683)
>>>> at org.apache.hadoop.hdfs.server.namenode.AuthorizationProvider
>>>> ProxyClientProtocol.addBlock(AuthorizationProviderProxyClien
>>>> tProtocol.java:214)
>>>> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServ
>>>> erSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSide
>>>> TranslatorPB.java:495)
>>>> at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocol
>>>> Protos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNam
>>>> enodeProtocolProtos.java)
>>>> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcIn
>>>> voker.call(ProtobufRpcEngine.java:617)
>>>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
>>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216)
>>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2212)
>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>> at javax.security.auth.Subject.doAs(Subject.java:422)
>>>> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGro
>>>> upInformation.java:1920)
>>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2210)
>>>>
>>>> at org.apache.hadoop.ipc.Client.call(Client.java:1475)
>>>> at org.apache.hadoop.ipc.Client.call(Client.java:1412)
>>>> at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(
>>>> ProtobufRpcEngine.java:229)
>>>> at com.sun.proxy.$Proxy11.addBlock(Unknown Source)
>>>> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTran
>>>> slatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:418)
>>>> at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)
>>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe
>>>> thodAccessorImpl.java:43)
>>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>>> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMeth
>>>> od(RetryInvocationHandler.java:191)
>>>> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(Ret
>>>> ryInvocationHandler.java:102)
>>>> at com.sun.proxy.$Proxy12.addBlock(Unknown Source)
>>>> at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFo
>>>> llowingBlock(DFSOutputStream.java:1459)
>>>> at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBloc
>>>> kOutputStream(DFSOutputStream.java:1255)
>>>> at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(
>>>> DFSOutputStream.java:449)
>>>>
>>>> 2017-07-17 12:29:24.164 ERROR (qtp985934102-5161057) [c:UNCLASS
>>>> s:shard43
>>>> r:core_node108 x:UNCLASS_shard43_replica1] o.a.s.s.HttpSolrCall null:
>>>> org.apache.solr.common.SolrException: Error logging add
>>>> at org.apache.solr.update.TransactionLog.write(TransactionLog.
>>>> java:418)
>>>> at org.apache.solr.update.UpdateLog.add(UpdateLog.java:532)
>>>> at org.apache.solr.update.UpdateLog.add(UpdateLog.java:516)
>>>> at org.apache.solr.update.processor.DistributedUpdateProcessor.
>>>> versionAdd(DistributedUpdateProcessor.java:1113)
>>>> at org.apache.solr.update.processor.DistributedUpdateProcessor.
>>>> processAdd(DistributedUpdateProcessor.java:748)
>>>> at org.apache.solr.update.processor.LogUpdateProcessorFactory$L
>>>> ogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103)
>>>> at org.apache.solr.handler.loader.JavabinLoader$1.update(Javabi
>>>> nLoader.java:98)
>>>> at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCod
>>>> ec$1.readOuterMostDocIterator(JavaBinUpdateRequestCodec.java:180)
>>>> at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCod
>>>> ec$1.readIterator(JavaBinUpdateRequestCodec.java:136)
>>>> at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinC
>>>> odec.java:306)
>>>> at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCode
>>>> c.java:251)
>>>> at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCod
>>>> ec$1.readNamedList(JavaBinUpdateRequestCodec.java:122)
>>>> at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinC
>>>> odec.java:271)
>>>> at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCode
>>>> c.java:251)
>>>> at org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCo
>>>> dec.java:173)
>>>> at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCod
>>>> ec.unmarshal(JavaBinUpdateRequestCodec.java:187)
>>>> at org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDoc
>>>> s(JavabinLoader.java:108)
>>>> at org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoa
>>>> der.java:55)
>>>> at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRe
>>>> questHandler.java:97)
>>>> at org.apache.solr.handler.ContentStreamHandlerBase.handleReque
>>>> stBody(ContentStreamHandlerBase.java:68)
>>>> at org.apache.solr.handler.RequestHandlerBase.handleRequest(Req
>>>> uestHandlerBase.java:173)
>>>> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2477)
>>>> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.
>>>> java:723)
>>>> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:
>>>> 529)
>>>> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDisp
>>>> atchFilter.java:361)
>>>> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDisp
>>>> atchFilter.java:305)
>>>> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilte
>>>> r(ServletHandler.java:1691)
>>>> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHan
>>>> dler.java:582)
>>>> at org.eclipse.jetty.server.handler.ScopedHandler.handle(Scoped
>>>> Handler.java:143)
>>>> at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHa
>>>> ndler.java:548)
>>>> at org.eclipse.jetty.server.session.SessionHandler.doHandle(
>>>> SessionHandler.java:226)
>>>> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(
>>>> ContextHandler.java:1180)
>>>> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHand
>>>> ler.java:512)
>>>> at org.eclipse.jetty.server.session.SessionHandler.doScope(
>>>> SessionHandler.java:185)
>>>> at org.eclipse.jetty.server.handler.ContextHandler.doScope(
>>>> ContextHandler.java:1112)
>>>> at org.eclipse.jetty.server.handler.ScopedHandler.handle(Scoped
>>>> Handler.java:141)
>>>> at org.eclipse.jetty.server.handler.ContextHandlerCollection.ha
>>>> ndle(ContextHandlerCollection.java:213)
>>>> at org.eclipse.jetty.server.handler.HandlerCollection.handle(
>>>> HandlerCollection.java:119)
>>>> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(Handl
>>>> erWrapper.java:134)
>>>> at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(Rewr
>>>> iteHandler.java:335)
>>>> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(Handl
>>>> erWrapper.java:134)
>>>> at org.eclipse.jetty.server.Server.handle(Server.java:534)
>>>> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.
>>>> java:320)
>>>> at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConne
>>>> ction.java:251)
>>>> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.
>>>> succeeded(AbstractConnection.java:273)
>>>> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.
>>>> java:95)
>>>> at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChann
>>>> elEndPoint.java:93)
>>>> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume
>>>> .executeProduceConsume(ExecuteProduceConsume.java:303)
>>>> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume
>>>> .produceConsume(ExecuteProduceConsume.java:148)
>>>> at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume
>>>> .run(ExecuteProduceConsume.java:136)
>>>> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(Queued
>>>> ThreadPool.java:671)
>>>> at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedT
>>>> hreadPool.java:589)
>>>> at java.lang.Thread.run(Thread.java:748)
>>>> Caused by: org.apache.hadoop.ipc.RemoteException(java.io.IOException):
>>>> File /solr6.6.0/UNCLASS/core_node108/data/tlog/tlog.0000000000000006211
>>>> could only be replicated to 0 nodes instead of minReplication (=1).
>>>> There
>>>> are 40 datanode(s) running and no node(s) are excluded in this
>>>> operation.
>>>> at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.c
>>>> hooseTarget4NewBlock(BlockManager.java:1622)
>>>> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAddit
>>>> ionalBlock(FSNamesystem.java:3351)
>>>> at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.add
>>>> Block(NameNodeRpcServer.java:683)
>>>> at org.apache.hadoop.hdfs.server.namenode.AuthorizationProvider
>>>> ProxyClientProtocol.addBlock(AuthorizationProviderProxyClien
>>>> tProtocol.java:214)
>>>> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServ
>>>> erSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSide
>>>> TranslatorPB.java:495)
>>>> at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocol
>>>> Protos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNam
>>>> enodeProtocolProtos.java)
>>>> at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcIn
>>>> voker.call(ProtobufRpcEngine.java:617)
>>>> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
>>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216)
>>>> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2212)
>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>> at javax.security.auth.Subject.doAs(Subject.java:422)
>>>> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGro
>>>> upInformation.java:1920)
>>>> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2210)
>>>>
>>>> at org.apache.hadoop.ipc.Client.call(Client.java:1475)
>>>> at org.apache.hadoop.ipc.Client.call(Client.java:1412)
>>>> at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(
>>>> ProtobufRpcEngine.java:229)
>>>> at com.sun.proxy.$Proxy11.addBlock(Unknown Source)
>>>> at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTran
>>>> slatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:418)
>>>> at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)
>>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe
>>>> thodAccessorImpl.java:43)
>>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>>> at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMeth
>>>> od(RetryInvocationHandler.java:191)
>>>> at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(Ret
>>>> ryInvocationHandler.java:102)
>>>> at com.sun.proxy.$Proxy12.addBlock(Unknown Source)
>>>> at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFo
>>>> llowingBlock(DFSOutputStream.java:1459)
>>>> at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBloc
>>>> kOutputStream(DFSOutputStream.java:1255)
>>>> at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(
>>>> DFSOutputStream.java:449)
>>>>
>>>> 2017-07-17 12:29:24.187 INFO
>>>> (zkCallback-5-thread-144-processing-n:juliet:9100_solr)
>>>> [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
>>>> state:SyncConnected type:NodeDataChanged
>>>> path:/collections/UNCLASS/state.json]
>>>> for collection [UNCLASS] has occurred - updating... (live nodes size:
>>>> [45])
>>>>
>>>> On the client side, the error looks like:
>>>> 2017-07-16 19:03:16,118 WARN
>>>> [com.ngc.bigdata.ie_solrindexer.IndexDocument]
>>>> Indexing error: org.apache.solr.client.solrj.i
>>>> mpl.CloudSolrClient$RouteException: Error from server at
>>>> http://leda:9100/solr/UNCLASS_shard44_replica3: Exception writing
>>>> document id COLLECT10086453202 to the index; possible analysis error.
>>>> for
>>>> collection: UNCLASS
>>>> org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: Error
>>>> from server at http://leda:9100/solr/UNCLASS_shard44_replica3: Exception
>>>> writing document id COLLECT10086453202 to the index; possible analysis
>>>> error.
>>>> at org.apache.solr.client.solrj.impl.CloudSolrClient.directUpda
>>>> te(CloudSolrClient.java:819)
>>>> at org.apache.solr.client.solrj.impl.CloudSolrClient.sendReques
>>>> t(CloudSolrClient.java:1263)
>>>> at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWit
>>>> hRetryOnStaleState(CloudSolrClient.java:1134)
>>>> at org.apache.solr.client.solrj.impl.CloudSolrClient.request(Cl
>>>> oudSolrClient.java:1073)
>>>> at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest
>>>> .java:160)
>>>> at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:
>>>> 106)
>>>> at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:
>>>> 71)
>>>> at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:
>>>> 85)
>>>> at com.ngc.bigdata.ie_solrindexer.IndexDocument.indexSolrDocs(
>>>> IndexDocument.java:959)
>>>> at com.ngc.bigdata.ie_solrindexer.IndexDocument.index(
>>>> IndexDocument.java:236)
>>>> at com.ngc.bigdata.ie_solrindexer.SolrIndexerProcessor.doWork(S
>>>> olrIndexerProcessor.java:63)
>>>> at com.ngc.intelenterprise.intelentutil.utils.Processor.run(
>>>> Processor.java:140)
>>>> at com.ngc.intelenterprise.intelentutil.jms.IntelEntQueueProc.
>>>> process(IntelEntQueueProc.java:208)
>>>> at org.apache.camel.processor.DelegateSyncProcessor.process(Del
>>>> egateSyncProcessor.java:63)
>>>> at org.apache.camel.management.InstrumentationProcessor.process
>>>> (InstrumentationProcessor.java:77)
>>>> at org.apache.camel.processor.RedeliveryErrorHandler.process(Re
>>>> deliveryErrorHandler.java:460)
>>>> at org.apache.camel.processor.CamelInternalProcessor.process(Ca
>>>> melInternalProcessor.java:190)
>>>> at org.apache.camel.processor.CamelInternalProcessor.process(Ca
>>>> melInternalProcessor.java:190)
>>>> at org.apache.camel.component.seda.SedaConsumer.sendToConsumers
>>>> (SedaConsumer.java:298)
>>>> at org.apache.camel.component.seda.SedaConsumer.doRun(SedaConsu
>>>> mer.java:207)
>>>> at org.apache.camel.component.seda.SedaConsumer.run(SedaConsume
>>>> r.java:154)
>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
>>>> Executor.java:1142)
>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
>>>> lExecutor.java:617)
>>>> at java.lang.Thread.run(Thread.java:748)
>>>> Caused by:
>>>> org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:
>>>> Error from server at http://leda:9100/solr/UNCLASS_shard44_replica3:
>>>> Exception writing document id COLLECT10086453202 to the index; possible
>>>> analysis error.
>>>> at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMeth
>>>> od(HttpSolrClient.java:610)
>>>> at org.apache.solr.client.solrj.impl.HttpSolrClient.request(Htt
>>>> pSolrClient.java:279)
>>>> at org.apache.solr.client.solrj.impl.HttpSolrClient.request(Htt
>>>> pSolrClient.java:268)
>>>> at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest
>>>> (LBHttpSolrClient.java:447)
>>>> at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(L
>>>> BHttpSolrClient.java:388)
>>>> at org.apache.solr.client.solrj.impl.CloudSolrClient.lambda$dir
>>>> ectUpdate$0(CloudSolrClient.java:796)
>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>>> at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolE
>>>> xecutor.lambda$execute$0(ExecutorUtil.java:229)
>>>> ... 3 more
>>>> 2017-07-16 19:03:16,134 ERROR
>>>> [com.ngc.bigdata.ie_solrindexer.IndexDocument]
>>>> Error indexing: org.apache.solr.client.solrj.i
>>>> mpl.CloudSolrClient$RouteException: Error from server at
>>>> http://leda:9100/solr/UNCLASS_shard44_replica3: Exception writing
>>>> document id COLLECT10086453202 to the index; possible analysis error.
>>>> for
>>>> collection: UNCLASS.
>>>> 2017-07-16 19:03:16,135 ERROR
>>>> [com.ngc.bigdata.ie_solrindexer.IndexDocument]
>>>> Exception during indexing: org.apache.solr.client.solrj.i
>>>> mpl.CloudSolrClient$RouteException: Error from server at
>>>> http://leda:9100/solr/UNCLASS_shard44_replica3: Exception writing
>>>> document id COLLECT10086453202 to the index; possible analysis error.
>>>>
>>>> I can fire them back up, but they only run for a short time before
>>>> getting more indexing errors. Several of the nodes show as down in the
>>>> cloud view. Any help would be appreciated! Thank you!
>>>>
>>>>
>>>> -Joe
>>>>
>>>>
>>
>> ---
>> This email has been checked for viruses by AVG.
>> http://www.avg.com
>>
>
