So far we've indexed about 46 million documents, but over the weekend,
these errors started coming up. I would expect that if there was a
basic issue, it would have started right away? We ran a test cluster
with just a few shards/replicas prior and didn't see any issues using
the same indexing code, but we're running a lot more indexers
simultaneously with the larger cluster; perhaps we're just overloading
HDFS? The same nodes that run Solr also run HDFS datanodes, but they
are pretty beefy machines; we're not swapping.
As Shawn pointed out, I will be checking the HDFS version (we're using
Cloudera CDH 5.10.2), and the HDFS logs.
-Joe
On 7/17/2017 10:16 AM, Susheel Kumar wrote:
There is some analysis error also. I would suggest to test the indexer on
just one shard setup first, then test for a replica (1 shard and 1 replica)
and then test for 2 shards and 2 replica. This would confirm if there is
basic issue with indexing / cluster setup.
On Mon, Jul 17, 2017 at 9:04 AM, Joe Obernberger <
joseph.obernber...@gmail.com> wrote:
Some more info:
When I stop all the indexers, in about 5-10 minutes the cluster goes all
green. When I start just one indexer, several nodes immediately go down
with the 'Error adding log' message.
I'm using CloudSolrClient.add(List<SolrInputDocument>) to do the
indexing. Is this correct for SolrCloud?
Thank you!
-Joe
On 7/17/2017 8:36 AM, Joe Obernberger wrote:
We've been indexing data on a 45 node cluster with 100 shards and 3
replicas, but our indexing processes have been stopping due to errors. On
the server side the error is "Error logging add". Stack trace:
2017-07-17 12:29:24.057 INFO (qtp985934102-5161548) [c:UNCLASS s:shard58
r:core_node290 x:UNCLASS_shard58_replica1] o.a.s.u.p.LogUpdateProcessorFactory
[UNCLASS_shard58_replica1] webapp=/solr path=/update
params={update.distrib=FROMLEADER&update.chain=add-unknown-
fields-to-the-schema&distrib.from=http://tarvos:9100/solr/
UNCLASS_shard58_replica2/&wt=javabin&version=2}{add=[
COLLECT20003218348784 (1573172872544780288), COLLECT20003218351447
(1573172872620277760), COLLECT20003218353085 (1573172872625520640),
COLLECT20003218357937 (1573172872627617792), COLLECT20003218361860
(1573172872629714944), COLLECT20003218362535 (1573172872631812096)]} 0 171
2017-07-17 12:29:24.160 INFO (qtp985934102-5160762) [c:UNCLASS s:shard13
r:core_node81 x:UNCLASS_shard13_replica1] o.a.s.u.p.LogUpdateProcessorFactory
[UNCLASS_shard13_replica1] webapp=/solr path=/update
params={update.distrib=FROMLEADER&update.chain=add-unknown-
fields-to-the-schema&distrib.from=http://tarvos:9100/solr/
UNCLASS_shard13_replica2/&wt=javabin&version=2}{add=[
COLLECT20003218344436 (1573172872538488832), COLLECT20003218347497
(1573172872620277760), COLLECT20003218351645 (1573172872625520640),
COLLECT20003218356965 (1573172872629714944), COLLECT20003218357775
(1573172872632860672), COLLECT20003218358017 (1573172872646492160),
COLLECT20003218358152 (1573172872650686464), COLLECT20003218359395
(1573172872651735040), COLLECT20003218362571 (1573172872652783616)]} 0 274
2017-07-17 12:29:24.163 INFO (qtp985934102-5161057) [c:UNCLASS s:shard43
r:core_node108 x:UNCLASS_shard43_replica1] o.a.s.u.p.LogUpdateProcessorFactory
[UNCLASS_shard43_replica1] webapp=/solr path=/update
params={update.distrib=FROMLEADER&update.chain=add-unknown-
fields-to-the-schema&distrib.from=http://tarvos:9100/solr/
UNCLASS_shard43_replica2/&wt=javabin&version=2}{} 0 0
2017-07-17 12:29:24.164 ERROR (qtp985934102-5161057) [c:UNCLASS s:shard43
r:core_node108 x:UNCLASS_shard43_replica1] o.a.s.h.RequestHandlerBase
org.apache.solr.common.SolrException: Error logging add
at org.apache.solr.update.TransactionLog.write(TransactionLog.
java:418)
at org.apache.solr.update.UpdateLog.add(UpdateLog.java:532)
at org.apache.solr.update.UpdateLog.add(UpdateLog.java:516)
at org.apache.solr.update.processor.DistributedUpdateProcessor.
versionAdd(DistributedUpdateProcessor.java:1113)
at org.apache.solr.update.processor.DistributedUpdateProcessor.
processAdd(DistributedUpdateProcessor.java:748)
at org.apache.solr.update.processor.LogUpdateProcessorFactory$L
ogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103)
at org.apache.solr.handler.loader.JavabinLoader$1.update(Javabi
nLoader.java:98)
at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCod
ec$1.readOuterMostDocIterator(JavaBinUpdateRequestCodec.java:180)
at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCod
ec$1.readIterator(JavaBinUpdateRequestCodec.java:136)
at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinC
odec.java:306)
at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCode
c.java:251)
at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCod
ec$1.readNamedList(JavaBinUpdateRequestCodec.java:122)
at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinC
odec.java:271)
at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCode
c.java:251)
at org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCo
dec.java:173)
at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCod
ec.unmarshal(JavaBinUpdateRequestCodec.java:187)
at org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDoc
s(JavabinLoader.java:108)
at org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoa
der.java:55)
at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRe
questHandler.java:97)
at org.apache.solr.handler.ContentStreamHandlerBase.handleReque
stBody(ContentStreamHandlerBase.java:68)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(Req
uestHandlerBase.java:173)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2477)
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.
java:723)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:
529)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDisp
atchFilter.java:361)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDisp
atchFilter.java:305)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilte
r(ServletHandler.java:1691)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHan
dler.java:582)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(Scoped
Handler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHa
ndler.java:548)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(
SessionHandler.java:226)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(
ContextHandler.java:1180)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHand
ler.java:512)
at org.eclipse.jetty.server.session.SessionHandler.doScope(
SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(
ContextHandler.java:1112)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(Scoped
Handler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.ha
ndle(ContextHandlerCollection.java:213)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(
HandlerCollection.java:119)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(Handl
erWrapper.java:134)
at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(Rewr
iteHandler.java:335)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(Handl
erWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:534)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.
java:320)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConne
ction.java:251)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.
succeeded(AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.
java:95)
at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChann
elEndPoint.java:93)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume
.executeProduceConsume(ExecuteProduceConsume.java:303)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume
.produceConsume(ExecuteProduceConsume.java:148)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume
.run(ExecuteProduceConsume.java:136)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(Queued
ThreadPool.java:671)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedT
hreadPool.java:589)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.hadoop.ipc.RemoteException(java.io.IOException):
File /solr6.6.0/UNCLASS/core_node108/data/tlog/tlog.0000000000000006211
could only be replicated to 0 nodes instead of minReplication (=1). There
are 40 datanode(s) running and no node(s) are excluded in this operation.
at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.c
hooseTarget4NewBlock(BlockManager.java:1622)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAddit
ionalBlock(FSNamesystem.java:3351)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.add
Block(NameNodeRpcServer.java:683)
at org.apache.hadoop.hdfs.server.namenode.AuthorizationProvider
ProxyClientProtocol.addBlock(AuthorizationProviderProxyClien
tProtocol.java:214)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServ
erSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSide
TranslatorPB.java:495)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocol
Protos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNam
enodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcIn
voker.call(ProtobufRpcEngine.java:617)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2212)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGro
upInformation.java:1920)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2210)
at org.apache.hadoop.ipc.Client.call(Client.java:1475)
at org.apache.hadoop.ipc.Client.call(Client.java:1412)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(
ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy11.addBlock(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTran
slatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:418)
at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe
thodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMeth
od(RetryInvocationHandler.java:191)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(Ret
ryInvocationHandler.java:102)
at com.sun.proxy.$Proxy12.addBlock(Unknown Source)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFo
llowingBlock(DFSOutputStream.java:1459)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBloc
kOutputStream(DFSOutputStream.java:1255)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(
DFSOutputStream.java:449)
2017-07-17 12:29:24.164 ERROR (qtp985934102-5161057) [c:UNCLASS s:shard43
r:core_node108 x:UNCLASS_shard43_replica1] o.a.s.s.HttpSolrCall null:
org.apache.solr.common.SolrException: Error logging add
at org.apache.solr.update.TransactionLog.write(TransactionLog.
java:418)
at org.apache.solr.update.UpdateLog.add(UpdateLog.java:532)
at org.apache.solr.update.UpdateLog.add(UpdateLog.java:516)
at org.apache.solr.update.processor.DistributedUpdateProcessor.
versionAdd(DistributedUpdateProcessor.java:1113)
at org.apache.solr.update.processor.DistributedUpdateProcessor.
processAdd(DistributedUpdateProcessor.java:748)
at org.apache.solr.update.processor.LogUpdateProcessorFactory$L
ogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103)
at org.apache.solr.handler.loader.JavabinLoader$1.update(Javabi
nLoader.java:98)
at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCod
ec$1.readOuterMostDocIterator(JavaBinUpdateRequestCodec.java:180)
at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCod
ec$1.readIterator(JavaBinUpdateRequestCodec.java:136)
at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinC
odec.java:306)
at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCode
c.java:251)
at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCod
ec$1.readNamedList(JavaBinUpdateRequestCodec.java:122)
at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinC
odec.java:271)
at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCode
c.java:251)
at org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCo
dec.java:173)
at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCod
ec.unmarshal(JavaBinUpdateRequestCodec.java:187)
at org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDoc
s(JavabinLoader.java:108)
at org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoa
der.java:55)
at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRe
questHandler.java:97)
at org.apache.solr.handler.ContentStreamHandlerBase.handleReque
stBody(ContentStreamHandlerBase.java:68)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(Req
uestHandlerBase.java:173)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2477)
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.
java:723)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:
529)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDisp
atchFilter.java:361)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDisp
atchFilter.java:305)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilte
r(ServletHandler.java:1691)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHan
dler.java:582)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(Scoped
Handler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHa
ndler.java:548)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(
SessionHandler.java:226)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(
ContextHandler.java:1180)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHand
ler.java:512)
at org.eclipse.jetty.server.session.SessionHandler.doScope(
SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(
ContextHandler.java:1112)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(Scoped
Handler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.ha
ndle(ContextHandlerCollection.java:213)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(
HandlerCollection.java:119)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(Handl
erWrapper.java:134)
at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(Rewr
iteHandler.java:335)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(Handl
erWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:534)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.
java:320)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConne
ction.java:251)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.
succeeded(AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.
java:95)
at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChann
elEndPoint.java:93)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume
.executeProduceConsume(ExecuteProduceConsume.java:303)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume
.produceConsume(ExecuteProduceConsume.java:148)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume
.run(ExecuteProduceConsume.java:136)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(Queued
ThreadPool.java:671)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedT
hreadPool.java:589)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.hadoop.ipc.RemoteException(java.io.IOException):
File /solr6.6.0/UNCLASS/core_node108/data/tlog/tlog.0000000000000006211
could only be replicated to 0 nodes instead of minReplication (=1). There
are 40 datanode(s) running and no node(s) are excluded in this operation.
at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.c
hooseTarget4NewBlock(BlockManager.java:1622)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAddit
ionalBlock(FSNamesystem.java:3351)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.add
Block(NameNodeRpcServer.java:683)
at org.apache.hadoop.hdfs.server.namenode.AuthorizationProvider
ProxyClientProtocol.addBlock(AuthorizationProviderProxyClien
tProtocol.java:214)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServ
erSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSide
TranslatorPB.java:495)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocol
Protos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNam
enodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcIn
voker.call(ProtobufRpcEngine.java:617)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2216)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2212)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGro
upInformation.java:1920)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2210)
at org.apache.hadoop.ipc.Client.call(Client.java:1475)
at org.apache.hadoop.ipc.Client.call(Client.java:1412)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(
ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy11.addBlock(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTran
slatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:418)
at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe
thodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMeth
od(RetryInvocationHandler.java:191)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(Ret
ryInvocationHandler.java:102)
at com.sun.proxy.$Proxy12.addBlock(Unknown Source)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFo
llowingBlock(DFSOutputStream.java:1459)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBloc
kOutputStream(DFSOutputStream.java:1255)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(
DFSOutputStream.java:449)
2017-07-17 12:29:24.187 INFO
(zkCallback-5-thread-144-processing-n:juliet:9100_solr)
[ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
state:SyncConnected type:NodeDataChanged path:/collections/UNCLASS/state.json]
for collection [UNCLASS] has occurred - updating... (live nodes size: [45])
On the client side, the error looks like:
2017-07-16 19:03:16,118 WARN [com.ngc.bigdata.ie_solrindexer.IndexDocument]
Indexing error: org.apache.solr.client.solrj.i
mpl.CloudSolrClient$RouteException: Error from server at
http://leda:9100/solr/UNCLASS_shard44_replica3: Exception writing
document id COLLECT10086453202 to the index; possible analysis error. for
collection: UNCLASS
org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: Error
from server at http://leda:9100/solr/UNCLASS_shard44_replica3: Exception
writing document id COLLECT10086453202 to the index; possible analysis
error.
at org.apache.solr.client.solrj.impl.CloudSolrClient.directUpda
te(CloudSolrClient.java:819)
at org.apache.solr.client.solrj.impl.CloudSolrClient.sendReques
t(CloudSolrClient.java:1263)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWit
hRetryOnStaleState(CloudSolrClient.java:1134)
at org.apache.solr.client.solrj.impl.CloudSolrClient.request(Cl
oudSolrClient.java:1073)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest
.java:160)
at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:
106)
at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:
71)
at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:
85)
at com.ngc.bigdata.ie_solrindexer.IndexDocument.indexSolrDocs(
IndexDocument.java:959)
at com.ngc.bigdata.ie_solrindexer.IndexDocument.index(
IndexDocument.java:236)
at com.ngc.bigdata.ie_solrindexer.SolrIndexerProcessor.doWork(S
olrIndexerProcessor.java:63)
at com.ngc.intelenterprise.intelentutil.utils.Processor.run(
Processor.java:140)
at com.ngc.intelenterprise.intelentutil.jms.IntelEntQueueProc.
process(IntelEntQueueProc.java:208)
at org.apache.camel.processor.DelegateSyncProcessor.process(Del
egateSyncProcessor.java:63)
at org.apache.camel.management.InstrumentationProcessor.process
(InstrumentationProcessor.java:77)
at org.apache.camel.processor.RedeliveryErrorHandler.process(Re
deliveryErrorHandler.java:460)
at org.apache.camel.processor.CamelInternalProcessor.process(Ca
melInternalProcessor.java:190)
at org.apache.camel.processor.CamelInternalProcessor.process(Ca
melInternalProcessor.java:190)
at org.apache.camel.component.seda.SedaConsumer.sendToConsumers
(SedaConsumer.java:298)
at org.apache.camel.component.seda.SedaConsumer.doRun(SedaConsu
mer.java:207)
at org.apache.camel.component.seda.SedaConsumer.run(SedaConsume
r.java:154)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
Executor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
lExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:
Error from server at http://leda:9100/solr/UNCLASS_shard44_replica3:
Exception writing document id COLLECT10086453202 to the index; possible
analysis error.
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMeth
od(HttpSolrClient.java:610)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(Htt
pSolrClient.java:279)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(Htt
pSolrClient.java:268)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest
(LBHttpSolrClient.java:447)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(L
BHttpSolrClient.java:388)
at org.apache.solr.client.solrj.impl.CloudSolrClient.lambda$dir
ectUpdate$0(CloudSolrClient.java:796)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolE
xecutor.lambda$execute$0(ExecutorUtil.java:229)
... 3 more
2017-07-16 19:03:16,134 ERROR [com.ngc.bigdata.ie_solrindexer.IndexDocument]
Error indexing: org.apache.solr.client.solrj.i
mpl.CloudSolrClient$RouteException: Error from server at
http://leda:9100/solr/UNCLASS_shard44_replica3: Exception writing
document id COLLECT10086453202 to the index; possible analysis error. for
collection: UNCLASS.
2017-07-16 19:03:16,135 ERROR [com.ngc.bigdata.ie_solrindexer.IndexDocument]
Exception during indexing: org.apache.solr.client.solrj.i
mpl.CloudSolrClient$RouteException: Error from server at
http://leda:9100/solr/UNCLASS_shard44_replica3: Exception writing
document id COLLECT10086453202 to the index; possible analysis error.
I can fire them back up, but they only run for a short time before
getting more indexing errors. Several of the nodes show as down in the
cloud view. Any help would be appreciated! Thank you!
-Joe
---
This email has been checked for viruses by AVG.
http://www.avg.com
