Joe:
I agree that 46 million docs later you'd expect things to have settled
out. However, I do note that you have
"add-unknown-fields-to-the-schema" in your error stack which means
you're using "field guessing", sometimes called data_driven. I would
recommend you do _not_ use this for production as, while it does the
best job it can it has to make assumptions about what the data looks
like based on the first document it sees which may later be violated.
Getting "possible analysis error" is one of the messages that happens
when this occurs.
The simple example is that if the first time data_driven sees "1"
it'll guess integer. If sometime later there's a doc with "1.0" it'll
generate a parse error.
I totally agree that 46 million docs later you'd expect all of this
kind of thing to have flushed out, but the "possible analysis error"
seems to be pointing that direction. If this is, indeed, the problem
you'll see better evidence on the Solr instance that's actually having
the problem. Unfortunately you'll just to look at one Solr log from
each shard to see whether this is an issue.
Best,
Erick
On Mon, Jul 17, 2017 at 7:23 AM, Joe Obernberger
<joseph.obernber...@gmail.com> wrote:
So far we've indexed about 46 million documents, but over the weekend,
these
errors started coming up. I would expect that if there was a basic
issue,
it would have started right away? We ran a test cluster with just a few
shards/replicas prior and didn't see any issues using the same indexing
code, but we're running a lot more indexers simultaneously with the
larger
cluster; perhaps we're just overloading HDFS? The same nodes that run
Solr
also run HDFS datanodes, but they are pretty beefy machines; we're not
swapping.
As Shawn pointed out, I will be checking the HDFS version (we're using
Cloudera CDH 5.10.2), and the HDFS logs.
-Joe
On 7/17/2017 10:16 AM, Susheel Kumar wrote:
There is some analysis error also. I would suggest to test the indexer
on
just one shard setup first, then test for a replica (1 shard and 1
replica)
and then test for 2 shards and 2 replica. This would confirm if there
is
basic issue with indexing / cluster setup.
On Mon, Jul 17, 2017 at 9:04 AM, Joe Obernberger <
joseph.obernber...@gmail.com> wrote:
Some more info:
When I stop all the indexers, in about 5-10 minutes the cluster goes
all
green. When I start just one indexer, several nodes immediately go
down
with the 'Error adding log' message.
I'm using CloudSolrClient.add(List<SolrInputDocument>) to do the
indexing. Is this correct for SolrCloud?
Thank you!
-Joe
On 7/17/2017 8:36 AM, Joe Obernberger wrote:
We've been indexing data on a 45 node cluster with 100 shards and 3
replicas, but our indexing processes have been stopping due to errors.
On
the server side the error is "Error logging add". Stack trace:
2017-07-17 12:29:24.057 INFO (qtp985934102-5161548) [c:UNCLASS
s:shard58
r:core_node290 x:UNCLASS_shard58_replica1]
o.a.s.u.p.LogUpdateProcessorFactory
[UNCLASS_shard58_replica1] webapp=/solr path=/update
params={update.distrib=FROMLEADER&update.chain=add-unknown-
fields-to-the-schema&distrib.from=http://tarvos:9100/solr/
UNCLASS_shard58_replica2/&wt=javabin&version=2}{add=[
COLLECT20003218348784 (1573172872544780288), COLLECT20003218351447
(1573172872620277760), COLLECT20003218353085 (1573172872625520640),
COLLECT20003218357937 (1573172872627617792), COLLECT20003218361860
(1573172872629714944), COLLECT20003218362535 (1573172872631812096)]} 0
171
2017-07-17 12:29:24.160 INFO (qtp985934102-5160762) [c:UNCLASS
s:shard13
r:core_node81 x:UNCLASS_shard13_replica1]
o.a.s.u.p.LogUpdateProcessorFactory
[UNCLASS_shard13_replica1] webapp=/solr path=/update
params={update.distrib=FROMLEADER&update.chain=add-unknown-
fields-to-the-schema&distrib.from=http://tarvos:9100/solr/
UNCLASS_shard13_replica2/&wt=javabin&version=2}{add=[
COLLECT20003218344436 (1573172872538488832), COLLECT20003218347497
(1573172872620277760), COLLECT20003218351645 (1573172872625520640),
COLLECT20003218356965 (1573172872629714944), COLLECT20003218357775
(1573172872632860672), COLLECT20003218358017 (1573172872646492160),
COLLECT20003218358152 (1573172872650686464), COLLECT20003218359395
(1573172872651735040), COLLECT20003218362571 (1573172872652783616)]} 0
274
2017-07-17 12:29:24.163 INFO (qtp985934102-5161057) [c:UNCLASS
s:shard43
r:core_node108 x:UNCLASS_shard43_replica1]
o.a.s.u.p.LogUpdateProcessorFactory
[UNCLASS_shard43_replica1] webapp=/solr path=/update
params={update.distrib=FROMLEADER&update.chain=add-unknown-
fields-to-the-schema&distrib.from=http://tarvos:9100/solr/
UNCLASS_shard43_replica2/&wt=javabin&version=2}{} 0 0
2017-07-17 12:29:24.164 ERROR (qtp985934102-5161057) [c:UNCLASS
s:shard43
r:core_node108 x:UNCLASS_shard43_replica1] o.a.s.h.RequestHandlerBase
org.apache.solr.common.SolrException: Error logging add
at org.apache.solr.update.TransactionLog.write(
TransactionLog.
java:418)
at org.apache.solr.update.UpdateLog.add(UpdateLog.java:532)
at org.apache.solr.update.UpdateLog.add(UpdateLog.java:516)
at org.apache.solr.update.processor.
DistributedUpdateProcessor.
versionAdd(DistributedUpdateProcessor.java:1113)
at org.apache.solr.update.processor.
DistributedUpdateProcessor.
processAdd(DistributedUpdateProcessor.java:748)
at org.apache.solr.update.processor.
LogUpdateProcessorFactory$L
ogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103)
at org.apache.solr.handler.loader.JavabinLoader$1.update(
Javabi
nLoader.java:98)
at org.apache.solr.client.solrj.request.
JavaBinUpdateRequestCod
ec$1.readOuterMostDocIterator(JavaBinUpdateRequestCodec.java:180)
at org.apache.solr.client.solrj.request.
JavaBinUpdateRequestCod
ec$1.readIterator(JavaBinUpdateRequestCodec.java:136)
at org.apache.solr.common.util.JavaBinCodec.readObject(
JavaBinC
odec.java:306)
at org.apache.solr.common.util.JavaBinCodec.readVal(
JavaBinCode
c.java:251)
at org.apache.solr.client.solrj.request.
JavaBinUpdateRequestCod
ec$1.readNamedList(JavaBinUpdateRequestCodec.java:122)
at org.apache.solr.common.util.JavaBinCodec.readObject(
JavaBinC
odec.java:271)
at org.apache.solr.common.util.JavaBinCodec.readVal(
JavaBinCode
c.java:251)
at org.apache.solr.common.util.JavaBinCodec.unmarshal(
JavaBinCo
dec.java:173)
at org.apache.solr.client.solrj.request.
JavaBinUpdateRequestCod
ec.unmarshal(JavaBinUpdateRequestCodec.java:187)
at org.apache.solr.handler.loader.JavabinLoader.
parseAndLoadDoc
s(JavabinLoader.java:108)
at org.apache.solr.handler.loader.JavabinLoader.load(
JavabinLoa
der.java:55)
at org.apache.solr.handler.UpdateRequestHandler$1.load(
UpdateRe
questHandler.java:97)
at org.apache.solr.handler.ContentStreamHandlerBase.
handleReque
stBody(ContentStreamHandlerBase.java:68)
at org.apache.solr.handler.RequestHandlerBase.
handleRequest(Req
uestHandlerBase.java:173)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2477)
at org.apache.solr.servlet.HttpSolrCall.execute(
HttpSolrCall.
java:723)
at org.apache.solr.servlet.HttpSolrCall.call(
HttpSolrCall.java:
529)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(
SolrDisp
atchFilter.java:361)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(
SolrDisp
atchFilter.java:305)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.
doFilte
r(ServletHandler.java:1691)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(
ServletHan
dler.java:582)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(
Scoped
Handler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(
SecurityHa
ndler.java:548)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(
SessionHandler.java:226)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(
ContextHandler.java:1180)
at org.eclipse.jetty.servlet.ServletHandler.doScope(
ServletHand
ler.java:512)
at org.eclipse.jetty.server.session.SessionHandler.doScope(
SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(
ContextHandler.java:1112)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(
Scoped
Handler.java:141)
at org.eclipse.jetty.server.handler.
ContextHandlerCollection.ha
ndle(ContextHandlerCollection.java:213)
at org.eclipse.jetty.server.handler.HandlerCollection.
handle(
HandlerCollection.java:119)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(
Handl
erWrapper.java:134)
at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(
Rewr
iteHandler.java:335)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(
Handl
erWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:534)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.
java:320)
at org.eclipse.jetty.server.HttpConnection.onFillable(
HttpConne
ction.java:251)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.
succeeded(AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.
java:95)
at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(
SelectChann
elEndPoint.java:93)
at org.eclipse.jetty.util.thread.
strategy.ExecuteProduceConsume
.executeProduceConsume(ExecuteProduceConsume.java:303)
at org.eclipse.jetty.util.thread.
strategy.ExecuteProduceConsume
.produceConsume(ExecuteProduceConsume.java:148)
at org.eclipse.jetty.util.thread.
strategy.ExecuteProduceConsume
.run(ExecuteProduceConsume.java:136)
at org.eclipse.jetty.util.thread.
QueuedThreadPool.runJob(Queued
ThreadPool.java:671)
at org.eclipse.jetty.util.thread.
QueuedThreadPool$2.run(QueuedT
hreadPool.java:589)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.hadoop.ipc.RemoteException(java.io.
IOException):
File /solr6.6.0/UNCLASS/core_node108/data/tlog/tlog.
0000000000000006211
could only be replicated to 0 nodes instead of minReplication (=1).
There
are 40 datanode(s) running and no node(s) are excluded in this
operation.
at org.apache.hadoop.hdfs.server.
blockmanagement.BlockManager.c
hooseTarget4NewBlock(BlockManager.java:1622)
at org.apache.hadoop.hdfs.server.
namenode.FSNamesystem.getAddit
ionalBlock(FSNamesystem.java:3351)
at org.apache.hadoop.hdfs.server.
namenode.NameNodeRpcServer.add
Block(NameNodeRpcServer.java:683)
at org.apache.hadoop.hdfs.server.
namenode.AuthorizationProvider
ProxyClientProtocol.addBlock(AuthorizationProviderProxyClien
tProtocol.java:214)
at org.apache.hadoop.hdfs.protocolPB.
ClientNamenodeProtocolServ
erSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSide
TranslatorPB.java:495)
at org.apache.hadoop.hdfs.protocol.proto.
ClientNamenodeProtocol
Protos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNam
enodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$
ProtoBufRpcIn
voker.call(ProtobufRpcEngine.java:617)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:
2216)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:
2212)
at java.security.AccessController.doPrivileged(Native
Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(
UserGro
upInformation.java:1920)
at org.apache.hadoop.ipc.Server$
Handler.run(Server.java:2210)
at org.apache.hadoop.ipc.Client.call(Client.java:1475)
at org.apache.hadoop.ipc.Client.call(Client.java:1412)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(
ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy11.addBlock(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.
ClientNamenodeProtocolTran
slatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:418)
at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown
Source)
at sun.reflect.DelegatingMethodAccessorImpl.
invoke(DelegatingMe
thodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.
invokeMeth
od(RetryInvocationHandler.java:191)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(
Ret
ryInvocationHandler.java:102)
at com.sun.proxy.$Proxy12.addBlock(Unknown Source)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.
locateFo
llowingBlock(DFSOutputStream.java:1459)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.
nextBloc
kOutputStream(DFSOutputStream.java:1255)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(
DFSOutputStream.java:449)
2017-07-17 12:29:24.164 ERROR (qtp985934102-5161057) [c:UNCLASS
s:shard43
r:core_node108 x:UNCLASS_shard43_replica1] o.a.s.s.HttpSolrCall null:
org.apache.solr.common.SolrException: Error logging add
at org.apache.solr.update.TransactionLog.write(
TransactionLog.
java:418)
at org.apache.solr.update.UpdateLog.add(UpdateLog.java:532)
at org.apache.solr.update.UpdateLog.add(UpdateLog.java:516)
at org.apache.solr.update.processor.
DistributedUpdateProcessor.
versionAdd(DistributedUpdateProcessor.java:1113)
at org.apache.solr.update.processor.
DistributedUpdateProcessor.
processAdd(DistributedUpdateProcessor.java:748)
at org.apache.solr.update.processor.
LogUpdateProcessorFactory$L
ogUpdateProcessor.processAdd(LogUpdateProcessorFactory.java:103)
at org.apache.solr.handler.loader.JavabinLoader$1.update(
Javabi
nLoader.java:98)
at org.apache.solr.client.solrj.request.
JavaBinUpdateRequestCod
ec$1.readOuterMostDocIterator(JavaBinUpdateRequestCodec.java:180)
at org.apache.solr.client.solrj.request.
JavaBinUpdateRequestCod
ec$1.readIterator(JavaBinUpdateRequestCodec.java:136)
at org.apache.solr.common.util.JavaBinCodec.readObject(
JavaBinC
odec.java:306)
at org.apache.solr.common.util.JavaBinCodec.readVal(
JavaBinCode
c.java:251)
at org.apache.solr.client.solrj.request.
JavaBinUpdateRequestCod
ec$1.readNamedList(JavaBinUpdateRequestCodec.java:122)
at org.apache.solr.common.util.JavaBinCodec.readObject(
JavaBinC
odec.java:271)
at org.apache.solr.common.util.JavaBinCodec.readVal(
JavaBinCode
c.java:251)
at org.apache.solr.common.util.JavaBinCodec.unmarshal(
JavaBinCo
dec.java:173)
at org.apache.solr.client.solrj.request.
JavaBinUpdateRequestCod
ec.unmarshal(JavaBinUpdateRequestCodec.java:187)
at org.apache.solr.handler.loader.JavabinLoader.
parseAndLoadDoc
s(JavabinLoader.java:108)
at org.apache.solr.handler.loader.JavabinLoader.load(
JavabinLoa
der.java:55)
at org.apache.solr.handler.UpdateRequestHandler$1.load(
UpdateRe
questHandler.java:97)
at org.apache.solr.handler.ContentStreamHandlerBase.
handleReque
stBody(ContentStreamHandlerBase.java:68)
at org.apache.solr.handler.RequestHandlerBase.
handleRequest(Req
uestHandlerBase.java:173)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2477)
at org.apache.solr.servlet.HttpSolrCall.execute(
HttpSolrCall.
java:723)
at org.apache.solr.servlet.HttpSolrCall.call(
HttpSolrCall.java:
529)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(
SolrDisp
atchFilter.java:361)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(
SolrDisp
atchFilter.java:305)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.
doFilte
r(ServletHandler.java:1691)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(
ServletHan
dler.java:582)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(
Scoped
Handler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(
SecurityHa
ndler.java:548)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(
SessionHandler.java:226)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(
ContextHandler.java:1180)
at org.eclipse.jetty.servlet.ServletHandler.doScope(
ServletHand
ler.java:512)
at org.eclipse.jetty.server.session.SessionHandler.doScope(
SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(
ContextHandler.java:1112)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(
Scoped
Handler.java:141)
at org.eclipse.jetty.server.handler.
ContextHandlerCollection.ha
ndle(ContextHandlerCollection.java:213)
at org.eclipse.jetty.server.handler.HandlerCollection.
handle(
HandlerCollection.java:119)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(
Handl
erWrapper.java:134)
at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(
Rewr
iteHandler.java:335)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(
Handl
erWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:534)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.
java:320)
at org.eclipse.jetty.server.HttpConnection.onFillable(
HttpConne
ction.java:251)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.
succeeded(AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.
java:95)
at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(
SelectChann
elEndPoint.java:93)
at org.eclipse.jetty.util.thread.
strategy.ExecuteProduceConsume
.executeProduceConsume(ExecuteProduceConsume.java:303)
at org.eclipse.jetty.util.thread.
strategy.ExecuteProduceConsume
.produceConsume(ExecuteProduceConsume.java:148)
at org.eclipse.jetty.util.thread.
strategy.ExecuteProduceConsume
.run(ExecuteProduceConsume.java:136)
at org.eclipse.jetty.util.thread.
QueuedThreadPool.runJob(Queued
ThreadPool.java:671)
at org.eclipse.jetty.util.thread.
QueuedThreadPool$2.run(QueuedT
hreadPool.java:589)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.hadoop.ipc.RemoteException(java.io.
IOException):
File /solr6.6.0/UNCLASS/core_node108/data/tlog/tlog.
0000000000000006211
could only be replicated to 0 nodes instead of minReplication (=1).
There
are 40 datanode(s) running and no node(s) are excluded in this
operation.
at org.apache.hadoop.hdfs.server.
blockmanagement.BlockManager.c
hooseTarget4NewBlock(BlockManager.java:1622)
at org.apache.hadoop.hdfs.server.
namenode.FSNamesystem.getAddit
ionalBlock(FSNamesystem.java:3351)
at org.apache.hadoop.hdfs.server.
namenode.NameNodeRpcServer.add
Block(NameNodeRpcServer.java:683)
at org.apache.hadoop.hdfs.server.
namenode.AuthorizationProvider
ProxyClientProtocol.addBlock(AuthorizationProviderProxyClien
tProtocol.java:214)
at org.apache.hadoop.hdfs.protocolPB.
ClientNamenodeProtocolServ
erSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSide
TranslatorPB.java:495)
at org.apache.hadoop.hdfs.protocol.proto.
ClientNamenodeProtocol
Protos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNam
enodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$
ProtoBufRpcIn
voker.call(ProtobufRpcEngine.java:617)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1073)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:
2216)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:
2212)
at java.security.AccessController.doPrivileged(Native
Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(
UserGro
upInformation.java:1920)
at org.apache.hadoop.ipc.Server$
Handler.run(Server.java:2210)
at org.apache.hadoop.ipc.Client.call(Client.java:1475)
at org.apache.hadoop.ipc.Client.call(Client.java:1412)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(
ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy11.addBlock(Unknown Source)
at org.apache.hadoop.hdfs.protocolPB.
ClientNamenodeProtocolTran
slatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:418)
at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown
Source)
at sun.reflect.DelegatingMethodAccessorImpl.
invoke(DelegatingMe
thodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.
invokeMeth
od(RetryInvocationHandler.java:191)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(
Ret
ryInvocationHandler.java:102)
at com.sun.proxy.$Proxy12.addBlock(Unknown Source)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.
locateFo
llowingBlock(DFSOutputStream.java:1459)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.
nextBloc
kOutputStream(DFSOutputStream.java:1255)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(
DFSOutputStream.java:449)
2017-07-17 12:29:24.187 INFO
(zkCallback-5-thread-144-processing-n:juliet:9100_solr)
[ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent
state:SyncConnected type:NodeDataChanged
path:/collections/UNCLASS/state.json]
for collection [UNCLASS] has occurred - updating... (live nodes size:
[45])
On the client side, the error looks like:
2017-07-16 19:03:16,118 WARN
[com.ngc.bigdata.ie_solrindexer.IndexDocument]
Indexing error: org.apache.solr.client.solrj.i
mpl.CloudSolrClient$RouteException: Error from server at
http://leda:9100/solr/UNCLASS_shard44_replica3: Exception writing
document id COLLECT10086453202 to the index; possible analysis error.
for
collection: UNCLASS
org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException:
Error
from server at http://leda:9100/solr/UNCLASS_shard44_replica3:
Exception
writing document id COLLECT10086453202 to the index; possible analysis
error.
at org.apache.solr.client.solrj.impl.CloudSolrClient.
directUpda
te(CloudSolrClient.java:819)
at org.apache.solr.client.solrj.impl.CloudSolrClient.
sendReques
t(CloudSolrClient.java:1263)
at org.apache.solr.client.solrj.impl.CloudSolrClient.
requestWit
hRetryOnStaleState(CloudSolrClient.java:1134)
at org.apache.solr.client.solrj.
impl.CloudSolrClient.request(Cl
oudSolrClient.java:1073)
at org.apache.solr.client.solrj.SolrRequest.process(
SolrRequest
.java:160)
at org.apache.solr.client.solrj.SolrClient.add(SolrClient.
java:
106)
at org.apache.solr.client.solrj.SolrClient.add(SolrClient.
java:
71)
at org.apache.solr.client.solrj.SolrClient.add(SolrClient.
java:
85)
at com.ngc.bigdata.ie_solrindexer.IndexDocument.
indexSolrDocs(
IndexDocument.java:959)
at com.ngc.bigdata.ie_solrindexer.IndexDocument.index(
IndexDocument.java:236)
at com.ngc.bigdata.ie_solrindexer.
SolrIndexerProcessor.doWork(S
olrIndexerProcessor.java:63)
at com.ngc.intelenterprise.intelentutil.utils.Processor.run(
Processor.java:140)
at com.ngc.intelenterprise.intelentutil.jms.
IntelEntQueueProc.
process(IntelEntQueueProc.java:208)
at org.apache.camel.processor.DelegateSyncProcessor.process(
Del
egateSyncProcessor.java:63)
at org.apache.camel.management.InstrumentationProcessor.
process
(InstrumentationProcessor.java:77)
at org.apache.camel.processor.RedeliveryErrorHandler.
process(Re
deliveryErrorHandler.java:460)
at org.apache.camel.processor.CamelInternalProcessor.
process(Ca
melInternalProcessor.java:190)
at org.apache.camel.processor.CamelInternalProcessor.
process(Ca
melInternalProcessor.java:190)
at org.apache.camel.component.seda.SedaConsumer.
sendToConsumers
(SedaConsumer.java:298)
at org.apache.camel.component.seda.SedaConsumer.doRun(
SedaConsu
mer.java:207)
at org.apache.camel.component.seda.SedaConsumer.run(
SedaConsume
r.java:154)
at java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPool
Executor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoo
lExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:
Error from server at http://leda:9100/solr/UNCLASS_shard44_replica3:
Exception writing document id COLLECT10086453202 to the index;
possible
analysis error.
at org.apache.solr.client.solrj.impl.HttpSolrClient.
executeMeth
od(HttpSolrClient.java:610)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(
Htt
pSolrClient.java:279)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(
Htt
pSolrClient.java:268)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.
doRequest
(LBHttpSolrClient.java:447)
at org.apache.solr.client.solrj.
impl.LBHttpSolrClient.request(L
BHttpSolrClient.java:388)
at org.apache.solr.client.solrj.impl.CloudSolrClient.lambda$
dir
ectUpdate$0(CloudSolrClient.java:796)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.solr.common.util.ExecutorUtil$
MDCAwareThreadPoolE
xecutor.lambda$execute$0(ExecutorUtil.java:229)
... 3 more
2017-07-16 19:03:16,134 ERROR
[com.ngc.bigdata.ie_solrindexer.IndexDocument]
Error indexing: org.apache.solr.client.solrj.i
mpl.CloudSolrClient$RouteException: Error from server at
http://leda:9100/solr/UNCLASS_shard44_replica3: Exception writing
document id COLLECT10086453202 to the index; possible analysis error.
for
collection: UNCLASS.
2017-07-16 19:03:16,135 ERROR
[com.ngc.bigdata.ie_solrindexer.IndexDocument]
Exception during indexing: org.apache.solr.client.solrj.i
mpl.CloudSolrClient$RouteException: Error from server at
http://leda:9100/solr/UNCLASS_shard44_replica3: Exception writing
document id COLLECT10086453202 to the index; possible analysis error.
I can fire them back up, but they only run for a short time before
getting more indexing errors. Several of the nodes show as down in
the
cloud view. Any help would be appreciated! Thank you!
-Joe
---
This email has been checked for viruses by AVG.
http://www.avg.com
