Hello,
Solr Set up
Solr 5.4.1, Zookeeper 3.4.6 (5 zookeeper ensemble)
We have one collection which has multiple shards (two shards for each week).
Each shard has a leader and a replica. We only write to the latest week- two
shards (four cores) which we refer to a ‘hot cores’. The rest, ‘cold cores’
are for queries. We have multiple solr processes running on an instance-
currently 5 each with a 15Gb Heap (there is 122G available memory). As the
index grows as the week goes on the heap size starts low and increase to around
9/10Gb. The index size on each core ends up around 8 million docs, 6.5Gb which
are stored on 40Gb drives. The zookeeper timeout is 60Secs.
The issue:
We are experiencing issues with connectivity and have started seeing errors
messages about being unable to connect to zookeeper. Most of the time solr
recovers itself after a while but we are seeing these ‘blips’ more and more
often with the last ‘blip’ ending up with manually restarting the hot cores. So
far this has only been seen on one shard at a time. All other shards in the
cluster don’t have an issue.
There is nothing in the zookeeper log. Below are the solr logs for the last
‘blip’.
I’ve looked at the heap size and its not hitting 15Gb (max around 11Gb). At
around the time of the blip the GC is 40sec, which is not over the timeout but
is however much larger than we normally see.
These blips are happening towards the end of the week when the index size gets
larger.
I’m not sure what is going on, is this a zookeeper issue or solr? What would be
causing solr to lose connection with zookeeper if it’s not the timeout? We have
checked the network and it doesn’t indicate a network issue.
Any suggests would be useful.
Error Logs for core A
2016-10-08 18:45:36.617 WARN (qtp697960108-32664) [c:xxx s:20161003_A
r:20161003_A54130 x:xxx] o.e.j.h.HttpParser badMessage:
java.lang.IllegalStateException: too much data after closed for
HttpChannelOverHttp@1b87c2cf{r=1370,c=false,a=IDLE,uri=-}
2016-10-08 18:45:36.717 WARN
(updateExecutor-2-thread-8523-processing-n:x.x.x.x:8987_solr x:xxx s:20161003_A
c:xxx r:20161003_A54130) [c:xxx s:20161003_A r:20161003_A54130 x:xxx]
o.a.s.c.ZkController Unable to read
/collections/xxx/leader_initiated_recovery/20161003_A/20161003_A54130 due to:
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode =
ConnectionLoss for
/collections/xxx/leader_initiated_recovery/20161003_A/20161003_A54130
2016-10-08 18:45:44.907 ERROR
(updateExecutor-2-thread-8523-processing-n:x.x.x.x:8987_solr x:xxx s:20161003_A
c:xxx r:20161003_A54130) [c:xxx s:20161003_A r:20161003_A54130 x:xxx]
o.a.s.u.PeerSync PeerSync: core=xxx url=http://x.x.x.x:8987/solr ERROR, update
log not in ACTIVE or REPLAY state. FSUpdateLog{state=BUFFERING,
tlog=tlog{file=/solrLog_8987/tlog/tlog.0000000000000011469 refcount=1}}
2016-10-08 18:45:44.908 WARN
(updateExecutor-2-thread-8523-processing-n:x.x.x.x:8987_solr x:xxx s:20161003_A
c:xxx r:20161003_A54130) [c:xxx s:20161003_A r:20161003_A54130 x:xxx]
o.a.s.u.PeerSync PeerSync: core=xxx url=http://x.x.x.x:8987/solr too many
updates received since start - startingUpdates no longer overlaps with our
currentUpdates
2016-10-08 18:47:25.772 WARN
(updateExecutor-2-thread-8523-processing-n:x.x.x.x:8987_solr x:xxx s:20161003_A
c:xxx r:20161003_A54130) [c:xxx s:20161003_A r:20161003_A54130 x:xxx]
o.a.s.h.IndexFetcher File _1ftq.si did not match. expected checksum is
4254234714 and actual is checksum 2090625558. expected length is 422 and actual
length is 422
2016-10-08 18:47:26.286 WARN
(zkCallback-3-thread-76-processing-n:x.x.x.x:8987_solr-EventThread) [ ]
o.a.s.c.RecoveryStrategy Stopping recovery for core=xxx
coreNodeName=20161003_A54130
2016-10-08 18:47:54.935 WARN
(updateExecutor-2-thread-8523-processing-n:x.x.x.x:8987_solr x:xxx s:20161003_A
c:xxx r:20161003_A54130) [c:xxx s:20161003_A r:20161003_A54130 x:xxx]
o.a.s.h.IndexFetcher File _1ftq.si did not match. expected checksum is
4254234714 and actual is checksum 2090625558. expected length is 422 and actual
length is 422
2016-10-08 18:47:54.939 WARN
(updateExecutor-2-thread-8523-processing-n:x.x.x.x:8987_solr x:xxx s:20161003_A
c:xxx r:20161003_A54130) [c:xxx s:20161003_A r:20161003_A54130 x:xxx]
o.a.s.h.IndexFetcher File _1ftq.cfs did not match. expected checksum is
3006669569 and actual is checksum 3691917. expected length is 35641114 and
actual length is 8402832
2016-10-08 18:47:55.084 WARN
(updateExecutor-2-thread-8523-processing-n:x.x.x.x:8987_solr x:xxx s:20161003_A
c:xxx r:20161003_A54130) [c:xxx s:20161003_A r:20161003_A54130 x:xxx]
o.a.s.h.IndexFetcher File _1ftq.cfe did not match. expected checksum is
3692862263 and actual is checksum 3783720915. expected length is 289 and actual
length is 289
2016-10-08 18:47:55.399 WARN
(updateExecutor-2-thread-8523-processing-n:x.x.x.x:8987_solr x:xxx s:20161003_A
c:xxx r:20161003_A54130) [c:xxx s:20161003_A r:20161003_A54130 x:xxx]
o.a.s.h.IndexFetcher File _1fti.cfs did not match. expected checksum is
3751057985 and actual is checksum 396270944. expected length is 9214354 and
actual length is 7147563
…
We then start getting errors
2016-10-08 18:51:31.202 WARN
(coreZkRegister-1-thread-8-processing-n:x.x.x.x:8987_solr x:xxx s:20161003_A
c:xxx r:20161003_A54130) [c:xxx s:20161003_A r:20161003_A54130 x:xxx]
o.a.s.c.RecoveryStrategy Stopping recovery for core=xxx
coreNodeName=20161003_A54130
2016-10-08 18:51:31.206 ERROR
(updateExecutor-2-thread-8523-processing-n:x.x.x.x:8987_solr x:xxx s:20161003_A
c:xxx r:20161003_A54130) [c:xxx s:20161003_A r:20161003_A54130 x:xxx]
o.a.s.c.RecoveryStrategy Error while trying to recover.
core=xxx:java.util.concurrent.ExecutionException:
org.apache.solr.client.solrj.SolrServerException: IOException occured when
talking to server at: http://x.x.x.x:8987/solr
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at
org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(RecoveryStrategy.java:607)
at
org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:364)
at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:226)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:232)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.client.solrj.SolrServerException: IOException
occured when talking to server at: http://x.x.x.x:8987/solr
at
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:589)
at
org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:285)
at
org.apache.solr.client.solrj.impl.HttpSolrClient$1.call(HttpSolrClient.java:281)
... 5 more
Caused by: java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
2016-10-08 18:51:42.896 WARN
(coreZkRegister-1-thread-9-processing-n:x.x.x.x:8987_solr x:xxx s:20161003_A
c:xxx r:20161003_A54130) [c:xxx s:20161003_A r:20161003_A54130 x:xxx]
o.a.s.c.RecoveryStrategy Stopping recovery for core=xxx
coreNodeName=20161003_A54130
2016-10-08 18:52:16.552 ERROR
(updateExecutor-2-thread-8523-processing-n:x.x.x.x:8987_solr x:xxx s:20161003_A
c:xxx r:20161003_A54130) [c:xxx s:20161003_A r:20161003_A54130 x:xxx]
o.a.s.u.PeerSync PeerSync: core=xxx url=http://x.x.x.x:8987/solr ERROR, update
log not in ACTIVE or REPLAY state. FSUpdateLog{state=BUFFERING,
tlog=tlog{file=/solrLog_8987/tlog/tlog.0000000000000011469 refcount=1}}
2016-10-08 18:52:16.553 WARN
(updateExecutor-2-thread-8523-processing-n:x.x.x.x:8987_solr x:xxx s:20161003_A
c:xxx r:20161003_A54130) [c:xxx s:20161003_A r:20161003_A54130 x:xxx]
o.a.s.u.PeerSync PeerSync: core=xxx url=http://x.x.x.x:8987/solr too many
updates received since start - startingUpdates no longer overlaps with our
currentUpdates
2016-10-08 19:04:40.098 WARN
(zkCallback-3-thread-196-processing-n:x.x.x.x:8987_solr) [c:xxx s:20161003_A
r:20161003_A54130 x:xxx] o.a.s.c.RecoveryStrategy Stopping recovery for
core=xxx coreNodeName=20161003_A54130
2016-10-08 19:05:11.864 WARN
(updateExecutor-2-thread-8523-processing-n:x.x.x.x:8987_solr x:xxx s:20161003_A
c:xxx r:20161003_A54130) [c:xxx s:20161003_A r:20161003_A54130 x:xxx]
o.a.s.c.SolrCore [xxx] PERFORMANCE WARNING: Overlapping onDeckSearchers=2
2016-10-08 19:07:40.446 WARN
(zkCallback-3-thread-76-processing-n:x.x.x.x:8987_solr-EventThread) [ ]
o.a.s.c.RecoveryStrategy Stopping recovery for core=xxx
coreNodeName=20161003_A54130
2016-10-08 19:10:22.427 ERROR (qtp697960108-32836) [c:xxx s:20161003_A
r:20161003_A54130 x:xxx] o.a.s.h.RequestHandlerBase
org.apache.solr.common.SolrException: No registered leader was found after
waiting for 4000ms , collection: xxx slice: 20161003_A
at
org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:643)
at
org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:629)
Error Logs for core B
2016-10-08 18:45:36.618 ERROR
(updateExecutor-2-thread-8085-processing-http:////x.x.x.x:8987//solr//xxx x:xxx
r:20161003_A54129 n:x.x.x.x:8987_solr s:20161003_A c:xxx) [c:xxx s:20161003_A
r:20161003_A54129 x:xxx] o.a.s.u.StreamingSolrClients error
org.apache.http.NoHttpResponseException: x.x.x.x:8987 failed to respond
at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:143)
at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
at
org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
at
org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
at
org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
at
org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
at
org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
at
org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
at
org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
at
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
at
org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.sendUpdateStream(ConcurrentUpdateSolrClient.java:279)
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.run(ConcurrentUpdateSolrClient.java:160)
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:232)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2016-10-08 18:45:36.618 ERROR
(updateExecutor-2-thread-8083-processing-http:////x.x.x.x:8987//solr//xxx x:xxx
r:20161003_A54129 n:x.x.x.x:8987_solr s:20161003_A c:xxx) [c:xxx s:20161003_A
r:20161003_A54129 x:xxx] o.a.s.u.StreamingSolrClients error
org.apache.http.NoHttpResponseException: x.x.x.x:8987 failed to respond
at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:143)
at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
at
org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
at
org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
at
org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
at
org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
at
org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
at
org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
at
org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
at
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
at
org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.sendUpdateStream(ConcurrentUpdateSolrClient.java:279)
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.run(ConcurrentUpdateSolrClient.java:160)
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:232)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2016-10-08 18:45:36.619 WARN (qtp2065530879-34668) [c:xxx s:20161003_A
r:20161003_A54129 x:xxx] o.a.s.u.p.DistributedUpdateProcessor Error sending
update to http://x.x.x.x:8987/solr
org.apache.http.NoHttpResponseException: x.x.x.x:8987 failed to respond
at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:143)
at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
at
org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
2016-10-08 18:45:36.620 WARN (qtp2065530879-34670) [c:xxx s:20161003_A
r:20161003_A54129 x:xxx] o.a.s.u.p.DistributedUpdateProcessor Error sending
update to http://x.x.x.x:8987/solr
org.apache.http.NoHttpResponseException: x.x.x.x:8987 failed to respond
at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:143)
at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
at
org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
at
org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
at
org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
at
org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
at
org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
at
org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
at
org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
at
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
at
org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.sendUpdateStream(ConcurrentUpdateSolrClient.java:279)
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.run(ConcurrentUpdateSolrClient.java:160)
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:232)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2016-10-08 18:45:36.620 ERROR (qtp2065530879-34668) [c:xxx s:20161003_A
r:20161003_A54129 x:xxx] o.a.s.u.p.DistributedUpdateProcessor Setting up to try
to start recovery on replica http://x.x.x.x:8987/solr/xxx/
org.apache.http.NoHttpResponseException: x.x.x.x:8987 failed to respond
at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:143)
at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
at
org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
at
org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
at
org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
at
org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
at
org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
at
org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
at
org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
at
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
at
org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
at
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.sendUpdateStream(ConcurrentUpdateSolrClient.java:279)
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.run(ConcurrentUpdateSolrClient.java:160)
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:232)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2016-10-08 18:45:36.620 ERROR (qtp2065530879-34670) [c:xxx s:20161003_A
r:20161003_A54129 x:xxx] o.a.s.u.p.DistributedUpdateProcessor Setting up to try
to start recovery on replica http://x.x.x.x:8987/solr/xxx/
org.apache.http.NoHttpResponseException: x.x.x.x:8987 failed to respond
at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:143)
at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
at
org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
at
org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
2016-10-08 19:10:22.248 ERROR (qtp2065530879-34820) [c:xxx s:20161003_A
r:20161003_A54129 x:xxx] o.a.s.h.RequestHandlerBase
org.apache.solr.common.SolrException: No registered leader was found after
waiting for 4000ms , collection: xxx slice: 20161003_A
at
org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:643)
at
org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:629)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.setupRequest(DistributedUpdateProcessor.java:360)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.setupRequest(DistributedUpdateProcessor.java:311)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:661)
at
org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:251)
at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:178)
at
org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:95)
at
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:70)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:156)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2073)
at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:658)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:457)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:223)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:181)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
at org.eclipse.jetty.server.Server.handle(Server.java:499)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
at
org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:745)
2016-10-08 19:34:05.557 ERROR
(coreZkRegister-1-thread-15-processing-n:x.x.x.x:8987_solr x:xxx s:20161003_A
c:xxx r:20161003_A54129) [c:xxx s:20161003_A r:20161003_A54129 x:xxx]
o.a.s.c.ZkController Error getting leader from zk
org.apache.solr.common.SolrException: No registered leader was found after
waiting for 1560000ms , collection: xxx slice: 20161003_A
at
org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:643)
at
org.apache.solr.common.cloud.ZkStateReader.getLeaderUrl(ZkStateReader.java:610)
at org.apache.solr.cloud.ZkController.getLeader(ZkController.java:1005)
at org.apache.solr.cloud.ZkController.register(ZkController.java:940)
at
org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:234)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:232)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2016-10-08 19:34:06.101 ERROR
(coreZkRegister-1-thread-14-processing-n:x.x.x.x:8987_solr x:xxx s:20161003_A
c:xxx r:20161003_A54129) [c:xxx s:20161003_A r:20161003_A54129 x:xxx]
o.a.s.c.ZkController Error getting leader from zk
org.apache.solr.common.SolrException: No registered leader was found after
waiting for 1560000ms , collection: xxx slice: 20161003_A
