Hi,
I am using solr cloud 4.6 with 2 shards. I have loaded close to 13Million
data in them. When I tried to give split shard, I am getting the following
errors in the log,
INFO - 2014-01-03 14:17:45.357;
org.apache.solr.cloud.DistributedQueue$LatchChildWatcher; Watcher fired on
path: /overseer/collection-queue-work/qnr-0000000002 state: SyncConnected
type NodeDataChanged
INFO - 2014-01-03 14:17:45.359;
org.apache.solr.cloud.DistributedQueue$LatchChildWatcher; Watcher fired on
path: /overseer/collection-queue-work state: SyncConnected type
NodeChildrenChanged
INFO - 2014-01-03 14:17:45.360;
org.apache.solr.cloud.OverseerCollectionProcessor; Overseer Collection
Processor: Message id:/overseer/collection-queue-work/qn-0000000002
complete,
response:{success={null={responseHeader={status=0,QTime=3689},core=collection1_shard1_1_replica1},null={responseHeader={status=0,QTime=3694},core=collection1_shard1_0_replica1},null={responseHeader={status=0,QTime=2004}},null={responseHeader={status=0,QTime=2005}}},failure={null=org.apache.solr.client.solrj.SolrServerException:Timeout
occured while waiting response from server at:
http://<machine:port>/solr},Operation splitshard caused
exception:=org.apache.solr.common.SolrException: SPLITSHARD failed to invoke
SPLIT core admin command,exception={msg=SPLITSHARD failed to invoke SPLIT
core admin command,rspCode=500}}
INFO - 2014-01-03 14:17:45.365; org.apache.solr.servlet.SolrDispatchFilter;
[admin] webapp=null path=/admin/collections
params={shard=shard1&action=SPLITSHARD&collection=collection1} status=500
QTime=15736
ERROR - 2014-01-03 14:17:45.366; org.apache.solr.common.SolrException;
null:org.apache.solr.common.SolrException: SPLITSHARD failed to invoke SPLIT
core admin command
at
org.apache.solr.handler.admin.CollectionsHandler.handleResponse(CollectionsHandler.java:200)
at
org.apache.solr.handler.admin.CollectionsHandler.handleSplitShardAction(CollectionsHandler.java:422)
at
org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:158)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
at
org.apache.solr.servlet.SolrDispatchFilter.handleAdminRequest(SolrDispatchFilter.java:662)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:197)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.eclipse.jetty.server.Server.handle(Server.java:368)
at
org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
at
org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
at
org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:942)
at
org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1004)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:640)
at
org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
at
org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
at
org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:619)
Before this exception, I am seeing couple of timeouts :
ERROR - 2014-01-03 14:17:45.350; org.apache.solr.common.SolrException;
Collection splitshard of splitshard
failed:org.apache.solr.common.SolrException: SPLITSHARD failed to invoke
SPLIT core admin command
at
org.apache.solr.cloud.OverseerCollectionProcessor.collectShardResponses(OverseerCollectionProcessor.java:978)
at
org.apache.solr.cloud.OverseerCollectionProcessor.splitShard(OverseerCollectionProcessor.java:776)
at
org.apache.solr.cloud.OverseerCollectionProcessor.processMessage(OverseerCollectionProcessor.java:239)
at
org.apache.solr.cloud.OverseerCollectionProcessor.run(OverseerCollectionProcessor.java:171)
at java.lang.Thread.run(Thread.java:619)
Caused by: org.apache.solr.client.solrj.SolrServerException: Timeout occured
while waiting response from server at: http://<machine:port>/solr
at
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:499)
at
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:197)
at
org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:156)
at
org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:118)
at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
... 1 more
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at
org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)
at
org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90)
at
org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:281)
at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:92)
at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:61)
at
org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:254)
at
org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:289)
at
org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:252)
at
org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:191)
at
org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:300)
at
org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:127)
at
org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:715)
at
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:520)
at
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
at
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
at
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
at
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:393)
... 11 more
INFO - 2014-01-03 14:17:35.347; org.apache.solr.update.SolrIndexSplitter;
SolrIndexSplitter: partitions=2 segments=26
INFO - 2014-01-03 14:17:38.131; org.apache.solr.update.SolrIndexSplitter;
SolrIndexSplitter: partition #0 range=80000000-bfffffff
ERROR - 2014-01-03 14:17:45.342;
org.apache.solr.cloud.OverseerCollectionProcessor; Error from shard:
<mahine:port>/solr
org.apache.solr.client.solrj.SolrServerException: Timeout occured while
waiting response from server at: http://<machine:port>/solr
at
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:499)
at
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:197)
at
org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:156)
at
org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:118)
at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at
org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)
at
org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90)
at
org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:281)
at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:92)
at
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:61)
at
org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:254)
at
org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:289)
at
org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:252)
at
org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:191)
at
org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:300)
at
org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:127)
at
org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:715)
at
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:520)
at
org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
Sorry for long stack trace.
Though it throws these exceptions, it still creates the cores and splits the
index based on hashrange. The UI says both the shard1 as well as shard1_0
and shard1_1 as active. So even though I have 2 split shards orginated from
shard1, I am not able to delete shard1 as it is still present as active.
I initially explored about https://issues.apache.org/jira/browse/SOLR-5314
and https://issues.apache.org/jira/browse/SOLR-5214. But in my case since it
looks more like timeout is triggering the issue, I upgraded from 4.5 to 4.6
as it was mentioned in SOLR-5314 that it is fixed in 4.6. But I am still
getting this issue. Any help regarding this would be greatly appreciated
--
View this message in context:
http://lucene.472066.n3.nabble.com/SPLITSHARD-failed-to-invoke-SPLIT-core-admin-command-tp4109431.html
Sent from the Solr - User mailing list archive at Nabble.com.
