Hi Sriram, People hit timeout issues when trying to split larger collections/shards. This is because of the timing out of internal calls.
About the UI showing everything as active, that's another open issue( SOLR-4777 <https://issues.apache.org/jira/browse/SOLR-4777>). You should instead check the clusterstate to see the state of the slices (both parent and child). What is the current state of the slices? On Fri, Jan 3, 2014 at 2:39 PM, sriram <srira...@ask.com> wrote: > Hi, > > I am using solr cloud 4.6 with 2 shards. I have loaded close to 13Million > data in them. When I tried to give split shard, I am getting the following > errors in the log, > > INFO - 2014-01-03 14:17:45.357; > org.apache.solr.cloud.DistributedQueue$LatchChildWatcher; Watcher fired on > path: /overseer/collection-queue-work/qnr-0000000002 state: SyncConnected > type NodeDataChanged > INFO - 2014-01-03 14:17:45.359; > org.apache.solr.cloud.DistributedQueue$LatchChildWatcher; Watcher fired on > path: /overseer/collection-queue-work state: SyncConnected type > NodeChildrenChanged > INFO - 2014-01-03 14:17:45.360; > org.apache.solr.cloud.OverseerCollectionProcessor; Overseer Collection > Processor: Message id:/overseer/collection-queue-work/qn-0000000002 > complete, > > response:{success={null={responseHeader={status=0,QTime=3689},core=collection1_shard1_1_replica1},null={responseHeader={status=0,QTime=3694},core=collection1_shard1_0_replica1},null={responseHeader={status=0,QTime=2004}},null={responseHeader={status=0,QTime=2005}}},failure={null=org.apache.solr.client.solrj.SolrServerException:Timeout > occured while waiting response from server at: > http://<machine:port>/solr},Operation splitshard caused > exception:=org.apache.solr.common.SolrException: SPLITSHARD failed to > invoke > SPLIT core admin command,exception={msg=SPLITSHARD failed to invoke SPLIT > core admin command,rspCode=500}} > INFO - 2014-01-03 14:17:45.365; > org.apache.solr.servlet.SolrDispatchFilter; > [admin] webapp=null path=/admin/collections > params={shard=shard1&action=SPLITSHARD&collection=collection1} status=500 > QTime=15736 > ERROR - 2014-01-03 14:17:45.366; org.apache.solr.common.SolrException; > null:org.apache.solr.common.SolrException: SPLITSHARD failed to invoke > SPLIT > core admin command > at > > org.apache.solr.handler.admin.CollectionsHandler.handleResponse(CollectionsHandler.java:200) > at > > org.apache.solr.handler.admin.CollectionsHandler.handleSplitShardAction(CollectionsHandler.java:422) > at > > org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:158) > at > > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135) > at > > org.apache.solr.servlet.SolrDispatchFilter.handleAdminRequest(SolrDispatchFilter.java:662) > at > > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257) > at > > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:197) > at > > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419) > at > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455) > at > > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) > at > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557) > at > > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231) > at > > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075) > at > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384) > at > > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) > at > > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009) > at > > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) > at > > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255) > at > > org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154) > at > > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) > at org.eclipse.jetty.server.Server.handle(Server.java:368) > at > > org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489) > at > > org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53) > at > > org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:942) > at > > org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1004) > at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:640) > at > org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) > at > > org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72) > at > > org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264) > at > > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) > at > > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) > at java.lang.Thread.run(Thread.java:619) > > > Before this exception, I am seeing couple of timeouts : > > ERROR - 2014-01-03 14:17:45.350; org.apache.solr.common.SolrException; > Collection splitshard of splitshard > failed:org.apache.solr.common.SolrException: SPLITSHARD failed to invoke > SPLIT core admin command > at > > org.apache.solr.cloud.OverseerCollectionProcessor.collectShardResponses(OverseerCollectionProcessor.java:978) > at > > org.apache.solr.cloud.OverseerCollectionProcessor.splitShard(OverseerCollectionProcessor.java:776) > at > > org.apache.solr.cloud.OverseerCollectionProcessor.processMessage(OverseerCollectionProcessor.java:239) > at > > org.apache.solr.cloud.OverseerCollectionProcessor.run(OverseerCollectionProcessor.java:171) > at java.lang.Thread.run(Thread.java:619) > Caused by: org.apache.solr.client.solrj.SolrServerException: Timeout > occured > while waiting response from server at: http://<machine:port>/solr > at > > org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:499) > at > > org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:197) > at > > org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:156) > at > > org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:118) > at > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) > at > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > ... 1 more > Caused by: java.net.SocketTimeoutException: Read timed out > at java.net.SocketInputStream.socketRead0(Native Method) > at java.net.SocketInputStream.read(SocketInputStream.java:129) > at > > org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166) > at > > org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90) > at > > org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:281) > at > > org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:92) > at > > org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:61) > at > > org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:254) > at > > org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:289) > at > > org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:252) > at > > org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:191) > at > > org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:300) > at > > org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:127) > at > > org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:715) > at > > org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:520) > at > > org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906) > at > > org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805) > at > > org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784) > at > > org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:393) > ... 11 more > > > > INFO - 2014-01-03 14:17:35.347; org.apache.solr.update.SolrIndexSplitter; > SolrIndexSplitter: partitions=2 segments=26 > INFO - 2014-01-03 14:17:38.131; org.apache.solr.update.SolrIndexSplitter; > SolrIndexSplitter: partition #0 range=80000000-bfffffff > ERROR - 2014-01-03 14:17:45.342; > org.apache.solr.cloud.OverseerCollectionProcessor; Error from shard: > <mahine:port>/solr > org.apache.solr.client.solrj.SolrServerException: Timeout occured while > waiting response from server at: http://<machine:port>/solr > at > > org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:499) > at > > org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:197) > at > > org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:156) > at > > org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:118) > at > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) > at > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:619) > Caused by: java.net.SocketTimeoutException: Read timed out > at java.net.SocketInputStream.socketRead0(Native Method) > at java.net.SocketInputStream.read(SocketInputStream.java:129) > at > > org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166) > at > > org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90) > at > > org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:281) > at > > org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:92) > at > > org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:61) > at > > org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:254) > at > > org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:289) > at > > org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:252) > at > > org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:191) > at > > org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:300) > at > > org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:127) > at > > org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:715) > at > > org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:520) > at > > org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906) > > > Sorry for long stack trace. > > Though it throws these exceptions, it still creates the cores and splits > the > index based on hashrange. The UI says both the shard1 as well as shard1_0 > and shard1_1 as active. So even though I have 2 split shards orginated from > shard1, I am not able to delete shard1 as it is still present as active. > > I initially explored about https://issues.apache.org/jira/browse/SOLR-5314 > and https://issues.apache.org/jira/browse/SOLR-5214. But in my case since > it > looks more like timeout is triggering the issue, I upgraded from 4.5 to 4.6 > as it was mentioned in SOLR-5314 that it is fixed in 4.6. But I am still > getting this issue. Any help regarding this would be greatly appreciated > > > > > -- > View this message in context: > http://lucene.472066.n3.nabble.com/SPLITSHARD-failed-to-invoke-SPLIT-core-admin-command-tp4109431.html > Sent from the Solr - User mailing list archive at Nabble.com. > -- Anshum Gupta http://www.anshumgupta.net
