Also, here's another issue that would help with the timeouts (SOLR-5477<https://issues.apache.org/jira/browse/SOLR-5477> ).
On Fri, Jan 3, 2014 at 3:34 PM, Anshum Gupta <ans...@anshumgupta.net> wrote: > Hi Sriram, > > People hit timeout issues when trying to split larger collections/shards. > This is because of the timing out of internal calls. > > About the UI showing everything as active, that's another open issue( > SOLR-4777 <https://issues.apache.org/jira/browse/SOLR-4777>). You should > instead check the clusterstate to see the state of the slices (both parent > and child). > What is the current state of the slices? > > > > On Fri, Jan 3, 2014 at 2:39 PM, sriram <srira...@ask.com> wrote: > >> Hi, >> >> I am using solr cloud 4.6 with 2 shards. I have loaded close to 13Million >> data in them. When I tried to give split shard, I am getting the following >> errors in the log, >> >> INFO - 2014-01-03 14:17:45.357; >> org.apache.solr.cloud.DistributedQueue$LatchChildWatcher; Watcher fired on >> path: /overseer/collection-queue-work/qnr-0000000002 state: SyncConnected >> type NodeDataChanged >> INFO - 2014-01-03 14:17:45.359; >> org.apache.solr.cloud.DistributedQueue$LatchChildWatcher; Watcher fired on >> path: /overseer/collection-queue-work state: SyncConnected type >> NodeChildrenChanged >> INFO - 2014-01-03 14:17:45.360; >> org.apache.solr.cloud.OverseerCollectionProcessor; Overseer Collection >> Processor: Message id:/overseer/collection-queue-work/qn-0000000002 >> complete, >> >> response:{success={null={responseHeader={status=0,QTime=3689},core=collection1_shard1_1_replica1},null={responseHeader={status=0,QTime=3694},core=collection1_shard1_0_replica1},null={responseHeader={status=0,QTime=2004}},null={responseHeader={status=0,QTime=2005}}},failure={null=org.apache.solr.client.solrj.SolrServerException:Timeout >> occured while waiting response from server at: >> http://<machine:port>/solr},Operation splitshard caused >> exception:=org.apache.solr.common.SolrException: SPLITSHARD failed to >> invoke >> SPLIT core admin command,exception={msg=SPLITSHARD failed to invoke SPLIT >> core admin command,rspCode=500}} >> INFO - 2014-01-03 14:17:45.365; >> org.apache.solr.servlet.SolrDispatchFilter; >> [admin] webapp=null path=/admin/collections >> params={shard=shard1&action=SPLITSHARD&collection=collection1} status=500 >> QTime=15736 >> ERROR - 2014-01-03 14:17:45.366; org.apache.solr.common.SolrException; >> null:org.apache.solr.common.SolrException: SPLITSHARD failed to invoke >> SPLIT >> core admin command >> at >> >> org.apache.solr.handler.admin.CollectionsHandler.handleResponse(CollectionsHandler.java:200) >> at >> >> org.apache.solr.handler.admin.CollectionsHandler.handleSplitShardAction(CollectionsHandler.java:422) >> at >> >> org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:158) >> at >> >> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135) >> at >> >> org.apache.solr.servlet.SolrDispatchFilter.handleAdminRequest(SolrDispatchFilter.java:662) >> at >> >> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:257) >> at >> >> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:197) >> at >> >> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419) >> at >> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455) >> at >> >> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137) >> at >> >> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557) >> at >> >> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231) >> at >> >> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075) >> at >> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384) >> at >> >> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193) >> at >> >> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009) >> at >> >> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) >> at >> >> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255) >> at >> >> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154) >> at >> >> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) >> at org.eclipse.jetty.server.Server.handle(Server.java:368) >> at >> >> org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489) >> at >> >> org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53) >> at >> >> org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:942) >> at >> >> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1004) >> at >> org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:640) >> at >> org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) >> at >> >> org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72) >> at >> >> org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264) >> at >> >> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) >> at >> >> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) >> at java.lang.Thread.run(Thread.java:619) >> >> >> Before this exception, I am seeing couple of timeouts : >> >> ERROR - 2014-01-03 14:17:45.350; org.apache.solr.common.SolrException; >> Collection splitshard of splitshard >> failed:org.apache.solr.common.SolrException: SPLITSHARD failed to invoke >> SPLIT core admin command >> at >> >> org.apache.solr.cloud.OverseerCollectionProcessor.collectShardResponses(OverseerCollectionProcessor.java:978) >> at >> >> org.apache.solr.cloud.OverseerCollectionProcessor.splitShard(OverseerCollectionProcessor.java:776) >> at >> >> org.apache.solr.cloud.OverseerCollectionProcessor.processMessage(OverseerCollectionProcessor.java:239) >> at >> >> org.apache.solr.cloud.OverseerCollectionProcessor.run(OverseerCollectionProcessor.java:171) >> at java.lang.Thread.run(Thread.java:619) >> Caused by: org.apache.solr.client.solrj.SolrServerException: Timeout >> occured >> while waiting response from server at: http://<machine:port>/solr >> at >> >> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:499) >> at >> >> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:197) >> at >> >> org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:156) >> at >> >> org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:118) >> at >> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) >> at java.util.concurrent.FutureTask.run(FutureTask.java:138) >> at >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) >> at >> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) >> at java.util.concurrent.FutureTask.run(FutureTask.java:138) >> at >> >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >> at >> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >> ... 1 more >> Caused by: java.net.SocketTimeoutException: Read timed out >> at java.net.SocketInputStream.socketRead0(Native Method) >> at java.net.SocketInputStream.read(SocketInputStream.java:129) >> at >> >> org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166) >> at >> >> org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90) >> at >> >> org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:281) >> at >> >> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:92) >> at >> >> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:61) >> at >> >> org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:254) >> at >> >> org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:289) >> at >> >> org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:252) >> at >> >> org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:191) >> at >> >> org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:300) >> at >> >> org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:127) >> at >> >> org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:715) >> at >> >> org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:520) >> at >> >> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906) >> at >> >> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805) >> at >> >> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784) >> at >> >> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:393) >> ... 11 more >> >> >> >> INFO - 2014-01-03 14:17:35.347; org.apache.solr.update.SolrIndexSplitter; >> SolrIndexSplitter: partitions=2 segments=26 >> INFO - 2014-01-03 14:17:38.131; org.apache.solr.update.SolrIndexSplitter; >> SolrIndexSplitter: partition #0 range=80000000-bfffffff >> ERROR - 2014-01-03 14:17:45.342; >> org.apache.solr.cloud.OverseerCollectionProcessor; Error from shard: >> <mahine:port>/solr >> org.apache.solr.client.solrj.SolrServerException: Timeout occured while >> waiting response from server at: http://<machine:port>/solr >> at >> >> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:499) >> at >> >> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:197) >> at >> >> org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:156) >> at >> >> org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:118) >> at >> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) >> at java.util.concurrent.FutureTask.run(FutureTask.java:138) >> at >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) >> at >> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) >> at java.util.concurrent.FutureTask.run(FutureTask.java:138) >> at >> >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >> at >> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >> at java.lang.Thread.run(Thread.java:619) >> Caused by: java.net.SocketTimeoutException: Read timed out >> at java.net.SocketInputStream.socketRead0(Native Method) >> at java.net.SocketInputStream.read(SocketInputStream.java:129) >> at >> >> org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166) >> at >> >> org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90) >> at >> >> org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:281) >> at >> >> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:92) >> at >> >> org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:61) >> at >> >> org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:254) >> at >> >> org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:289) >> at >> >> org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:252) >> at >> >> org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:191) >> at >> >> org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:300) >> at >> >> org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:127) >> at >> >> org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:715) >> at >> >> org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:520) >> at >> >> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906) >> >> >> Sorry for long stack trace. >> >> Though it throws these exceptions, it still creates the cores and splits >> the >> index based on hashrange. The UI says both the shard1 as well as shard1_0 >> and shard1_1 as active. So even though I have 2 split shards orginated >> from >> shard1, I am not able to delete shard1 as it is still present as active. >> >> I initially explored about >> https://issues.apache.org/jira/browse/SOLR-5314 >> and https://issues.apache.org/jira/browse/SOLR-5214. But in my case >> since it >> looks more like timeout is triggering the issue, I upgraded from 4.5 to >> 4.6 >> as it was mentioned in SOLR-5314 that it is fixed in 4.6. But I am still >> getting this issue. Any help regarding this would be greatly appreciated >> >> >> >> >> -- >> View this message in context: >> http://lucene.472066.n3.nabble.com/SPLITSHARD-failed-to-invoke-SPLIT-core-admin-command-tp4109431.html >> Sent from the Solr - User mailing list archive at Nabble.com. >> > > > > -- > > Anshum Gupta > http://www.anshumgupta.net > -- Anshum Gupta http://www.anshumgupta.net
