Thanks Eric, It looked like the garbage collection was blocking the other processes.
I updated the SOLR_JAVA_MEM="-Xms1g -Xmx4g" as it was the default before and looked like the garbage collection was triggered too frequent. Lets see how it goes now. Thanks again for the support. On Mon, Feb 20, 2017 at 11:50 AM, Erick Erickson <erickerick...@gmail.com> wrote: > The first place to look for something like his is garbage collection. > Are you hitting any really long stop-the-world GC pauses? > > Best, > Erick > > On Sun, Feb 19, 2017 at 2:21 PM, Sadheera Vithanage <sadhee...@gmail.com> > wrote: > > Hi Experts, > > > > I have a solr cloud node (Just 1 node for now with a zookeeper running on > > the same machine) running on ubuntu and It has been running without > issues > > for a while. > > > > This morning I noticed below error in the error log. > > > > > > *2017-02-19 20:27:54.724 ERROR (qtp97730845-4968) [ ] > > o.a.s.s.HttpSolrCall null:java.io.IOException: > > java.util.concurrent.TimeoutException: Idle timeout expired: > 50001/50000 ms* > > * at > > org.eclipse.jetty.util.SharedBlockingCallback$Blocker.block( > SharedBlockingCallback.java:226)* > > * at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:164)* > > * at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:530)* > > * at > > org.apache.commons.io.output.ProxyOutputStream.write( > ProxyOutputStream.java:55)* > > * at > > org.apache.solr.response.QueryResponseWriterUtil$1. > write(QueryResponseWriterUtil.java:54)* > > * at java.io.OutputStream.write(OutputStream.java:116)* > > * at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)* > > * at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282)* > > * at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)* > > * at java.io.OutputStreamWriter.write(OutputStreamWriter.java:207)* > > * at org.apache.solr.util.FastWriter.flush(FastWriter.java:140)* > > * at org.apache.solr.util.FastWriter.write(FastWriter.java:54)* > > * at > > org.apache.solr.response.JSONWriter.writeStr( > JSONResponseWriter.java:454)* > > * at > > org.apache.solr.response.TextResponseWriter.writeVal( > TextResponseWriter.java:128)* > > * at > > org.apache.solr.response.JSONWriter.writeSolrDocument( > JSONResponseWriter.java:346)* > > * at > > org.apache.solr.response.TextResponseWriter.writeSolrDocumentList( > TextResponseWriter.java:239)* > > * at > > org.apache.solr.response.TextResponseWriter.writeVal( > TextResponseWriter.java:163)* > > * at > > org.apache.solr.response.JSONWriter.writeNamedListAsMapWithDups( > JSONResponseWriter.java:184)* > > * at > > org.apache.solr.response.JSONWriter.writeNamedList( > JSONResponseWriter.java:300)* > > * at > > org.apache.solr.response.JSONWriter.writeResponse( > JSONResponseWriter.java:96)* > > * at > > org.apache.solr.response.JSONResponseWriter.write( > JSONResponseWriter.java:55)* > > * at > > org.apache.solr.response.QueryResponseWriterUtil.writeQueryResponse( > QueryResponseWriterUtil.java:65)* > > * at > > org.apache.solr.servlet.HttpSolrCall.writeResponse( > HttpSolrCall.java:728)* > > * at > > org.apache.solr.servlet.HttpSolrCall.handleAdminRequest( > HttpSolrCall.java:667)* > > * at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:441)* > > * at > > org.apache.solr.servlet.SolrDispatchFilter.doFilter( > SolrDispatchFilter.java:303)* > > * at > > org.apache.solr.servlet.SolrDispatchFilter.doFilter( > SolrDispatchFilter.java:254)* > > * at > > org.eclipse.jetty.servlet.ServletHandler$CachedChain. > doFilter(ServletHandler.java:1668)* > > * at > > org.eclipse.jetty.servlet.ServletHandler.doHandle( > ServletHandler.java:581)* > > * at > > org.eclipse.jetty.server.handler.ScopedHandler.handle( > ScopedHandler.java:143)* > > * at > > org.eclipse.jetty.security.SecurityHandler.handle( > SecurityHandler.java:548)* > > * at > > org.eclipse.jetty.server.session.SessionHandler. > doHandle(SessionHandler.java:226)* > > * at > > org.eclipse.jetty.server.handler.ContextHandler. > doHandle(ContextHandler.java:1160)* > > * at > > org.eclipse.jetty.servlet.ServletHandler.doScope( > ServletHandler.java:511)* > > * at > > org.eclipse.jetty.server.session.SessionHandler. > doScope(SessionHandler.java:185)* > > * at > > org.eclipse.jetty.server.handler.ContextHandler. > doScope(ContextHandler.java:1092)* > > * at > > org.eclipse.jetty.server.handler.ScopedHandler.handle( > ScopedHandler.java:141)* > > * at > > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle( > ContextHandlerCollection.java:213)* > > * at > > org.eclipse.jetty.server.handler.HandlerCollection. > handle(HandlerCollection.java:119)* > > * at > > org.eclipse.jetty.server.handler.HandlerWrapper.handle( > HandlerWrapper.java:134)* > > * at org.eclipse.jetty.server.Server.handle(Server.java:518)* > > * at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)* > > * at > > org.eclipse.jetty.server.HttpConnection.onFillable( > HttpConnection.java:244)* > > * at > > org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded( > AbstractConnection.java:273)* > > * at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)* > > * at > > org.eclipse.jetty.io.SelectChannelEndPoint$2.run( > SelectChannelEndPoint.java:93)* > > * at > > org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume. > produceAndRun(ExecuteProduceConsume.java:246)* > > * at > > org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run( > ExecuteProduceConsume.java:156)* > > * at > > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob( > QueuedThreadPool.java:654)* > > * at > > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run( > QueuedThreadPool.java:572)* > > * at java.lang.Thread.run(Thread.java:745)* > > *Caused by: java.util.concurrent.TimeoutException: Idle timeout expired: > > 50001/50000 ms* > > * at > > org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:166)* > > * at org.eclipse.jetty.io.IdleTimeout$1.run(IdleTimeout.java:50)* > > * at > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)* > > * at java.util.concurrent.FutureTask.run(FutureTask.java:266)* > > * at > > java.util.concurrent.ScheduledThreadPoolExecutor$ > ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)* > > * at > > java.util.concurrent.ScheduledThreadPoolExecutor$ > ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)* > > * at > > java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1142)* > > * at > > java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:617)* > > * ... 1 more* > > > > It has been there for 3 minutes and nothing after that. > > > > Any Clue? > > > > -- > > Regards > > > > Sadheera Vithanage > -- Regards Sadheera Vithanage
