The first place to look for something like his is garbage collection. Are you hitting any really long stop-the-world GC pauses?
Best, Erick On Sun, Feb 19, 2017 at 2:21 PM, Sadheera Vithanage <sadhee...@gmail.com> wrote: > Hi Experts, > > I have a solr cloud node (Just 1 node for now with a zookeeper running on > the same machine) running on ubuntu and It has been running without issues > for a while. > > This morning I noticed below error in the error log. > > > *2017-02-19 20:27:54.724 ERROR (qtp97730845-4968) [ ] > o.a.s.s.HttpSolrCall null:java.io.IOException: > java.util.concurrent.TimeoutException: Idle timeout expired: 50001/50000 ms* > * at > org.eclipse.jetty.util.SharedBlockingCallback$Blocker.block(SharedBlockingCallback.java:226)* > * at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:164)* > * at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:530)* > * at > org.apache.commons.io.output.ProxyOutputStream.write(ProxyOutputStream.java:55)* > * at > org.apache.solr.response.QueryResponseWriterUtil$1.write(QueryResponseWriterUtil.java:54)* > * at java.io.OutputStream.write(OutputStream.java:116)* > * at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)* > * at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282)* > * at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)* > * at java.io.OutputStreamWriter.write(OutputStreamWriter.java:207)* > * at org.apache.solr.util.FastWriter.flush(FastWriter.java:140)* > * at org.apache.solr.util.FastWriter.write(FastWriter.java:54)* > * at > org.apache.solr.response.JSONWriter.writeStr(JSONResponseWriter.java:454)* > * at > org.apache.solr.response.TextResponseWriter.writeVal(TextResponseWriter.java:128)* > * at > org.apache.solr.response.JSONWriter.writeSolrDocument(JSONResponseWriter.java:346)* > * at > org.apache.solr.response.TextResponseWriter.writeSolrDocumentList(TextResponseWriter.java:239)* > * at > org.apache.solr.response.TextResponseWriter.writeVal(TextResponseWriter.java:163)* > * at > org.apache.solr.response.JSONWriter.writeNamedListAsMapWithDups(JSONResponseWriter.java:184)* > * at > org.apache.solr.response.JSONWriter.writeNamedList(JSONResponseWriter.java:300)* > * at > org.apache.solr.response.JSONWriter.writeResponse(JSONResponseWriter.java:96)* > * at > org.apache.solr.response.JSONResponseWriter.write(JSONResponseWriter.java:55)* > * at > org.apache.solr.response.QueryResponseWriterUtil.writeQueryResponse(QueryResponseWriterUtil.java:65)* > * at > org.apache.solr.servlet.HttpSolrCall.writeResponse(HttpSolrCall.java:728)* > * at > org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:667)* > * at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:441)* > * at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:303)* > * at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:254)* > * at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)* > * at > org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)* > * at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)* > * at > org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)* > * at > org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)* > * at > org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)* > * at > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)* > * at > org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)* > * at > org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)* > * at > org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)* > * at > org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)* > * at > org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)* > * at > org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)* > * at org.eclipse.jetty.server.Server.handle(Server.java:518)* > * at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)* > * at > org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)* > * at > org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)* > * at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)* > * at > org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)* > * at > org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)* > * at > org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)* > * at > org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)* > * at > org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)* > * at java.lang.Thread.run(Thread.java:745)* > *Caused by: java.util.concurrent.TimeoutException: Idle timeout expired: > 50001/50000 ms* > * at > org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:166)* > * at org.eclipse.jetty.io.IdleTimeout$1.run(IdleTimeout.java:50)* > * at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)* > * at java.util.concurrent.FutureTask.run(FutureTask.java:266)* > * at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)* > * at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)* > * at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)* > * at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)* > * ... 1 more* > > It has been there for 3 minutes and nothing after that. > > Any Clue? > > -- > Regards > > Sadheera Vithanage
