Idle timeout expired: 50000/50000 ms

Thu, 28 Apr 2016 14:15:43 -0700 

Hi,
I operate several collections (about 7-8) all using the same 5-node ZooKeeper cluster. They've been in production for 3 months, with only 2 previous issues where a Solr node went down. 


Tonight, during several updates to the various collections, a handful 
failed due to the below error.



Could this be related to ZooKeeper in any way?  If so, what could I 
check to ensure everything is running smoothly?


The collections are a mix of 1 and 2 shards, all with 1 replica.

Updates are performed in batches of 1000 in JSON files.

Are there any other things I could/should be checking?


$VAR1 = {
         'error' => {
                      'code' => 500,

                      'msg' => 'java.util.concurrent.TimeoutException: 
Idle timeout expired: 50000/50000 ms',
                      'trace' => 'java.io.IOException: 
java.util.concurrent.TimeoutException: Idle timeout expired: 50000/50000 ms
    at 
org.eclipse.jetty.util.SharedBlockingCallback$Blocker.block(SharedBlockingCallback.java:234)
    at 
org.eclipse.jetty.server.HttpInputOverHTTP.blockForContent(HttpInputOverHTTP.java:66)
    at 
org.eclipse.jetty.server.HttpInput$1.waitForContent(HttpInput.java:476)

    at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:121)
    at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
    at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
    at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
    at java.io.InputStreamReader.read(InputStreamReader.java:184)
    at org.noggit.JSONParser.fill(JSONParser.java:196)
    at org.noggit.JSONParser.getMore(JSONParser.java:203)
    at org.noggit.JSONParser.readStringChars2(JSONParser.java:646)
    at org.noggit.JSONParser.readStringChars(JSONParser.java:626)
    at org.noggit.JSONParser.getStringChars(JSONParser.java:1029)
    at org.noggit.JSONParser.getString(JSONParser.java:1017)

    at 
org.apache.solr.handler.loader.JsonLoader$SingleThreadedJsonLoader.parseDoc(JsonLoader.java:501)
    at 
org.apache.solr.handler.loader.JsonLoader$SingleThreadedJsonLoader.handleAdds(JsonLoader.java:470)
    at 
org.apache.solr.handler.loader.JsonLoader$SingleThreadedJsonLoader.processUpdate(JsonLoader.java:135)
    at 
org.apache.solr.handler.loader.JsonLoader$SingleThreadedJsonLoader.load(JsonLoader.java:114)

    at org.apache.solr.handler.loader.JsonLoader.load(JsonLoader.java:77)

    at 
org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:95)
    at 
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:70)
    at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:156)

    at org.apache.solr.core.SolrCore.execute(SolrCore.java:2073)
    at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:658)
    at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:457)

    at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:222)
    at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:181)
    at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
    at 
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
    at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
    at 
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
    at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
    at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
    at 
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
    at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
    at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
    at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
    at 
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)
    at 
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
    at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)

    at org.eclipse.jetty.server.Server.handle(Server.java:499)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)

    at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
    at 
org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)
    at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
    at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)

    at java.lang.Thread.run(Thread.java:745)

Caused by: java.util.concurrent.TimeoutException: Idle timeout expired: 
50000/50000 ms
    at 
org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:161)

    at org.eclipse.jetty.io.IdleTimeout$1.run(IdleTimeout.java:50)

    at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

    at java.util.concurrent.FutureTask.run(FutureTask.java:266)

    at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
    at 
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

    ... 1 more
'
                    },
         'responseHeader' => {
                               'QTime' => 77068,
                               'status' => 500
                             }
       };






















					Reply via email to