Ah thanks for explaining that! Thanks Jay Potharaju
On Mon, May 7, 2018 at 9:45 AM, Emir Arnautović < emir.arnauto...@sematext.com> wrote: > Node A receives batch of documents to index. It forwards documents to > shards that are on the node B. Node B is having issues with GC so it takes > a while to respond. Node A sees it as read timeout and reports it in logs. > So the issue is on node B not node A. > > Emir > -- > Monitoring - Log Management - Alerting - Anomaly Detection > Solr & Elasticsearch Consulting Support Training - http://sematext.com/ > > > > > On 7 May 2018, at 18:39, Jay Potharaju <jspothar...@gmail.com> wrote: > > > > Yes, the nodes are well balanced. I am just using these boxes for > indexing > > the data and is not serving any traffic at this time. The error > indicates > > it is having issues errors on the shards that are hosted on the box and > not > > on the other box. > > I will check GC logs to see if there were any issues. > > thanks > > > > Thanks > > Jay Potharaju > > > > > > On Mon, May 7, 2018 at 9:34 AM, Emir Arnautović < > > emir.arnauto...@sematext.com> wrote: > > > >> Hi Jay, > >> My first guess would be that there was some major GC on other box so it > >> did not respond on time. Are your nodes well balanced - do they serve > equal > >> amount of data? > >> > >> Thanks, > >> Emir > >> -- > >> Monitoring - Log Management - Alerting - Anomaly Detection > >> Solr & Elasticsearch Consulting Support Training - http://sematext.com/ > >> > >> > >> > >>> On 7 May 2018, at 18:11, Jay Potharaju <jspothar...@gmail.com> wrote: > >>> > >>> Hi, > >>> I am seeing the following lines in the error log. My setup has 2 nodes > in > >>> the solrcloud cluster, each node has 3 shards with no replication. From > >> the > >>> error log it seems like all the shards on this box are throwing async > >>> exception errors. Other node in the cluster does not have any errors in > >> the > >>> logs. Any suggestions on how to tackle this error? > >>> > >>> Solr setup > >>> Solr:6.6.3 > >>> 2Nodes: 3 shards each > >>> > >>> > >>> ERROR org.apache.solr.servlet.HttpSolrCall [test_shard3_replica1] ? > >>> null:org.apache.solr.update.processor.DistributedUpdateProcessor$ > >> DistributedUpdatesAsyncException: > >>> Async exception during distributed update: Read timed out > >>> at > >>> org.apache.solr.update.processor.DistributedUpdateProcessor.doFinish( > >> DistributedUpdateProcessor.java:972) > >>> at > >>> org.apache.solr.update.processor.DistributedUpdateProcessor.finish( > >> DistributedUpdateProcessor.java:1911) > >>> at > >>> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody( > >> ContentStreamHandlerBase.java:78) > >>> at > >>> org.apache.solr.handler.RequestHandlerBase.handleRequest( > >> RequestHandlerBase.java:173) > >>> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2477) > >>> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:723) > >>> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:529) > >>> at > >>> org.apache.solr.servlet.SolrDispatchFilter.doFilter( > >> SolrDispatchFilter.java:361) > >>> at > >>> org.apache.solr.servlet.SolrDispatchFilter.doFilter( > >> SolrDispatchFilter.java:305) > >>> at > >>> org.eclipse.jetty.servlet.ServletHandler$CachedChain. > >> doFilter(ServletHandler.java:1691) > >>> at > >>> org.eclipse.jetty.servlet.ServletHandler.doHandle( > >> ServletHandler.java:582) > >>> at > >>> org.eclipse.jetty.server.handler.ScopedHandler.handle( > >> ScopedHandler.java:143) > >>> at > >>> org.eclipse.jetty.security.SecurityHandler.handle( > >> SecurityHandler.java:548) > >>> at > >>> org.eclipse.jetty.server.session.SessionHandler. > >> doHandle(SessionHandler.java:226) > >>> at > >>> org.eclipse.jetty.server.handler.ContextHandler. > >> doHandle(ContextHandler.java:1180) > >>> at org.eclipse.jetty.servlet.ServletHandler.doScope( > >> ServletHandler.java:512) > >>> at > >>> org.eclipse.jetty.server.session.SessionHandler. > >> doScope(SessionHandler.java:185) > >>> at > >>> org.eclipse.jetty.server.handler.ContextHandler. > >> doScope(ContextHandler.java:1112) > >>> at > >>> org.eclipse.jetty.server.handler.ScopedHandler.handle( > >> ScopedHandler.java:141) > >>> at > >>> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle( > >> ContextHandlerCollection.java:213) > >>> at > >>> org.eclipse.jetty.server.handler.HandlerCollection. > >> handle(HandlerCollection.java:119) > >>> at > >>> org.eclipse.jetty.server.handler.HandlerWrapper.handle( > >> HandlerWrapper.java:134) > >>> at > >>> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle( > >> RewriteHandler.java:335) > >>> at > >>> org.eclipse.jetty.server.handler.HandlerWrapper.handle( > >> HandlerWrapper.java:134) > >>> at org.eclipse.jetty.server.Server.handle(Server.java:534) > >>> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320) > >>> at > >>> org.eclipse.jetty.server.HttpConnection.onFillable( > >> HttpConnection.java:251) > >>> at > >>> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded( > >> AbstractConnection.java:273) > >>> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) > >>> at > >>> org.eclipse.jetty.io.SelectChannelEndPoint$2.run( > >> SelectChannelEndPoint.java:93) > >>> at > >>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob( > >> QueuedThreadPool.java:671) > >>> at > >>> org.eclipse.jetty.util.thread.QueuedThreadPool$2.run( > >> QueuedThreadPool.java:589) > >>> at java.lang.Thread.run(Unknown Source) > >>> > >>> > >>> Thanks > >>> Jay > >> > >> > >
