Looks like something reset the connection between the Solr server you are talking to and the one hosting the other shard ( http://192.168.120.202:8080/solr)
I would check the logs of the _other_ server and see if something is there. Otherwise, I would look into something like a firewall in between two Solr servers and perhaps a response taking so long that the firewall kills the connection in between. Regards, Alex. On 21/11/2014 6:02 am, "馬 飛虎" <f...@chip1stop.com> wrote: > > Hi > > We used Solr4.6 for search and there was a exception occured randomly. > The exception message in application(use SolrJ) was : > > org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: > org.apache.solr.client.solrj.SolrServerException: IOException occured > when talking to server at: http://192.168.xxx.xxx:8080/solr > at > org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:495) > at > org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:199) > at > org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:90) > at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:310) > at search.Searcher.search(Searcher.java:379) > at search.Searcher.doSearchFull(Searcher.java:105) > at > com.c1s.common.search.YuukyuuSearcher.doSearchFull(YuukyuuSearcher.java:128) > at > com.c1s.common.dispPartInfoList.DispPartInfoListUtils.getNewPatSyubetu(DispPartInfoListUtils.java:802) > at > com.c1s.common.dispPartInfoList.DispPartInfoListUtils.getDetailPartInfo(DispPartInfoListUtils.java:1459) > at > userweb.action.dispDetail.DispDetailAction.doExecute(DispDetailAction.java:411) > at com.c1s.action.BaseAction.execute(BaseAction.java:193) > at > org.apache.struts.action.RequestProcessor.processActionPerform(RequestProcessor.java:431) > at > org.seasar.struts.processor.S2RequestProcessor.processActionPerform(S2RequestProcessor.java:145) > at org.seasar.struts.processor.AcceptorImpl.process(AcceptorImpl.java:133) > at > org.seasar.struts.processor.S2RequestProcessor.process(S2RequestProcessor.java:74) > at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1196) > at org.apache.struts.action.ActionServlet.doGet(ActionServlet.java:414) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:718) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:831) > at > org.apache.catalina.core.ApplicationFilterChain.servletService(ApplicationFilterChain.java:411) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:317) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:198) > at > com.c1s.action.RequestParameterFilter.doFilter(RequestParameterFilter.java:84) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:230) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:198) > at > org.seasar.extension.filter.EncodingFilter.doFilter(EncodingFilter.java:62) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:230) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:198) > at org.seasar.struts.filter.S2StrutsFilter.doFilter(S2StrutsFilter.java:50) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:230) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:198) > at > org.seasar.framework.container.filter.S2ContainerFilter.doFilter(S2ContainerFilter.java:60) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:230) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:198) > at com.c1s.action.SessionFilter.doFilter(SessionFilter.java:99) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:230) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:198) > at com.c1s.action.UrlFilter.doFilter(UrlFilter.java:354) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:230) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:198) > at > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:288) > at > org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:271) > at > org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:202) > at > org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632) > at > org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577) > at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:94) > at > com.sun.enterprise.web.PESessionLockingStandardPipeline.invoke(PESessionLockingStandardPipeline.java:96) > at > com.sun.enterprise.ee.web.sessmgmt.SessionLockingStandardPipeline.invoke(SessionLockingStandardPipeline.java:120) > at > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:206) > at > org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632) > at > org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577) > at > org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:571) > at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1080) > at > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:150) > at > org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:632) > at > org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:577) > at > org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:571) > at org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1080) > at org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:272) > at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:200) > at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:283) > at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:773) > at > org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:703) > at > org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:895) > at > org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:666) > at java.lang.Thread.run(Thread.java:619) > > > > And the message in solr server is : > org.apache.solr.common.SolrException: > org.apache.solr.client.solrj.SolrServerException: IOException occured > when talking to server at: http://192.168.120.202:8080/solr > at > org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:302) > at > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135) > at org.apache.solr.core.SolrCore.execute(SolrCore.java:1859) > at > org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:721) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:417) > at > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:201) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) > at > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220) > at > org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122) > at > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170) > at > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98) > at > org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950) > at > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) > at > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408) > at > org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1040) > at > org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607) > at > org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:313) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.apache.solr.client.solrj.SolrServerException: IOException > occured when talking to server at: http://192.168.120.202:8080/solr > at > org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:507) > at > org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:199) > at > org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:156) > at > org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:118) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > ... 3 more > Caused by: java.net.SocketException: Connection reset > at java.net.SocketInputStream.read(SocketInputStream.java:196) > at java.net.SocketInputStream.read(SocketInputStream.java:122) > at > org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166) > at > org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90) > at > org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:281) > at > org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:92) > at > org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:61) > at > org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:254) > at > org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:289) > at > org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:252) > at > org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:191) > at > org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:300) > at > org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:127) > at > org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:715) > at > org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:520) > at > org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906) > at > org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805) > at > org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784) > at > org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:395) > ... 9 more > > We thought this occured because of the solrServer has too many > connections at 1 second, > But we used Jmeter to simulate a very heavy load on the solrSever,and > the similar Excepion was not hanppened. > > We used SolrJ in the application,and the SolrServer class was > HttpSolrServer class with the default HttpClient. > The solr was run on Tomcat,and the Tomcat config was the default config. > And now we still receive this Exception a few times a day. > > Any ideas on what the root cause of this issue may be and how we can > remediate it. > > Thanks.