Re: Replicaiton Fails with Unreachable error when master host is responding.

[Jed Glazner]

Anybody? On 04/27/2011 01:51 PM, Jed Glazner wrote: Hello All, I'm having a very strange problem that I just can't figure out. The slave is not able to replicate from the master, even though the master is reachable from the slave machine. I can telnet to the port it's running on, I can use text based browsers to navigate the master from the slave. I just don't understand why it won't replicate. The admin screen gives me an Unreachable in the status, and in the log there is an exception thrown. Details below: BACKGROUND: OS: Arch Linux Solr Version: svn revision 1096983 from https://svn.apache.org/repos/asf/lucene/dev/branches/branch_3x/ No custom plugins, just whatever came with the version above. Java Setup: java version "1.6.0_22" OpenJDK Runtime Environment (IcedTea6 1.10) (ArchLinux-6.b22_1.10-1-x86_64) OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode) We have 3 cores running, all 3 cores are not able to replicate. The admin on the slave shows the Master as http://solr-master-01_dev.la.bo:8983/solr/music/replication - *Unreachable* Replicaiton def on the slave 529 <requestHandler name="/replication" class="solr.ReplicationHandler" > 530 <lst name="${slave:slave}"> 531 <str name="masterUrl">http://solr-master-01_dev.la.bo:8983/solr/music/replication</str> 532 <str name="pollInterval">00:15:00</str> 533 </lst> 534 </requestHandler> Replication def on the master: 529 <requestHandler name="/replication" class="solr.ReplicationHandler" > 530 <lst name="${master:master}"> 531 <str name="replicateAfter">commit</str> 532 <str name="replicateAfter">startup</str> 533 <str name="confFiles">schema.xml,stopwords.txt</str> 534 </lst> 535 </requestHandler> Below is the log start to finish for replication attempts, note that it says connection refused, however, I can telnet to 8983 from the slave to the master, so I know it's up and reachable from the slave: telnet solr-master-01_dev.la.bo 8983 Trying 172.12.65.58... Connected to solr-master-01_dev.la.bo. Escape character is '^]'. I double checked the master to make sure that it didn't have replication turned off, and it's not. So I should be able to replicate but it can't. I just dont' know what else to check. The log from the slave is below. Apr 27, 2011 7:39:45 PM org.apache.solr.request.SolrQueryResponse <init> WARNING: org.apache.solr.request.SolrQueryResponse is deprecated. Please use the corresponding class in org.apache.solr.response Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry INFO: I/O exception (java.net.ConnectException) caught when processing request: Connection refused Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry INFO: Retrying request Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry INFO: I/O exception (java.net.ConnectException) caught when processing request: Connection refused Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry INFO: Retrying request Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry INFO: I/O exception (java.net.ConnectException) caught when processing request: Connection refused Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry INFO: Retrying request Apr 27, 2011 7:39:45 PM org.apache.solr.handler.ReplicationHandler getReplicationDetails WARNING: Exception while invoking 'details' method for replication on master java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:327) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:193) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:180) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:384) at java.net.Socket.connect(Socket.java:546) at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at org.apache.commons.httpclient.protocol.ReflectionSocketFactory.createSocket(ReflectionSocketFactory.java:140) at org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:125) at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) at org.apache.solr.handler.SnapPuller.getNamedListResponse(SnapPuller.java:193) at org.apache.solr.handler.SnapPuller.getCommandResponse(SnapPuller.java:188) at org.apache.solr.handler.ReplicationHandler.getReplicationDetails(ReplicationHandler.java:588) at org.apache.solr.handler.ReplicationHandler.handleRequestBody(ReplicationHandler.java:183) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1360) at org.apache.jsp.admin.replication.index_jsp.executeCommand(org.apache.jsp.admin.replication.index_jsp:50) at org.apache.jsp.admin.replication.index_jsp._jspService(org.apache.jsp.admin.replication.index_jsp:232) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:109) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:389) at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:486) at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:380) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) at org.mortbay.jetty.servlet.Dispatcher.forward(Dispatcher.java:327) at org.mortbay.jetty.servlet.Dispatcher.forward(Dispatcher.java:126) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:275) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Apr 27, 2011 7:39:45 PM org.apache.solr.core.SolrCore execute INFO: [music] webapp=null path=null params={command=details} status=0 QTime=66 Apr 27, 2011 7:39:45 PM org.apache.solr.request.SolrQueryResponse <init> WARNING: org.apache.solr.request.SolrQueryResponse is deprecated. Please use the corresponding class in org.apache.solr.response Apr 27, 2011 7:39:45 PM org.apache.solr.core.SolrCore execute INFO: [music] webapp=null path=null params={command=fetchindex} status=0 QTime=1 Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry INFO: I/O exception (java.net.ConnectException) caught when processing request: Connection refused Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry INFO: Retrying request Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry INFO: I/O exception (java.net.ConnectException) caught when processing request: Connection refused Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry INFO: Retrying request Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry INFO: I/O exception (java.net.ConnectException) caught when processing request: Connection refused Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry INFO: Retrying request Apr 27, 2011 7:39:45 PM org.apache.solr.handler.SnapPuller fetchLatestIndex SEVERE: Master at: http://solr-master-01_dev.la.bo:8983/solr/music/replication is not available. Index fetch failed. Exception: Connection refused Apr 27, 2011 7:40:00 PM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry INFO: I/O exception (java.net.ConnectException) caught when processing request: Connection refused Apr 27, 2011 7:40:00 PM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry INFO: Retrying request Apr 27, 2011 7:40:00 PM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry INFO: I/O exception (java.net.ConnectException) caught when processing request: Connection refused Apr 27, 2011 7:40:00 PM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry INFO: Retrying request Apr 27, 2011 7:40:00 PM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry INFO: I/O exception (java.net.ConnectException) caught when processing request: Connection refused Apr 27, 2011 7:40:00 PM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry INFO: Retrying request Apr 27, 2011 7:40:00 PM org.apache.solr.handler.SnapPuller fetchLatestIndex SEVERE: Master at: http://solr-master-01_dev:8983/solr/rovi/replication is not available. Index fetch failed. Exception: Connection refused -- This email and its attachments (if any) are for the sole use of the intended recipient, and may contain private, confidential, and privileged material. Any review, copying, or distribution of this email, its attachments or the information contained herein is strictly prohibited. If you are not the intended recipient, please contact the sender immediately and permanently delete the original and any copies of this email and any attachments.