Re: Replicaiton Fails with Unreachable error when master host is responding.

Tue, 03 May 2011 22:14:46 -0700

So it turns out that it's the host names. According the DNS RFC underscores are not valid in host names. Most DNS servers now support them, but it's not in the rfc strictly speaking. So there must be something in the underlying java classes that bork when using underscores in host names, though I didn't see anything in the stack trace that indicated an invalid host name exception. That was most the issue though. Once I changed the host name to the master's IP address replication worked great. So I'm working with our IT to remove underscores from our host names.
Just thought I would post my answer here in case anyone else had that issue. 

Thanks.

Jed.

On 04/28/2011 02:03 PM, Mike Sokolov wrote:

No clue. Try wireshark to gather more data?

On 04/28/2011 02:53 PM, Jed Glazner wrote:

Anybody?

On 04/27/2011 01:51 PM, Jed Glazner wrote:

Hello All,

I'm having a very strange problem that I just can't figure out. The
slave is not able to replicate from the master, even though the master
is reachable from the slave machine.  I can telnet to the port it's
running on, I can use text based browsers to navigate the master from
the slave. I just don't understand why it won't replicate.  The admin
screen gives me an Unreachable in the status, and in the log there is an
exception thrown.  Details below:

BACKGROUND:

OS: Arch Linux
Solr Version: svn revision 1096983 from
https://svn.apache.org/repos/asf/lucene/dev/branches/branch_3x/
No custom plugins, just whatever came with the version above.
Java Setup:

java version "1.6.0_22"
OpenJDK Runtime Environment (IcedTea6 1.10) (ArchLinux-6.b22_1.10-1-x86_64)
OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode)

We have 3 cores running, all 3 cores are not able to replicate.

The admin on the slave shows  the Master as
http://solr-master-01_dev.la.bo:8983/solr/music/replication  - *Unreachable*
Replicaiton def on the slave

    529<requestHandler name="/replication" class="solr.ReplicationHandler">
    530<lst name="${slave:slave}">
    531<str
name="masterUrl">http://solr-master-01_dev.la.bo:8983/solr/music/replication</str>
    532<str name="pollInterval">00:15:00</str>
    533</lst>
    534</requestHandler>

Replication def on the master:

    529<requestHandler name="/replication" class="solr.ReplicationHandler">
    530<lst name="${master:master}">
    531<str name="replicateAfter">commit</str>
    532<str name="replicateAfter">startup</str>
    533<str name="confFiles">schema.xml,stopwords.txt</str>
    534</lst>
    535</requestHandler>

Below is the log start to finish for replication attempts, note that it
says connection refused, however, I can telnet to 8983 from the slave to
the master, so I know it's up and reachable from the slave:

telnet solr-master-01_dev.la.bo 8983
Trying 172.12.65.58...
Connected to solr-master-01_dev.la.bo.
Escape character is '^]'.

I double checked the master to make sure that it didn't have replication
turned off, and it's not.  So I should be able to replicate but it
can't.  I just dont' know what else to check.  The log from the slave is
below.

Apr 27, 2011 7:39:45 PM org.apache.solr.request.SolrQueryResponse<init>
WARNING: org.apache.solr.request.SolrQueryResponse is deprecated. Please
use the corresponding class in org.apache.solr.response
Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector
executeWithRetry
INFO: I/O exception (java.net.ConnectException) caught when processing
request: Connection refused
Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector
executeWithRetry
INFO: Retrying request
Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector
executeWithRetry
INFO: I/O exception (java.net.ConnectException) caught when processing
request: Connection refused
Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector
executeWithRetry
INFO: Retrying request
Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector
executeWithRetry
INFO: I/O exception (java.net.ConnectException) caught when processing
request: Connection refused
Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector
executeWithRetry
INFO: Retrying request
Apr 27, 2011 7:39:45 PM org.apache.solr.handler.ReplicationHandler
getReplicationDetails
WARNING: Exception while invoking 'details' method for replication on
master
java.net.ConnectException: Connection refused
       at java.net.PlainSocketImpl.socketConnect(Native Method)
       at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:327)
       at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:193)
       at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:180)
       at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:384)
       at java.net.Socket.connect(Socket.java:546)
       at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
       at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
       at java.lang.reflect.Method.invoke(Method.java:616)
       at
org.apache.commons.httpclient.protocol.ReflectionSocketFactory.createSocket(ReflectionSocketFactory.java:140)
       at
org.apache.commons.httpclient.protocol.DefaultProtocolSocketFactory.createSocket(DefaultProtocolSocketFactory.java:125)
       at
org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707)
       at
org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361)
       at
org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387)
       at
org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
       at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
       at
org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
       at
org.apache.solr.handler.SnapPuller.getNamedListResponse(SnapPuller.java:193)
       at
org.apache.solr.handler.SnapPuller.getCommandResponse(SnapPuller.java:188)
       at
org.apache.solr.handler.ReplicationHandler.getReplicationDetails(ReplicationHandler.java:588)
       at
org.apache.solr.handler.ReplicationHandler.handleRequestBody(ReplicationHandler.java:183)
       at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
       at org.apache.solr.core.SolrCore.execute(SolrCore.java:1360)
       at
org.apache.jsp.admin.replication.index_jsp.executeCommand(org.apache.jsp.admin.replication.index_jsp:50)
       at
org.apache.jsp.admin.replication.index_jsp._jspService(org.apache.jsp.admin.replication.index_jsp:232)
       at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:109)
       at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
       at
org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:389)
       at
org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:486)
       at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:380)
       at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
       at
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
       at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
       at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
       at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
       at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
       at
org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
       at org.mortbay.jetty.servlet.Dispatcher.forward(Dispatcher.java:327)
       at org.mortbay.jetty.servlet.Dispatcher.forward(Dispatcher.java:126)
       at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:275)
       at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
       at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
       at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
       at
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
       at
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
       at
org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
       at
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
       at
org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
       at
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
       at org.mortbay.jetty.Server.handle(Server.java:326)
       at
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
       at
org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945)
       at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
       at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
       at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
       at
org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
       at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Apr 27, 2011 7:39:45 PM org.apache.solr.core.SolrCore execute
INFO: [music] webapp=null path=null params={command=details} status=0
QTime=66
Apr 27, 2011 7:39:45 PM org.apache.solr.request.SolrQueryResponse<init>
WARNING: org.apache.solr.request.SolrQueryResponse is deprecated. Please
use the corresponding class in org.apache.solr.response
Apr 27, 2011 7:39:45 PM org.apache.solr.core.SolrCore execute
INFO: [music] webapp=null path=null params={command=fetchindex} status=0
QTime=1
Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector
executeWithRetry
INFO: I/O exception (java.net.ConnectException) caught when processing
request: Connection refused
Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector
executeWithRetry
INFO: Retrying request
Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector
executeWithRetry
INFO: I/O exception (java.net.ConnectException) caught when processing
request: Connection refused
Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector
executeWithRetry
INFO: Retrying request
Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector
executeWithRetry
INFO: I/O exception (java.net.ConnectException) caught when processing
request: Connection refused
Apr 27, 2011 7:39:45 PM org.apache.commons.httpclient.HttpMethodDirector
executeWithRetry
INFO: Retrying request
Apr 27, 2011 7:39:45 PM org.apache.solr.handler.SnapPuller fetchLatestIndex
SEVERE: Master at:
http://solr-master-01_dev.la.bo:8983/solr/music/replication  is not
available. Index fetch failed. Exception: Connection refused
Apr 27, 2011 7:40:00 PM org.apache.commons.httpclient.HttpMethodDirector
executeWithRetry
INFO: I/O exception (java.net.ConnectException) caught when processing
request: Connection refused
Apr 27, 2011 7:40:00 PM org.apache.commons.httpclient.HttpMethodDirector
executeWithRetry
INFO: Retrying request
Apr 27, 2011 7:40:00 PM org.apache.commons.httpclient.HttpMethodDirector
executeWithRetry
INFO: I/O exception (java.net.ConnectException) caught when processing
request: Connection refused
Apr 27, 2011 7:40:00 PM org.apache.commons.httpclient.HttpMethodDirector
executeWithRetry
INFO: Retrying request
Apr 27, 2011 7:40:00 PM org.apache.commons.httpclient.HttpMethodDirector
executeWithRetry
INFO: I/O exception (java.net.ConnectException) caught when processing
request: Connection refused
Apr 27, 2011 7:40:00 PM org.apache.commons.httpclient.HttpMethodDirector
executeWithRetry
INFO: Retrying request
Apr 27, 2011 7:40:00 PM org.apache.solr.handler.SnapPuller fetchLatestIndex
SEVERE: Master at:http://solr-master-01_dev:8983/solr/rovi/replication
is not available. Index fetch failed. Exception: Connection refused




--
Jed Glazner (801-360-0181)

This email and its attachments (if any) are for the sole use of the
intended recipient, and may contain private, confidential, and
privileged material. Any review, copying, or distribution of this
email, its attachments or the information contained herein is strictly
prohibited. If you are not the intended recipient, please contact the
sender immediately and permanently delete the original and any copies
of this email and any attachments.

























					Reply via email to