The replication does not work for me
I have a big master solr and I want to start replicating it. I can see that the slave is downloading data from the master... I see a directory index.20100302093000 gets created in data/ next to index... I can see its size growing but then the directory gets deleted Here is the complete trace (I added a couple of LOG messages and compile solr) [2010-03-02 21:24:00] [pool-3-thread-1] DEBUG(MultiThreadedHttpConnectionManager.java:961) - Notifying no-one, there are no waiting threads [2010-03-02 21:24:00] [pool-3-thread-1] INFO (SnapPuller.java:278) - Number of files in latest index in master: 163 [2010-03-02 21:24:00] [pool-3-thread-1] DEBUG(SnapPuller.java:536) - downloadIndexFiles(downloadCompleteIndex=false,tmpIdxDir=../solr/data/index.20100302092400,latestVersion=1266003907838) [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(SnapPuller.java:541) - -->localIndexFile=/opt/solr_env/solr/data/index/_7h0y.fdx [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(SnapPuller.java:900) - fetchFile() [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(PostMethod.java:265) - enter PostMethod.addParameter(String, String) [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(EntityEnclosingMethod.java:150) - enter EntityEnclosingMethod.clearRequestBody() [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(PostMethod.java:265) - enter PostMethod.addParameter(String, String) [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(EntityEnclosingMethod.java:150) - enter EntityEnclosingMethod.clearRequestBody() [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(PostMethod.java:265) - enter PostMethod.addParameter(String, String) [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(EntityEnclosingMethod.java:150) - enter EntityEnclosingMethod.clearRequestBody() [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(PostMethod.java:265) - enter PostMethod.addParameter(String, String) [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(EntityEnclosingMethod.java:150) - enter EntityEnclosingMethod.clearRequestBody() [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(PostMethod.java:265) - enter PostMethod.addParameter(String, String) [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(EntityEnclosingMethod.java:150) - enter EntityEnclosingMethod.clearRequestBody() [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpClient.java:321) - enter HttpClient.executeMethod(HttpMethod) [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpClient.java:374) - enter HttpClient.executeMethod(HostConfiguration,HttpMethod,HttpState) [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(MultiThreadedHttpConnectionManager.java:405) - enter HttpConnectionManager.getConnectionWithTimeout(HostConfiguration, long) [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(MultiThreadedHttpConnectionManager.java:412) - HttpConnectionManager.getConnection: config = HostConfiguration[host=http://myserver.com:8983], timeout = 0 [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(MultiThreadedHttpConnectionManager.java:805) - enter HttpConnectionManager.ConnectionPool.getHostPool(HostConfiguration) [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(MultiThreadedHttpConnectionManager.java:805) - enter HttpConnectionManager.ConnectionPool.getHostPool(HostConfiguration) [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(MultiThreadedHttpConnectionManager.java:839) - Getting free connection, hostConfig=HostConfiguration[host=http://myserver.com:8983] [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodDirector.java:379) - Attempt number 1 to process request [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1079) - enter HttpMethodBase.execute(HttpState, HttpConnection) [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:2057) - enter HttpMethodBase.writeRequest(HttpState, HttpConnection) [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:2212) - enter HttpMethodBase.writeRequestLine(HttpState, HttpConnection) [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1496) - enter HttpMethodBase.generateRequestLine(HttpConnection, String, String, String, String) [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - >> "POST /solr/replication HTTP/1.1[\r][\n]" [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:1032) - enter HttpConnection.print(String) [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:942) - enter HttpConnection.write(byte[]) [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:963) - enter HttpConnection.write(byte[], int, int) [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:2175) - enter HttpMethodBase.writeRequestHeaders(HttpState,HttpConnection) [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(EntityEnclosingMethod.java:370) - enter EntityEnclosingMethod.addRequestHeaders(HttpState, HttpConnection) [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(ExpectContinueMethod.java:183) - enter ExpectContinueMethod.addRequestHeaders(HttpState, HttpConnection) [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1417) - enter HttpMethodBase.addRequestHeaders(HttpState, HttpConnection) [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1442) - enter HttpMethodBase.addUserAgentRequestHeaders(HttpState, HttpConnection) [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1328) - enter HttpMethodBase.addHostRequestHeader(HttpState, HttpConnection) [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(HttpMethodBase.java:1352) - Adding Host request header [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1266) - enter HttpMethodBase.addCookieRequestHeader(HttpState, HttpConnection) [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpState.java:173) - enter HttpState.getCookies() [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(CookieSpecBase.java:549) - enter CookieSpecBase.match(String, int, String, boolean, Cookie[]) [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1379) - enter HttpMethodBase.addProxyConnectionHeader(HttpState, HttpConnection) [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(EntityEnclosingMethod.java:401) - enter EntityEnclosingMethod.addContentLengthRequestHeader(HttpState, HttpConnection) [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(EntityEnclosingMethod.java:325) - enter EntityEnclosingMethod.getRequestContentLength() [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(PostMethod.java:135) - enter PostMethod.hasRequestContent() [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(PostMethod.java:240) - enter PostMethod.getParameters() [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:2324) - enter getContentCharSet( Header contentheader ) [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(HttpMethodBase.java:2342) - Default charset used: ISO-8859-1 [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:386) - HttpMethodBase.addRequestHeader(Header) [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(PostMethod.java:240) - enter PostMethod.getParameters() [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:2324) - enter getContentCharSet( Header contentheader ) [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HeaderElement.java:238) - enter HeaderElement.parseElements(String) [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HeaderElement.java:194) - enter HeaderElement.parseElements(char[]) [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HeaderElement.java:279) - enter HeaderElement.getParameterByName(String) [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(HttpMethodBase.java:2342) - Default charset used: ISO-8859-1 [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - >> "User-Agent: Jakarta Commons-HttpClient/3.1[\r][\n]" [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:1032) - enter HttpConnection.print(String) [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:942) - enter HttpConnection.write(byte[]) [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:963) - enter HttpConnection.write(byte[], int, int) [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - >> "Host: myserver.com:8983[\r][\n]" [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:1032) - enter HttpConnection.print(String) [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:942) - enter HttpConnection.write(byte[]) [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:963) - enter HttpConnection.write(byte[], int, int) [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - >> "Content-Length: 89[\r][\n]" [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:1032) - enter HttpConnection.print(String) [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:942) - enter HttpConnection.write(byte[]) [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:963) - enter HttpConnection.write(byte[], int, int) [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - >> "Content-Type: application/x-www-form-urlencoded[\r][\n]" [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:1032) - enter HttpConnection.print(String) [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:942) - enter HttpConnection.write(byte[]) [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:963) - enter HttpConnection.write(byte[], int, int) [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:1001) - enter HttpConnection.writeLine() [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:942) - enter HttpConnection.write(byte[]) [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:963) - enter HttpConnection.write(byte[], int, int) [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - >> "[\r][\n]" [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(EntityEnclosingMethod.java:469) - enter EntityEnclosingMethod.writeRequestBody(HttpState, HttpConnection) [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(PostMethod.java:135) - enter PostMethod.hasRequestContent() [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(EntityEnclosingMethod.java:325) - enter EntityEnclosingMethod.getRequestContentLength() [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(PostMethod.java:135) - enter PostMethod.hasRequestContent() [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:840) - enter HttpConnection.getRequestOutputStream() [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:84) - >> "command=filecontent&indexversion=1266003907838&file=_7h0y.fdx&checksum=true&wt=filestream" [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(EntityEnclosingMethod.java:508) - Request body sent [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:826) - enter HttpConnection.flushRequestOutputStream() [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1730) - enter HttpMethodBase.readResponse(HttpState, HttpConnection) [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1964) - enter HttpMethodBase.readStatusLine(HttpState, HttpConnection) [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:1113) - enter HttpConnection.readLine() [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:105) - enter HttpParser.readLine(InputStream, String) [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:74) - enter HttpParser.readRawLine() [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - << "HTTP/1.1 200 OK[\r][\n]" [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - << "HTTP/1.1 200 OK[\r][\n]" [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1930) - enter HttpMethodBase.readResponseHeaders(HttpState,HttpConnection) [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:857) - enter HttpConnection.getResponseInputStream() [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:170) - enter HeaderParser.parseHeaders(InputStream, String) [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:105) - enter HttpParser.readLine(InputStream, String) [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:74) - enter HttpParser.readRawLine() [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - << "Date: Tue, 02 Mar 2010 21:24:40 GMT[\r][\n]" [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:105) - enter HttpParser.readLine(InputStream, String) [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:74) - enter HttpParser.readRawLine() [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - << "Content-Type: application/octet-stream[\r][\n]" [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:105) - enter HttpParser.readLine(InputStream, String) [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:74) - enter HttpParser.readRawLine() [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - << "Transfer-Encoding: chunked[\r][\n]" [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:105) - enter HttpParser.readLine(InputStream, String) [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:74) - enter HttpParser.readRawLine() [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - << "Server: Jetty(6.1.19)[\r][\n]" [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:105) - enter HttpParser.readLine(InputStream, String) [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:74) - enter HttpParser.readRawLine() [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - << "[\r][\n]" [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1579) - enter HttpMethodBase.processResponseHeaders(HttpState, HttpConnection) [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1615) - enter HttpMethodBase.processCookieHeaders(Header[], HttpState, HttpConnection) [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1781) - enter HttpMethodBase.readResponseBody(HttpState, HttpConnection) [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:1816) - enter HttpMethodBase.readResponseBody(HttpConnection) [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:857) - enter HttpConnection.getResponseInputStream() [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpMethodBase.java:2281) - enter HttpMethodBase.canResponseHaveBody(int) [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HeaderElement.java:238) - enter HeaderElement.parseElements(String) [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HeaderElement.java:194) - enter HeaderElement.parseElements(char[]) [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:894) - enter HttpConnection.isResponseAvailable(int) [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:84) - << "4" [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:84) - << "[\r]" [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - << "[\n]" [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:84) - << "[0x0][0x0][0x0][0x0]" [2010-03-02 21:24:40] [pool-3-thread-1] WARN (SnapPuller.java:950) - No content recieved for file: {size=5324, name=_7h0y.fdx, lastmodified=1267565039000} [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:84) - << "[\r]" [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - << "[\n]" [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:84) - << "0" [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:84) - << "[\r]" [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - << "[\n]" [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:170) - enter HeaderParser.parseHeaders(InputStream, String) [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:105) - enter HttpParser.readLine(InputStream, String) [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpParser.java:74) - enter HttpParser.readRawLine() [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:84) - << "[\r]" [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - << "[\n]" [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(Wire.java:70) - << "[\r][\n]" [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(HttpMethodBase.java:1024) - Resorting to protocol version default close connection policy [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(HttpMethodBase.java:1028) - Should NOT close connection, using HTTP/1.1 [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:874) - enter HttpConnection.isResponseAvailable() [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(HttpConnection.java:1174) - enter HttpConnection.releaseConnection() [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(HttpConnection.java:1178) - Releasing connection back to connection manager. [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(MultiThreadedHttpConnectionManager.java:630) - enter HttpConnectionManager.releaseConnection(HttpConnection) [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(MultiThreadedHttpConnectionManager.java:979) - Freeing connection, hostConfig=HostConfiguration[host=http://myserver.com:8983] [2010-03-02 21:24:40] [pool-3-thread-1] TRACE(MultiThreadedHttpConnectionManager.java:805) - enter HttpConnectionManager.ConnectionPool.getHostPool(HostConfiguration) [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(IdleConnectionHandler.java:76) - Adding connection at: 1267565080137 [2010-03-02 21:24:40] [pool-3-thread-1] DEBUG(MultiThreadedHttpConnectionManager.java:961) - Notifying no-one, there are no waiting threads [2010-03-02 21:24:40] [pool-3-thread-1] ERROR(ReplicationHandler.java:266) - SnapPull failed org.apache.solr.common.SolrException: Unable to download _7h0y.fdx completely. Downloaded 0!=5324 at org.apache.solr.handler.SnapPuller$FileFetcher.cleanup(SnapPuller.java:1041) at org.apache.solr.handler.SnapPuller$FileFetcher.fetchFile(SnapPuller.java:921) at org.apache.solr.handler.SnapPuller.downloadIndexFiles(SnapPuller.java:545) at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:294) at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:264) at org.apache.solr.handler.SnapPuller$1.run(SnapPuller.java:159) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417) at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:280) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:135) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:65) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:146) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:170) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675) at java.lang.Thread.run(Thread.java:595)
