Hi
Replication seems to be in an endless loop. Anybody any idea?
See below for logs.
If you need more info, just let me know...
INFO - 2016-09-01 14:30:42.563; [c:lvs s:shard1 r:core_node10
x:lvs_shard1_replica1] org.apache.solr.core.SolrDeletionPolicy;
SolrDeletionPolicy.onCommit: commits: num=2
commit{dir=NRTCachingDirectory(MMapDirectory@/var/solr/data/lvs_shard1_replica1/data/index.20160901140036922
lockFactory=org.apache.lucene.store.NativeFSLockFactory@59509f2;
maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_vpo,generation=41100}
commit{dir=NRTCachingDirectory(MMapDirectory@/var/solr/data/lvs_shard1_replica1/data/index.20160901140036922
lockFactory=org.apache.lucene.store.NativeFSLockFactory@59509f2;
maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_vpp,generation=41101}
INFO - 2016-09-01 14:30:42.563; [c:lvs s:shard1 r:core_node10
x:lvs_shard1_replica1] org.apache.solr.core.SolrDeletionPolicy; newest
commit generation = 41101
INFO - 2016-09-01 14:30:42.565; [c:lvs s:shard1 r:core_node10
x:lvs_shard1_replica1] org.apache.solr.update.DirectUpdateHandler2;
end_commit_flush
INFO - 2016-09-01 14:30:42.565; [c:lvs s:shard1 r:core_node10
x:lvs_shard1_replica1] org.apache.solr.update.DirectUpdateHandler2;
start
commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
INFO - 2016-09-01 14:30:42.603; [c:lvs s:shard1 r:core_node10
x:lvs_shard1_replica1] org.apache.solr.core.SolrDeletionPolicy;
SolrDeletionPolicy.onCommit: commits: num=2
commit{dir=NRTCachingDirectory(MMapDirectory@/var/solr/data/lvs_shard1_replica1/data/index.20160901140036922
lockFactory=org.apache.lucene.store.NativeFSLockFactory@59509f2;
maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_vpp,generation=41101}
commit{dir=NRTCachingDirectory(MMapDirectory@/var/solr/data/lvs_shard1_replica1/data/index.20160901140036922
lockFactory=org.apache.lucene.store.NativeFSLockFactory@59509f2;
maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_vpq,generation=41102}
INFO - 2016-09-01 14:30:42.603; [c:lvs s:shard1 r:core_node10
x:lvs_shard1_replica1] org.apache.solr.core.SolrDeletionPolicy; newest
commit generation = 41102
INFO - 2016-09-01 14:30:42.664; [c:lvs s:shard1 r:core_node10
x:lvs_shard1_replica1] org.apache.solr.search.SolrIndexSearcher; Opening
[Searcher@3436f207[lvs_shard1_replica1] realtime]
INFO - 2016-09-01 14:30:42.674; [c:lvs s:shard1 r:core_node10
x:lvs_shard1_replica1] org.apache.solr.update.DirectUpdateHandler2;
end_commit_flush
ERROR - 2016-09-01 14:30:43.653; [c:intradesk s:shard1 r:core_node5
x:intradesk_shard1_replica1] org.apache.solr.common.SolrException; Index
fetch failed :org.apache.solr.common.SolrException: Unable to download
_6f46_cj.liv completely. Downloaded 0!=5596
at
org.apache.solr.handler.IndexFetcher$FileFetcher.cleanup(IndexFetcher.java:1554)
at
org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1437)
at
org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:852)
at
org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:428)
at
org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:251)
at
org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:388)
at
org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:156)
at
org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:408)
at
org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:221)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$22(ExecutorUtil.java:229)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
ERROR - 2016-09-01 14:30:43.654; [c:intradesk s:shard1 r:core_node5
x:intradesk_shard1_replica1] org.apache.solr.common.SolrException; Error
while trying to recover:org.apache.solr.common.SolrException:
Replication for recovery failed.
at
org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:159)
at
org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:408)
at
org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:221)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$22(ExecutorUtil.java:229)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
INFO - 2016-09-01 14:30:43.654; [c:intradesk s:shard1 r:core_node5
x:intradesk_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy;
Replay not started, or was not successful... still buffering updates.
ERROR - 2016-09-01 14:30:43.654; [c:intradesk s:shard1 r:core_node5
x:intradesk_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy;
Recovery failed - trying again... (0)
INFO - 2016-09-01 14:30:43.654; [c:intradesk s:shard1 r:core_node5
x:intradesk_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy;
Wait [2.0] seconds before trying to recover again (attempt=1)
INFO - 2016-09-01 14:30:53.655; [c:intradesk s:shard1 r:core_node5
x:intradesk_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy;
Begin buffering updates. core=[intradesk_shard1_replica1]
INFO - 2016-09-01 14:30:53.747; [c:intradesk s:shard1 r:core_node5
x:intradesk_shard1_replica1] org.apache.solr.update.UpdateLog;
Restarting buffering. previous=RecoveryInfo{adds=0 deletes=0
deleteByQuery=0 errors=0 positionOfStart=0}
INFO - 2016-09-01 14:30:53.748; [c:intradesk s:shard1 r:core_node5
x:intradesk_shard1_replica1] org.apache.solr.update.UpdateLog; Starting
to buffer updates. FSUpdateLog{state=BUFFERING,
tlog=tlog{file=/var/solr/data/intradesk_shard1_replica1/data/tlog/tlog.0000000000000122833
refcount=1}}
INFO - 2016-09-01 14:30:53.748; [c:intradesk s:shard1 r:core_node5
x:intradesk_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy;
Publishing state of core [intradesk_shard1_replica1] as recovering,
leader is [http://solr20-gs:8983/solr/intradesk_shard1_replica2/] and I
am [http://solr20-dcg:8983/solr/intradesk_shard1_replica1/]
INFO - 2016-09-01 14:30:53.748; [c:intradesk s:shard1 r:core_node5
x:intradesk_shard1_replica1] org.apache.solr.cloud.ZkController;
publishing state=recovering
INFO - 2016-09-01 14:30:53.768; [c:intradesk s:shard1 r:core_node5
x:intradesk_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy;
Sending prep recovery command to [http://solr20-gs:8983/solr];
[WaitForState:
action=PREPRECOVERY&core=intradesk_shard1_replica2&nodeName=solr20-dcg:8983_solr&coreNodeName=core_node5&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true]
INFO - 2016-09-01 14:30:53.780; [ ]
org.apache.solr.common.cloud.ZkStateReader$StateWatcher; A cluster state
change: [WatchedEvent state:SyncConnected type:NodeDataChanged
path:/collections/intradesk/state.json] for collection [intradesk] has
occurred - updating... (live nodes size: [21])
INFO - 2016-09-01 14:30:53.781; [ ]
org.apache.solr.common.cloud.ZkStateReader; Updating data for
[intradesk] from [12593] to [12594]
INFO - 2016-09-01 14:31:00.776; [c:intradesk s:shard1 r:core_node5
x:intradesk_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy;
Starting Replication Recovery.
INFO - 2016-09-01 14:31:00.776; [c:intradesk s:shard1 r:core_node5
x:intradesk_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy;
Attempting to replicate from
[http://solr20-gs:8983/solr/intradesk_shard1_replica2/].
INFO - 2016-09-01 14:31:00.783; [c:intradesk s:shard1 r:core_node5
x:intradesk_shard1_replica1] org.apache.solr.handler.IndexFetcher;
Master's generation: 123527
INFO - 2016-09-01 14:31:00.783; [c:intradesk s:shard1 r:core_node5
x:intradesk_shard1_replica1] org.apache.solr.handler.IndexFetcher;
Slave's generation: 2
INFO - 2016-09-01 14:31:00.783; [c:intradesk s:shard1 r:core_node5
x:intradesk_shard1_replica1] org.apache.solr.handler.IndexFetcher;
Starting replication process
INFO - 2016-09-01 14:31:01.461; [c:intradesk s:shard1 r:core_node5
x:intradesk_shard1_replica1] org.apache.solr.handler.IndexFetcher;
Number of files in latest index in master: 210
INFO - 2016-09-01 14:31:01.462; [c:intradesk s:shard1 r:core_node5
x:intradesk_shard1_replica1]
org.apache.solr.core.CachingDirectoryFactory; return new directory for
/var/solr/data/intradesk_shard1_replica1/data/index.20160901143101462
INFO - 2016-09-01 14:31:01.463; [c:intradesk s:shard1 r:core_node5
x:intradesk_shard1_replica1] org.apache.solr.handler.IndexFetcher;
Starting download (fullCopy=true) to
NRTCachingDirectory(MMapDirectory@/var/solr/data/intradesk_shard1_replica1/data/index.20160901143101462
lockFactory=org.apache.lucene.store.NativeFSLockFactory@59509f2;
maxCacheMB=48.0 maxMergeSizeMB=4.0)