Replication Index fetch failed

Thu, 01 Sep 2016 05:35:59 -0700 

Hi

Replication seems to be in an endless loop. Anybody any idea?
See below for logs.

If you need more info, just let me know...
INFO - 2016-09-01 14:30:42.563; [c:lvs s:shard1 r:core_node10 x:lvs_shard1_replica1] org.apache.solr.core.SolrDeletionPolicy; SolrDeletionPolicy.onCommit: commits: num=2 commit{dir=NRTCachingDirectory(MMapDirectory@/var/solr/data/lvs_shard1_replica1/data/index.20160901140036922 lockFactory=org.apache.lucene.store.NativeFSLockFactory@59509f2; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_vpo,generation=41100} commit{dir=NRTCachingDirectory(MMapDirectory@/var/solr/data/lvs_shard1_replica1/data/index.20160901140036922 lockFactory=org.apache.lucene.store.NativeFSLockFactory@59509f2; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_vpp,generation=41101} INFO - 2016-09-01 14:30:42.563; [c:lvs s:shard1 r:core_node10 x:lvs_shard1_replica1] org.apache.solr.core.SolrDeletionPolicy; newest commit generation = 41101 INFO - 2016-09-01 14:30:42.565; [c:lvs s:shard1 r:core_node10 x:lvs_shard1_replica1] org.apache.solr.update.DirectUpdateHandler2; end_commit_flush INFO - 2016-09-01 14:30:42.565; [c:lvs s:shard1 r:core_node10 x:lvs_shard1_replica1] org.apache.solr.update.DirectUpdateHandler2; start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false} INFO - 2016-09-01 14:30:42.603; [c:lvs s:shard1 r:core_node10 x:lvs_shard1_replica1] org.apache.solr.core.SolrDeletionPolicy; SolrDeletionPolicy.onCommit: commits: num=2 commit{dir=NRTCachingDirectory(MMapDirectory@/var/solr/data/lvs_shard1_replica1/data/index.20160901140036922 lockFactory=org.apache.lucene.store.NativeFSLockFactory@59509f2; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_vpp,generation=41101} commit{dir=NRTCachingDirectory(MMapDirectory@/var/solr/data/lvs_shard1_replica1/data/index.20160901140036922 lockFactory=org.apache.lucene.store.NativeFSLockFactory@59509f2; maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_vpq,generation=41102} INFO - 2016-09-01 14:30:42.603; [c:lvs s:shard1 r:core_node10 x:lvs_shard1_replica1] org.apache.solr.core.SolrDeletionPolicy; newest commit generation = 41102 INFO - 2016-09-01 14:30:42.664; [c:lvs s:shard1 r:core_node10 x:lvs_shard1_replica1] org.apache.solr.search.SolrIndexSearcher; Opening [Searcher@3436f207[lvs_shard1_replica1] realtime] INFO - 2016-09-01 14:30:42.674; [c:lvs s:shard1 r:core_node10 x:lvs_shard1_replica1] org.apache.solr.update.DirectUpdateHandler2; end_commit_flush ERROR - 2016-09-01 14:30:43.653; [c:intradesk s:shard1 r:core_node5 x:intradesk_shard1_replica1] org.apache.solr.common.SolrException; Index fetch failed :org.apache.solr.common.SolrException: Unable to download _6f46_cj.liv completely. Downloaded 0!=5596 at org.apache.solr.handler.IndexFetcher$FileFetcher.cleanup(IndexFetcher.java:1554) at org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1437) at org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:852) at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:428) at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:251) at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:388) at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:156) at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:408) at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:221) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$22(ExecutorUtil.java:229) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
    at java.lang.Thread.run(Thread.java:745)
ERROR - 2016-09-01 14:30:43.654; [c:intradesk s:shard1 r:core_node5 x:intradesk_shard1_replica1] org.apache.solr.common.SolrException; Error while trying to recover:org.apache.solr.common.SolrException: Replication for recovery failed. at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:159) at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:408) at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:221) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) 
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$22(ExecutorUtil.java:229) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
    at java.lang.Thread.run(Thread.java:745)
INFO - 2016-09-01 14:30:43.654; [c:intradesk s:shard1 r:core_node5 x:intradesk_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy; Replay not started, or was not successful... still buffering updates. ERROR - 2016-09-01 14:30:43.654; [c:intradesk s:shard1 r:core_node5 x:intradesk_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy; Recovery failed - trying again... (0) INFO - 2016-09-01 14:30:43.654; [c:intradesk s:shard1 r:core_node5 x:intradesk_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy; Wait [2.0] seconds before trying to recover again (attempt=1) INFO - 2016-09-01 14:30:53.655; [c:intradesk s:shard1 r:core_node5 x:intradesk_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy; Begin buffering updates. core=[intradesk_shard1_replica1] INFO - 2016-09-01 14:30:53.747; [c:intradesk s:shard1 r:core_node5 x:intradesk_shard1_replica1] org.apache.solr.update.UpdateLog; Restarting buffering. previous=RecoveryInfo{adds=0 deletes=0 deleteByQuery=0 errors=0 positionOfStart=0} INFO - 2016-09-01 14:30:53.748; [c:intradesk s:shard1 r:core_node5 x:intradesk_shard1_replica1] org.apache.solr.update.UpdateLog; Starting to buffer updates. FSUpdateLog{state=BUFFERING, tlog=tlog{file=/var/solr/data/intradesk_shard1_replica1/data/tlog/tlog.0000000000000122833 refcount=1}} INFO - 2016-09-01 14:30:53.748; [c:intradesk s:shard1 r:core_node5 x:intradesk_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy; Publishing state of core [intradesk_shard1_replica1] as recovering, leader is [http://solr20-gs:8983/solr/intradesk_shard1_replica2/] and I am [http://solr20-dcg:8983/solr/intradesk_shard1_replica1/] INFO - 2016-09-01 14:30:53.748; [c:intradesk s:shard1 r:core_node5 x:intradesk_shard1_replica1] org.apache.solr.cloud.ZkController; publishing state=recovering INFO - 2016-09-01 14:30:53.768; [c:intradesk s:shard1 r:core_node5 x:intradesk_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy; Sending prep recovery command to [http://solr20-gs:8983/solr]; [WaitForState: action=PREPRECOVERY&core=intradesk_shard1_replica2&nodeName=solr20-dcg:8983_solr&coreNodeName=core_node5&state=recovering&checkLive=true&onlyIfLeader=true&onlyIfLeaderActive=true] INFO - 2016-09-01 14:30:53.780; [ ] org.apache.solr.common.cloud.ZkStateReader$StateWatcher; A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/intradesk/state.json] for collection [intradesk] has occurred - updating... (live nodes size: [21]) INFO - 2016-09-01 14:30:53.781; [ ] org.apache.solr.common.cloud.ZkStateReader; Updating data for [intradesk] from [12593] to [12594] INFO - 2016-09-01 14:31:00.776; [c:intradesk s:shard1 r:core_node5 x:intradesk_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy; Starting Replication Recovery. INFO - 2016-09-01 14:31:00.776; [c:intradesk s:shard1 r:core_node5 x:intradesk_shard1_replica1] org.apache.solr.cloud.RecoveryStrategy; Attempting to replicate from [http://solr20-gs:8983/solr/intradesk_shard1_replica2/]. INFO - 2016-09-01 14:31:00.783; [c:intradesk s:shard1 r:core_node5 x:intradesk_shard1_replica1] org.apache.solr.handler.IndexFetcher; Master's generation: 123527 INFO - 2016-09-01 14:31:00.783; [c:intradesk s:shard1 r:core_node5 x:intradesk_shard1_replica1] org.apache.solr.handler.IndexFetcher; Slave's generation: 2 INFO - 2016-09-01 14:31:00.783; [c:intradesk s:shard1 r:core_node5 x:intradesk_shard1_replica1] org.apache.solr.handler.IndexFetcher; Starting replication process INFO - 2016-09-01 14:31:01.461; [c:intradesk s:shard1 r:core_node5 x:intradesk_shard1_replica1] org.apache.solr.handler.IndexFetcher; Number of files in latest index in master: 210 INFO - 2016-09-01 14:31:01.462; [c:intradesk s:shard1 r:core_node5 x:intradesk_shard1_replica1] org.apache.solr.core.CachingDirectoryFactory; return new directory for /var/solr/data/intradesk_shard1_replica1/data/index.20160901143101462 INFO - 2016-09-01 14:31:01.463; [c:intradesk s:shard1 r:core_node5 x:intradesk_shard1_replica1] org.apache.solr.handler.IndexFetcher; Starting download (fullCopy=true) to NRTCachingDirectory(MMapDirectory@/var/solr/data/intradesk_shard1_replica1/data/index.20160901143101462 lockFactory=org.apache.lucene.store.NativeFSLockFactory@59509f2; maxCacheMB=48.0 maxMergeSizeMB=4.0)

Reply via email to