Any other JVM settings change possible? On Tue, Apr 28, 2020, 10:15 PM Sethuraman, Ganesh <sethuram...@dnb.com.invalid> wrote:
> Hi > > We are using SolrCloud 7.2.1 with 3 node Zookeeper ensemble. We have 92 > collection each on avg. having 8 shards and 2 replica with 2 EC2 nodes, > with JVM size of 18GB (G1 GC). We need your help with the Issue we faced > today: The issue is SolrCloud server went into a degraded collections (for > few collections) when the Solr backup and the Solr batch CSV update load > happened at the same time as backup. The CSV data load was about ~5 GB per > shard/replica. We think this happened after zkClient disconnect happened as > noted below. We had to restart Solr to bring it back to normal. > > > 1. Is it not suggested to run backup and Solr batch CSV update large > load at the same time? > 2. In the past we have seen two CSV batch update load in parallel > causes issues, is this also not suggested (this issue is not related to > that)? > 3. Do you think we should increase Zookeeper timeout? > 4. How do we know if we need to up the JVM Max memory, and by how much? > 5. We also see that once the Solr goes into degraded collection and > recovery failed, it NEVER get back to normal, even after when there is no > load. Is this a bug? > > The GC information and Solr Log below > > > https://gceasy.io/my-gc-report.jsp?p=c2hhcmVkLzIwMjAvMDQvMjkvLS0wMl9zb2xyX2djLmxvZy56aXAtLTEtNDAtMzE=&channel=WEB > > > 2020-04-27 07:34:07.322 WARN > (zkConnectionManagerCallback-6-thread-1-processing-n:mysolrsever.com:6010_solr-SendThread(zoo-prd-n1:2181)) > [ ] o.a.z.ClientCnxn Client session timed out, have not heard from server > in 10775ms for sessionid 0x171a6fb51310008 > .... > 2020-04-27 07:34:07.426 WARN > (zkConnectionManagerCallback-6-thread-1-processing-n:mysolrsever.com:6010_solr-EventThread) > [ ] o.a.s.c.c.ConnectionManager zkClient has disconnected > > > > > SOLR Log Below (Curtailed WARN log) > -------------------- > 2020-04-27 07:26:45.402 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of > 587 bytes) > java.io.EOFException > at > org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168) > at > org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160) > at > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579) > at > org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545) > at > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526) > at > org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008) > at > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566) > at > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345) > at > org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420) > at > org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225) > at > org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626) > at > org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308) > at > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292) > at > com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > 2020-04-27 07:26:45.405 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of > 587 bytes) > java.io.EOFException > at > org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168) > at > org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160) > at > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579) > at > org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545) > at > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526) > at > org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008) > at > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566) > at > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345) > at > org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420) > at > org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225) > at > org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626) > at > org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308) > at > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292) > at > com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > 2020-04-27 07:26:45.406 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of > 587 bytes) > java.io.EOFException > at > org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168) > at > org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160) > at > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579) > at > org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545) > at > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526) > at > org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008) > at > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566) > at > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345) > at > org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420) > at > org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225) > at > org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626) > at > org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308) > at > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292) > at > com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > 2020-04-27 07:26:45.408 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of > 587 bytes) > java.io.EOFException > at > org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168) > at > org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160) > at > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579) > at > org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545) > at > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526) > at > org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008) > at > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566) > at > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345) > at > org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420) > at > org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225) > at > org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626) > at > org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308) > at > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292) > at > com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > 2020-04-27 07:26:45.410 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of > 587 bytes) > java.io.EOFException > at > org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168) > at > org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160) > at > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579) > at > org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545) > at > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526) > at > org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008) > at > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566) > at > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345) > at > org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420) > at > org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225) > at > org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626) > at > org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308) > at > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292) > at > com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > 2020-04-27 07:26:45.412 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of > 587 bytes) > java.io.EOFException > at > org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168) > at > org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160) > at > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579) > at > org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545) > at > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526) > at > org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008) > at > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566) > at > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345) > at > org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420) > at > org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225) > at > org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626) > at > org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308) > at > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292) > at > com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > 2020-04-27 07:26:45.412 ERROR > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher Error fetching file, doing one > retry...:org.apache.solr.common.SolrException: Unable to download > _1kr_r.liv completely. Downloaded 0!=587 > at > org.apache.solr.handler.IndexFetcher$FileFetcher.cleanup(IndexFetcher.java:1673) > at > org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1556) > at > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526) > at > org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008) > at > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566) > at > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345) > at > org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420) > at > org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225) > at > org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626) > at > org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308) > at > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292) > at > com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > > 2020-04-27 07:26:45.414 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of > 587 bytes) > java.io.EOFException > at > org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168) > at > org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160) > at > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579) > at > org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545) > at > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1531) > at > org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008) > at > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566) > at > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345) > at > org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420) > at > org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225) > at > org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626) > at > org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308) > at > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292) > at > com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > 2020-04-27 07:26:45.414 ERROR > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher Error deleting file: _1kr_r.liv > java.nio.file.NoSuchFileException: > /data/dnb/cloud/node1/solr/mycollection_shard13_replica_n48/data/index.20200427072258517/_1kr_r.liv > at > sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) > at > sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) > at > sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) > at > sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:244) > at > sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103) > at java.nio.file.Files.delete(Files.java:1126) > at > org.apache.lucene.store.FSDirectory.privateDeleteFile(FSDirectory.java:373) > at > org.apache.lucene.store.FSDirectory.deleteFile(FSDirectory.java:335) > at > org.apache.lucene.store.NRTCachingDirectory.deleteFile(NRTCachingDirectory.java:119) > at > org.apache.solr.handler.IndexFetcher$DirectoryFile.delete(IndexFetcher.java:1757) > at > org.apache.solr.handler.IndexFetcher$FileFetcher.cleanup(IndexFetcher.java:1667) > at > org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1556) > at > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1531) > at > org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008) > at > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566) > at > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345) > at > org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420) > at > org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225) > at > org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626) > at > org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308) > at > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292) > at > com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > 2020-04-27 07:26:45.449 WARN > (recoveryExecutor-4-thread-699-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard2_replica_n4 s:shard2 c:mycollection r:core_node7) > [c:mycollection s:shard2 r:core_node7 x:mycollection_shard2_replica_n4] > o.a.s.h.IndexFetcher File _1kq.cfs did not match. expected checksum is > 3934327071 and actual is checksum 3095753427. expected length is 33235371 > and actual length is 54180642 > 2020-04-27 07:26:45.449 WARN > (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) > [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] > o.a.s.h.IndexFetcher File _1em.fnm did not match. expected checksum is > 138861107 and actual is checksum 188997082. expected length is 107460 and > actual length is 107460 > 2020-04-27 07:26:45.457 WARN > (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) > [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] > o.a.s.h.IndexFetcher File _1mt_Lucene50_0.tim did not match. expected > checksum is 2947323803 and actual is checksum 979431341. expected length is > 32176 and actual length is 26089 > 2020-04-27 07:26:45.521 WARN > (recoveryExecutor-4-thread-648-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard6_replica_n20 s:shard6 c:mycollection r:core_node23) > [c:mycollection s:shard6 r:core_node23 x:mycollection_shard6_replica_n20] > o.a.s.h.IndexFetcher File _1gk_Lucene50_0.doc did not match. expected > checksum is 2087731087 and actual is checksum 3540174119. expected length > is 110 and actual length is 110 > 2020-04-27 07:26:45.521 WARN > (recoveryExecutor-4-thread-657-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard1_replica_n1 s:shard1 c:mycollection r:core_node3) > [c:mycollection s:shard1 r:core_node3 x:mycollection_shard1_replica_n1] > o.a.s.h.IndexFetcher File _1mo.cfs did not match. expected checksum is > 777834759 and actual is checksum 289576304. expected length is 59980991 and > actual length is 6236424 > 2020-04-27 07:26:45.521 WARN > (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) > [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] > o.a.s.h.IndexFetcher File _1mt.fdx did not match. expected checksum is > 1321210176 and actual is checksum 224636564. expected length is 114 and > actual length is 106 > 2020-04-27 07:26:45.521 WARN > (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) > [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] > o.a.s.h.IndexFetcher File _1em.fdt did not match. expected checksum is > 3790133021 and actual is checksum 2516047051. expected length is 1477755 > and actual length is 1477850 > 2020-04-27 07:26:45.578 WARN > (recoveryExecutor-4-thread-648-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard6_replica_n20 s:shard6 c:mycollection r:core_node23) > [c:mycollection s:shard6 r:core_node23 x:mycollection_shard6_replica_n20] > o.a.s.h.IndexFetcher File _1gk_Lucene50_0.tim did not match. expected > checksum is 3690299979 and actual is checksum 411724865. expected length is > 4392270 and actual length is 4396498 > 2020-04-27 07:26:45.588 WARN > (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) > [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] > o.a.s.h.IndexFetcher File _1em_Lucene50_0.doc did not match. expected > checksum is 937224718 and actual is checksum 1176441880. expected length is > 110 and actual length is 110 > 2020-04-27 07:26:45.733 WARN > (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) > [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] > o.a.s.h.IndexFetcher File _1em_Lucene50_0.tim did not match. expected > checksum is 1429458386 and actual is checksum 4155623862. expected length > is 2213909 and actual length is 2213939 > 2020-04-27 07:26:45.754 ERROR > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.ReplicationHandler Index fetch failed > :org.apache.solr.common.SolrException: Unable to download _1kr_r.liv > completely. Downloaded 0!=587 > at > org.apache.solr.handler.IndexFetcher$FileFetcher.cleanup(IndexFetcher.java:1673) > at > org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1556) > at > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1531) > at > org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008) > at > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566) > at > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345) > at > org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420) > at > org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225) > at > org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626) > at > org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308) > at > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292) > at > com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > > 2020-04-27 07:26:45.755 ERROR > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.c.RecoveryStrategy Error while trying to > recover:org.apache.solr.common.SolrException: Replication for recovery > failed. > at > org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:228) > at > org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626) > at > org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308) > at > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292) > at > com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > > 2020-04-27 07:26:45.755 ERROR > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.c.RecoveryStrategy Recovery failed - trying again... (1) > 2020-04-27 07:26:45.780 WARN > (recoveryExecutor-4-thread-648-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard6_replica_n20 s:shard6 c:mycollection r:core_node23) > [c:mycollection s:shard6 r:core_node23 x:mycollection_shard6_replica_n20] > o.a.s.h.IndexFetcher File _1gk_Lucene70_0.dvm did not match. expected > checksum is 1474431645 and actual is checksum 1271168044. expected length > is 275570 and actual length is 274506 > 2020-04-27 07:26:45.834 WARN > (recoveryExecutor-4-thread-648-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard6_replica_n20 s:shard6 c:mycollection r:core_node23) > [c:mycollection s:shard6 r:core_node23 x:mycollection_shard6_replica_n20] > o.a.s.h.IndexFetcher File _1gk.fdx did not match. expected checksum is > 4129522944 and actual is checksum 4080519415. expected length is 6626 and > actual length is 6626 > 2020-04-27 07:26:45.843 WARN > (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) > [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] > o.a.s.h.IndexFetcher File _1em.si did not match. expected checksum is > 2493366746 and actual is checksum 31587394. expected length is 546 and > actual length is 546 > 2020-04-27 07:26:45.882 WARN > (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) > [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] > o.a.s.h.IndexFetcher File _1mv.fdx did not match. expected checksum is > 1392813257 and actual is checksum 3455988143. expected length is 103 and > actual length is 111 > 2020-04-27 07:26:45.882 WARN > (recoveryExecutor-4-thread-648-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard6_replica_n20 s:shard6 c:mycollection r:core_node23) > [c:mycollection s:shard6 r:core_node23 x:mycollection_shard6_replica_n20] > o.a.s.h.IndexFetcher File _1gk_Lucene50_0.tip did not match. expected > checksum is 2301226889 and actual is checksum 29696594. expected length is > 86459 and actual length is 85786 > 2020-04-27 07:26:45.882 WARN > (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) > [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] > o.a.s.h.IndexFetcher File _1em_Lucene70_0.dvm did not match. expected > checksum is 112676849 and actual is checksum 354222195. expected length is > 286474 and actual length is 286546 > 2020-04-27 07:26:45.911 WARN > (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) > [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] > o.a.s.h.IndexFetcher File _1mv.fnm did not match. expected checksum is > 2203892485 and actual is checksum 1165086749. expected length is 107460 and > actual length is 106580 > 2020-04-27 07:26:45.911 WARN > (recoveryExecutor-4-thread-648-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard6_replica_n20 s:shard6 c:mycollection r:core_node23) > [c:mycollection s:shard6 r:core_node23 x:mycollection_shard6_replica_n20] > o.a.s.h.IndexFetcher File _1gk_2z.liv did not match. expected checksum is > 3175370827 and actual is checksum 520836483. expected length is 51396 and > actual length is 51404 > 2020-04-27 07:26:45.912 WARN > (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) > [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] > o.a.s.h.IndexFetcher File _1em.fdx did not match. expected checksum is > 967260408 and actual is checksum 969726862. expected length is 3180 and > actual length is 3180 > 2020-04-27 07:26:45.914 WARN > (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) > [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] > o.a.s.h.IndexFetcher File _1mv.fdt did not match. expected checksum is > 719562508 and actual is checksum 956062095. expected length is 16717 and > actual length is 21454 > 2020-04-27 07:26:45.915 WARN > (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) > [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] > o.a.s.h.IndexFetcher File _1em_Lucene50_0.tip did not match. expected > checksum is 3833337361 and actual is checksum 493358852. expected length is > 43996 and actual length is 44026 > 2020-04-27 07:26:45.916 WARN > (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) > [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] > o.a.s.h.IndexFetcher File _1mv_Lucene50_0.doc did not match. expected > checksum is 1533628328 and actual is checksum 3735455395. expected length > is 110 and actual length is 110 > 2020-04-27 07:26:45.917 WARN > (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) > [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] > o.a.s.h.IndexFetcher File _1em_Lucene70_0.dvd did not match. expected > checksum is 75152145 and actual is checksum 1665090946. expected length is > 497241188 and actual length is 497254814 > 2020-04-27 07:26:45.918 WARN > (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) > [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] > o.a.s.h.IndexFetcher File _1mv_Lucene50_0.tim did not match. expected > checksum is 1349491281 and actual is checksum 3808036883. expected length > is 25153 and actual length is 32322 > 2020-04-27 07:26:45.920 WARN > (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) > [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] > o.a.s.h.IndexFetcher File _1mv_Lucene50_0.tip did not match. expected > checksum is 3820733560 and actual is checksum 2819887160. expected length > is 571 and actual length is 725 > 2020-04-27 07:26:46.034 WARN > (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) > [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] > o.a.s.h.IndexFetcher File _1mv_Lucene70_0.dvm did not match. expected > checksum is 1547030244 and actual is checksum 1122196351. expected length > is 324313 and actual length is 323553 > 2020-04-27 07:26:46.170 WARN > (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) > [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] > o.a.s.h.IndexFetcher File _1mv.si did not match. expected checksum is > 751850967 and actual is checksum 1026527313. expected length is 508 and > actual length is 508 > and actual length is 319133 > 2020-04-27 07:27:30.567 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1mr_Lucene70_0.dvd did not match. expected > checksum is 3435029878 and actual is checksum 1202327807. expected length > is 3337719 and actual length is 3375106 > 2020-04-27 07:27:30.578 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1mr.fdx did not match. expected checksum is > 2264443519 and actual is checksum 2299249501. expected length is 104 and > actual length is 102 > 2020-04-27 07:27:30.580 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1mr.si did not match. expected checksum is > 1184027188 and actual is checksum 2810465617. expected length is 508 and > actual length is 508 > 2020-04-27 07:27:30.622 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1mv.fdx did not match. expected checksum is > 2246788287 and actual is checksum 2682862227. expected length is 92 and > actual length is 96 > 2020-04-27 07:27:30.625 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1mv.fnm did not match. expected checksum is > 3227520043 and actual is checksum 2778425164. expected length is 107137 and > actual length is 107137 > 2020-04-27 07:27:30.627 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1mv.fdt did not match. expected checksum is > 1964200539 and actual is checksum 3997289541. expected length is 7398 and > actual length is 9042 > 2020-04-27 07:27:30.630 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1mv_Lucene50_0.doc did not match. expected > checksum is 2165407577 and actual is checksum 2563543943. expected length > is 110 and actual length is 110 > 2020-04-27 07:27:30.632 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1mv_Lucene50_0.tim did not match. expected > checksum is 1427836076 and actual is checksum 225462388. expected length is > 11162 and actual length is 13623 > 2020-04-27 07:27:30.634 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1mv_Lucene50_0.tip did not match. expected > checksum is 2998276982 and actual is checksum 1966428087. expected length > is 296 and actual length is 353 > 2020-04-27 07:27:30.637 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1mv_Lucene70_0.dvm did not match. expected > checksum is 2717609594 and actual is checksum 673086945. expected length is > 315491 and actual length is 312531 > 2020-04-27 07:27:30.640 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1mv.si did not match. expected checksum is > 2885031416 and actual is checksum 762513018. expected length is 508 and > actual length is 508 > 2020-04-27 07:27:30.643 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1mv_Lucene70_0.dvd did not match. expected > checksum is 602612626 and actual is checksum 2826680041. expected length is > 1292092 and actual length is 1655171 > 2020-04-27 07:27:30.669 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1mx_Lucene50_0.tip did not match. expected > checksum is 401385741 and actual is checksum 3373755803. expected length is > 621 and actual length is 715 > 2020-04-27 07:27:30.671 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1mx.si did not match. expected checksum is > 3708405765 and actual is checksum 1057460545. expected length is 508 and > actual length is 508 > 2020-04-27 07:27:30.674 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1mx_Lucene70_0.dvm did not match. expected > checksum is 2755907323 and actual is checksum 1931314059. expected length > is 327125 and actual length is 318737 > 2020-04-27 07:27:30.677 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1mx.fnm did not match. expected checksum is > 77505520 and actual is checksum 2602905677. expected length is 107240 and > actual length is 107460 > 2020-04-27 07:27:30.680 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1mx_Lucene50_0.doc did not match. expected > checksum is 4204960097 and actual is checksum 2276943505. expected length > is 110 and actual length is 110 > 2020-04-27 07:27:30.682 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1mx.fdt did not match. expected checksum is > 3414743450 and actual is checksum 3890346633. expected length is 18415 and > actual length is 21922 > 2020-04-27 07:27:30.685 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1mx_Lucene50_0.tim did not match. expected > checksum is 2879829129 and actual is checksum 790212338. expected length is > 27860 and actual length is 33235 > 2020-04-27 07:27:30.687 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1mx.fdx did not match. expected checksum is > 3356980932 and actual is checksum 1544608917. expected length is 108 and > actual length is 112 > 2020-04-27 07:27:30.690 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1mx_Lucene70_0.dvd did not match. expected > checksum is 1979425912 and actual is checksum 584894070. expected length is > 3726110 and actual length is 4689653 > 2020-04-27 07:27:30.700 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1mz.si did not match. expected checksum is > 3754169108 and actual is checksum 527001228. expected length is 508 and > actual length is 508 > 2020-04-27 07:27:30.702 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1mz_Lucene70_0.dvd did not match. expected > checksum is 3981710860 and actual is checksum 1749011618. expected length > is 1938123 and actual length is 33694 > 2020-04-27 07:27:30.710 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1mz.fdx did not match. expected checksum is > 1515379146 and actual is checksum 3212722427. expected length is 98 and > actual length is 84 > 2020-04-27 07:27:30.712 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1mz.fnm did not match. expected checksum is > 1819400085 and actual is checksum 3372088644. expected length is 107020 and > actual length is 103887 > 2020-04-27 07:27:30.715 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1mz_Lucene50_0.doc did not match. expected > checksum is 1401562358 and actual is checksum 2224139078. expected length > is 110 and actual length is 110 > 2020-04-27 07:27:30.718 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1mz_Lucene50_0.tim did not match. expected > checksum is 3386701421 and actual is checksum 1434467863. expected length > is 15944 and actual length is 685 > 2020-04-27 07:27:30.720 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1mz.fdt did not match. expected checksum is > 4259543754 and actual is checksum 3203989793. expected length is 10616 and > actual length is 439 > 2020-04-27 07:27:30.722 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1mz_Lucene50_0.tip did not match. expected > checksum is 725882371 and actual is checksum 3650374401. expected length is > 383 and actual length is 114 > 2020-04-27 07:27:30.725 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1mz_Lucene70_0.dvm did not match. expected > checksum is 3717717087 and actual is checksum 3712473868. expected length > is 303729 and actual length is 117582 > 2020-04-27 07:27:30.728 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n0.fdx did not match. expected checksum is > 3506718582 and actual is checksum 342046803. expected length is 93 and > actual length is 104 > 2020-04-27 07:27:30.731 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n0.si did not match. expected checksum is > 2920971663 and actual is checksum 2142622028. expected length is 508 and > actual length is 508 > 2020-04-27 07:27:30.733 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n0_Lucene50_0.doc did not match. expected > checksum is 723455769 and actual is checksum 2731538349. expected length is > 110 and actual length is 110 > 2020-04-27 07:27:30.735 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n0_Lucene50_0.tim did not match. expected > checksum is 2624342764 and actual is checksum 1283943229. expected length > is 11424 and actual length is 26688 > 2020-04-27 07:27:30.738 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n0_Lucene50_0.tip did not match. expected > checksum is 978416968 and actual is checksum 1807767534. expected length is > 330 and actual length is 597 > 2020-04-27 07:27:30.740 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n0_Lucene70_0.dvm did not match. expected > checksum is 3599774596 and actual is checksum 2764190632. expected length > is 318017 and actual length is 311617 > 2020-04-27 07:27:30.743 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n0_Lucene70_0.dvd did not match. expected > checksum is 3141314417 and actual is checksum 3847788889. expected length > is 1258075 and actual length is 3539994 > 2020-04-27 07:27:30.749 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n0.fnm did not match. expected checksum is > 1404317797 and actual is checksum 2471843701. expected length is 105295 and > actual length is 107020 > 2020-04-27 07:27:30.752 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n0.fdt did not match. expected checksum is > 3412288246 and actual is checksum 922481525. expected length is 7580 and > actual length is 17733 > 2020-04-27 07:27:30.754 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n1_Lucene50_0.doc did not match. expected > checksum is 2675540020 and actual is checksum 1452209674. expected length > is 110 and actual length is 110 > 2020-04-27 07:27:30.757 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n1_Lucene50_0.tim did not match. expected > checksum is 665797990 and actual is checksum 2134373365. expected length is > 10715 and actual length is 3418 > 2020-04-27 07:27:30.759 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n1.fdt did not match. expected checksum is > 2006427884 and actual is checksum 527297847. expected length is 7178 and > actual length is 2326 > 2020-04-27 07:27:30.761 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n1.fnm did not match. expected checksum is > 200552893 and actual is checksum 3290522141. expected length is 106580 and > actual length is 105206 > 2020-04-27 07:27:30.764 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n1_Lucene50_0.tip did not match. expected > checksum is 1511033560 and actual is checksum 3689083691. expected length > is 278 and actual length is 154 > 2020-04-27 07:27:30.767 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n1_Lucene70_0.dvm did not match. expected > checksum is 1435050119 and actual is checksum 6864243. expected length is > 323217 and actual length is 338063 > 2020-04-27 07:27:30.770 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n1.si did not match. expected checksum is > 1107038327 and actual is checksum 3068870234. expected length is 508 and > actual length is 508 > 2020-04-27 07:27:30.772 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n1.fdx did not match. expected checksum is > 2853806777 and actual is checksum 459866651. expected length is 93 and > actual length is 87 > 2020-04-27 07:27:30.775 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n1_Lucene70_0.dvd did not match. expected > checksum is 1356537478 and actual is checksum 2594306289. expected length > is 1179215 and actual length is 249458 > 2020-04-27 07:27:30.780 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n2.fnm did not match. expected checksum is > 1785795479 and actual is checksum 65014613. expected length is 106580 and > actual length is 106800 > 2020-04-27 07:27:30.783 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n2_Lucene50_0.tip did not match. expected > checksum is 1824666616 and actual is checksum 2857527783. expected length > is 507 and actual length is 637 > 2020-04-27 07:27:30.785 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n2.fdt did not match. expected checksum is > 1384446542 and actual is checksum 3981590407. expected length is 13686 and > actual length is 18516 > 2020-04-27 07:27:30.788 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n2_Lucene70_0.dvm did not match. expected > checksum is 3380549662 and actual is checksum 2002436369. expected length > is 317777 and actual length is 319213 > 2020-04-27 07:27:30.791 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n2_Lucene50_0.doc did not match. expected > checksum is 1594954666 and actual is checksum 1892723425. expected length > is 110 and actual length is 110 > 2020-04-27 07:27:30.794 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n2_Lucene50_0.tim did not match. expected > checksum is 305271241 and actual is checksum 854278263. expected length is > 20522 and actual length is 27836 > 2020-04-27 07:27:30.796 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n2.si did not match. expected checksum is > 2220527769 and actual is checksum 2016379681. expected length is 508 and > actual length is 508 > 2020-04-27 07:27:30.799 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n2.fdx did not match. expected checksum is > 1648054590 and actual is checksum 137025071. expected length is 100 and > actual length is 105 > 2020-04-27 07:27:30.801 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n2_Lucene70_0.dvd did not match. expected > checksum is 262113808 and actual is checksum 327562225. expected length is > 2731808 and actual length is 3909012 > 2020-04-27 07:27:30.809 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n3_Lucene70_0.dvd did not match. expected > checksum is 2157667750 and actual is checksum 3043505208. expected length > is 36906 and actual length is 163443 > 2020-04-27 07:27:30.812 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n3.fdt did not match. expected checksum is > 3456088243 and actual is checksum 1715362430. expected length is 453 and > actual length is 1636 > 2020-04-27 07:27:30.814 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n3.fnm did not match. expected checksum is > 4028738113 and actual is checksum 4245480574. expected length is 104876 and > actual length is 105515 > 2020-04-27 07:27:30.817 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n3_Lucene50_0.doc did not match. expected > checksum is 3120632072 and actual is checksum 1945528118. expected length > is 110 and actual length is 110 > 2020-04-27 07:27:30.819 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n3_Lucene50_0.tip did not match. expected > checksum is 3303166970 and actual is checksum 1740028553. expected length > is 114 and actual length is 137 > 2020-04-27 07:27:30.822 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n3_Lucene50_0.tim did not match. expected > checksum is 126700369 and actual is checksum 162509958. expected length is > 700 and actual length is 2398 > 2020-04-27 07:27:30.825 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n3.fdx did not match. expected checksum is > 3157051968 and actual is checksum 137679212. expected length is 84 and > actual length is 86 > 2020-04-27 07:27:30.827 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n3_Lucene70_0.dvm did not match. expected > checksum is 3969090178 and actual is checksum 2364978843. expected length > is 127576 and actual length is 266581 > 2020-04-27 07:27:30.830 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n3.si did not match. expected checksum is > 3730892058 and actual is checksum 2734980133. expected length is 508 and > actual length is 508 > 2020-04-27 07:27:30.832 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n4.fnm did not match. expected checksum is > 3259266514 and actual is checksum 2051283039. expected length is 104855 and > actual length is 106819 > 2020-04-27 07:27:30.835 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n4_Lucene50_0.tim did not match. expected > checksum is 380153843 and actual is checksum 580580220. expected length is > 1822 and actual length is 1737 > 2020-04-27 07:27:30.838 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n4_Lucene50_0.doc did not match. expected > checksum is 4094454161 and actual is checksum 421930789. expected length is > 110 and actual length is 110 > 2020-04-27 07:27:30.840 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n4.fdt did not match. expected checksum is > 911575783 and actual is checksum 2090965125. expected length is 1244 and > actual length is 1201 > 2020-04-27 07:27:30.842 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n4_Lucene50_0.tip did not match. expected > checksum is 1279063865 and actual is checksum 765869778. expected length is > 128 and actual length is 130 > 2020-04-27 07:27:30.845 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n4_Lucene70_0.dvm did not match. expected > checksum is 1689040450 and actual is checksum 627554920. expected length is > 223473 and actual length is 230157 > 2020-04-27 07:27:30.848 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n4.fdx did not match. expected checksum is > 1278110969 and actual is checksum 1444534251. expected length is 86 and > actual length is 86 > 2020-04-27 07:27:30.850 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n4_Lucene70_0.dvd did not match. expected > checksum is 1992192696 and actual is checksum 3455180678. expected length > is 119844 and actual length is 117860 > 2020-04-27 07:27:30.853 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n4.si did not match. expected checksum is > 2776236859 and actual is checksum 2572132550. expected length is 508 and > actual length is 508 > 2020-04-27 07:27:35.866 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1lu.si did not match. expected checksum is > 2573424453 and actual is checksum 3612352961. expected length is 546 and > actual length is 437 > 2020-04-27 07:27:35.868 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1kr.cfe did not match. expected checksum is > 2592320603 and actual is checksum 1374033039. expected length is 289 and > actual length is 289 > 2020-04-27 07:27:35.872 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1kr.si did not match. expected checksum is > 2911257988 and actual is checksum 2011202433. expected length is 437 and > actual length is 437 > 2020-04-27 07:27:35.874 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1kr.cfs did not match. expected checksum is > 1304802713 and actual is checksum 454030092. expected length is 7598425 and > actual length is 49265169 > 2020-04-27 07:27:37.234 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1mo.si did not match. expected checksum is > 1422458436 and actual is checksum 703149874. expected length is 437 and > actual length is 437 > 2020-04-27 07:27:38.117 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1mo.cfs did not match. expected checksum is > 1892273984 and actual is checksum 2087644984. expected length is 50710602 > and actual length is 10402264 > 2020-04-27 07:27:38.532 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1mo.cfe did not match. expected checksum is > 3864394245 and actual is checksum 1846889307. expected length is 289 and > actual length is 289 > 2020-04-27 07:27:38.575 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1m9.fdx did not match. expected checksum is > 660375856 and actual is checksum 571052523. expected length is 114 and > actual length is 84 > 2020-04-27 07:27:38.590 WARN > (recoveryExecutor-2639-thread-1-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard2_replica_n4 s:shard2 c:mycollection r:core_node7) > [c:mycollection s:shard2 r:core_node7 x:mycollection_shard2_replica_n4] > o.a.s.u.UpdateLog Log replay finished. recoveryInfo=RecoveryInfo{adds=31744 > deletes=0 deleteByQuery=0 errors=0 positionOfStart=2720536} > 2020-04-27 07:27:39.259 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1m9_Lucene70_0.dvm did not match. expected > checksum is 421440641 and actual is checksum 2075160524. expected length is > 337361 and actual length is 141820 > 2020-04-27 07:27:44.881 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1m9_Lucene70_0.dvd did not match. expected > checksum is 47189924 and actual is checksum 3640637009. expected length is > 5235629 and actual length is 48679 > 2020-04-27 07:27:45.335 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1m9_Lucene50_0.tip did not match. expected > checksum is 2305624674 and actual is checksum 704932512. expected length is > 693 and actual length is 115 > 2020-04-27 07:27:45.939 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1m9.fdt did not match. expected checksum is > 1157089232 and actual is checksum 2388254324. expected length is 23435 and > actual length is 673 > 2020-04-27 07:27:45.941 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1m9.si did not match. expected checksum is > 2811885732 and actual is checksum 2854129172. expected length is 508 and > actual length is 508 > 2020-04-27 07:27:45.943 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1m9_Lucene50_0.doc did not match. expected > checksum is 1925626873 and actual is checksum 1921615605. expected length > is 110 and actual length is 110 > 2020-04-27 07:27:45.945 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1m9_Lucene50_0.tim did not match. expected > checksum is 3627283142 and actual is checksum 3796132842. expected length > is 34596 and actual length is 958 > 2020-04-27 07:27:45.947 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1m9.fnm did not match. expected checksum is > 1291024098 and actual is checksum 2700484857. expected length is 107020 and > actual length is 104209 > 2020-04-27 07:27:45.949 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1mm_Lucene50_0.tip did not match. expected > checksum is 2785033346 and actual is checksum 1911387906. expected length > is 816 and actual length is 760 > 2020-04-27 07:27:45.951 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1mm_Lucene70_0.dvd did not match. expected > checksum is 942596692 and actual is checksum 3039336477. expected length is > 5557044 and actual length is 4847564 > 2020-04-27 07:27:45.974 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1mm.fdx did not match. expected checksum is > 143536399 and actual is checksum 2962535892. expected length is 116 and > actual length is 111 > 2020-04-27 07:27:45.976 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1mm_Lucene70_0.dvm did not match. expected > checksum is 3884436961 and actual is checksum 1345841996. expected length > is 339801 and actual length is 336809 > 2020-04-27 07:27:45.980 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1mm.fdt did not match. expected checksum is > 3902419774 and actual is checksum 3791519088. expected length is 23306 and > actual length is 21047 > 2020-04-27 07:27:45.981 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1mm.si did not match. expected checksum is > 45470103 and actual is checksum 235515609. expected length is 508 and > actual length is 508 > 2020-04-27 07:27:45.983 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1mm_Lucene50_0.doc did not match. expected > checksum is 408594907 and actual is checksum 2650340971. expected length is > 110 and actual length is 110 > 2020-04-27 07:27:45.985 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1mm_Lucene50_0.tim did not match. expected > checksum is 761408352 and actual is checksum 685575313. expected length is > 35256 and actual length is 31644 > 2020-04-27 07:27:45.987 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1mm.fnm did not match. expected checksum is > 811204953 and actual is checksum 1244308773. expected length is 107460 and > actual length is 107460 > 2020-04-27 07:27:45.989 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1mr.fnm did not match. expected checksum is > 3813441918 and actual is checksum 2133371346. expected length is 107460 and > actual length is 106360 > 2020-04-27 07:27:45.991 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1mr.fdt did not match. expected checksum is > 4263416408 and actual is checksum 3722461284. expected length is 16698 and > actual length is 21392 > 2020-04-27 07:27:45.993 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1mr_Lucene50_0.doc did not match. expected > checksum is 632422377 and actual is checksum 501799146. expected length is > 110 and actual length is 110 > 2020-04-27 07:27:45.994 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1mr_Lucene50_0.tim did not match. expected > checksum is 3170165892 and actual is checksum 4152937488. expected length > is 25090 and actual length is 32328 > 2020-04-27 07:27:46.000 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1mr_Lucene50_0.tip did not match. expected > checksum is 2679120655 and actual is checksum 3243853760. expected length > is 553 and actual length is 726 > 2020-04-27 07:27:46.001 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1mr_Lucene70_0.dvm did not match. expected > checksum is 2991034554 and actual is checksum 1980969905. expected length > is 321785 and actual length is 322117 > 2020-04-27 07:27:46.004 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _
