FWIW I've had some luck with strategy 3 (increase zk timeout) when you overwhelm the connection to zk or the disk on zk.
Is zk on the same boxes as solr? On Tue, Apr 28, 2020 at 10:15 PM Sethuraman, Ganesh <sethuram...@dnb.com.invalid> wrote: > > Hi > > We are using SolrCloud 7.2.1 with 3 node Zookeeper ensemble. We have 92 > collection each on avg. having 8 shards and 2 replica with 2 EC2 nodes, with > JVM size of 18GB (G1 GC). We need your help with the Issue we faced today: > The issue is SolrCloud server went into a degraded collections (for few > collections) when the Solr backup and the Solr batch CSV update load happened > at the same time as backup. The CSV data load was about ~5 GB per > shard/replica. We think this happened after zkClient disconnect happened as > noted below. We had to restart Solr to bring it back to normal. > > > 1. Is it not suggested to run backup and Solr batch CSV update large load > at the same time? > 2. In the past we have seen two CSV batch update load in parallel causes > issues, is this also not suggested (this issue is not related to that)? > 3. Do you think we should increase Zookeeper timeout? > 4. How do we know if we need to up the JVM Max memory, and by how much? > 5. We also see that once the Solr goes into degraded collection and > recovery failed, it NEVER get back to normal, even after when there is no > load. Is this a bug? > > The GC information and Solr Log below > > https://gceasy.io/my-gc-report.jsp?p=c2hhcmVkLzIwMjAvMDQvMjkvLS0wMl9zb2xyX2djLmxvZy56aXAtLTEtNDAtMzE=&channel=WEB > > > 2020-04-27 07:34:07.322 WARN > (zkConnectionManagerCallback-6-thread-1-processing-n:mysolrsever.com:6010_solr-SendThread(zoo-prd-n1:2181)) > [ ] o.a.z.ClientCnxn Client session timed out, have not heard from server > in 10775ms for sessionid 0x171a6fb51310008 > .... > 2020-04-27 07:34:07.426 WARN > (zkConnectionManagerCallback-6-thread-1-processing-n:mysolrsever.com:6010_solr-EventThread) > [ ] o.a.s.c.c.ConnectionManager zkClient has disconnected > > > > > SOLR Log Below (Curtailed WARN log) > -------------------- > 2020-04-27 07:26:45.402 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of 587 > bytes) > java.io.EOFException > at > org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168) > at > org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160) > at > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579) > at > org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545) > at > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526) > at > org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008) > at > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566) > at > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345) > at > org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420) > at > org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225) > at > org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626) > at > org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308) > at > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292) > at > com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > 2020-04-27 07:26:45.405 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of 587 > bytes) > java.io.EOFException > at > org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168) > at > org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160) > at > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579) > at > org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545) > at > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526) > at > org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008) > at > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566) > at > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345) > at > org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420) > at > org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225) > at > org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626) > at > org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308) > at > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292) > at > com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > 2020-04-27 07:26:45.406 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of 587 > bytes) > java.io.EOFException > at > org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168) > at > org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160) > at > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579) > at > org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545) > at > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526) > at > org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008) > at > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566) > at > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345) > at > org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420) > at > org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225) > at > org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626) > at > org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308) > at > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292) > at > com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > 2020-04-27 07:26:45.408 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of 587 > bytes) > java.io.EOFException > at > org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168) > at > org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160) > at > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579) > at > org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545) > at > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526) > at > org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008) > at > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566) > at > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345) > at > org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420) > at > org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225) > at > org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626) > at > org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308) > at > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292) > at > com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > 2020-04-27 07:26:45.410 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of 587 > bytes) > java.io.EOFException > at > org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168) > at > org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160) > at > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579) > at > org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545) > at > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526) > at > org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008) > at > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566) > at > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345) > at > org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420) > at > org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225) > at > org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626) > at > org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308) > at > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292) > at > com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > 2020-04-27 07:26:45.412 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of 587 > bytes) > java.io.EOFException > at > org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168) > at > org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160) > at > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579) > at > org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545) > at > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526) > at > org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008) > at > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566) > at > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345) > at > org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420) > at > org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225) > at > org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626) > at > org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308) > at > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292) > at > com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > 2020-04-27 07:26:45.412 ERROR > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher Error fetching file, doing one > retry...:org.apache.solr.common.SolrException: Unable to download _1kr_r.liv > completely. Downloaded 0!=587 > at > org.apache.solr.handler.IndexFetcher$FileFetcher.cleanup(IndexFetcher.java:1673) > at > org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1556) > at > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526) > at > org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008) > at > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566) > at > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345) > at > org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420) > at > org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225) > at > org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626) > at > org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308) > at > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292) > at > com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > > 2020-04-27 07:26:45.414 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of 587 > bytes) > java.io.EOFException > at > org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168) > at > org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160) > at > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579) > at > org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545) > at > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1531) > at > org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008) > at > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566) > at > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345) > at > org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420) > at > org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225) > at > org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626) > at > org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308) > at > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292) > at > com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > 2020-04-27 07:26:45.414 ERROR > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher Error deleting file: _1kr_r.liv > java.nio.file.NoSuchFileException: > /data/dnb/cloud/node1/solr/mycollection_shard13_replica_n48/data/index.20200427072258517/_1kr_r.liv > at > sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) > at > sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) > at > sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) > at > sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:244) > at > sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103) > at java.nio.file.Files.delete(Files.java:1126) > at > org.apache.lucene.store.FSDirectory.privateDeleteFile(FSDirectory.java:373) > at > org.apache.lucene.store.FSDirectory.deleteFile(FSDirectory.java:335) > at > org.apache.lucene.store.NRTCachingDirectory.deleteFile(NRTCachingDirectory.java:119) > at > org.apache.solr.handler.IndexFetcher$DirectoryFile.delete(IndexFetcher.java:1757) > at > org.apache.solr.handler.IndexFetcher$FileFetcher.cleanup(IndexFetcher.java:1667) > at > org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1556) > at > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1531) > at > org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008) > at > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566) > at > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345) > at > org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420) > at > org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225) > at > org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626) > at > org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308) > at > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292) > at > com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > 2020-04-27 07:26:45.449 WARN > (recoveryExecutor-4-thread-699-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard2_replica_n4 s:shard2 c:mycollection r:core_node7) > [c:mycollection s:shard2 r:core_node7 x:mycollection_shard2_replica_n4] > o.a.s.h.IndexFetcher File _1kq.cfs did not match. expected checksum is > 3934327071 and actual is checksum 3095753427. expected length is 33235371 and > actual length is 54180642 > 2020-04-27 07:26:45.449 WARN > (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) > [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] > o.a.s.h.IndexFetcher File _1em.fnm did not match. expected checksum is > 138861107 and actual is checksum 188997082. expected length is 107460 and > actual length is 107460 > 2020-04-27 07:26:45.457 WARN > (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) > [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] > o.a.s.h.IndexFetcher File _1mt_Lucene50_0.tim did not match. expected > checksum is 2947323803 and actual is checksum 979431341. expected length is > 32176 and actual length is 26089 > 2020-04-27 07:26:45.521 WARN > (recoveryExecutor-4-thread-648-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard6_replica_n20 s:shard6 c:mycollection r:core_node23) > [c:mycollection s:shard6 r:core_node23 x:mycollection_shard6_replica_n20] > o.a.s.h.IndexFetcher File _1gk_Lucene50_0.doc did not match. expected > checksum is 2087731087 and actual is checksum 3540174119. expected length is > 110 and actual length is 110 > 2020-04-27 07:26:45.521 WARN > (recoveryExecutor-4-thread-657-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard1_replica_n1 s:shard1 c:mycollection r:core_node3) > [c:mycollection s:shard1 r:core_node3 x:mycollection_shard1_replica_n1] > o.a.s.h.IndexFetcher File _1mo.cfs did not match. expected checksum is > 777834759 and actual is checksum 289576304. expected length is 59980991 and > actual length is 6236424 > 2020-04-27 07:26:45.521 WARN > (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) > [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] > o.a.s.h.IndexFetcher File _1mt.fdx did not match. expected checksum is > 1321210176 and actual is checksum 224636564. expected length is 114 and > actual length is 106 > 2020-04-27 07:26:45.521 WARN > (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) > [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] > o.a.s.h.IndexFetcher File _1em.fdt did not match. expected checksum is > 3790133021 and actual is checksum 2516047051. expected length is 1477755 and > actual length is 1477850 > 2020-04-27 07:26:45.578 WARN > (recoveryExecutor-4-thread-648-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard6_replica_n20 s:shard6 c:mycollection r:core_node23) > [c:mycollection s:shard6 r:core_node23 x:mycollection_shard6_replica_n20] > o.a.s.h.IndexFetcher File _1gk_Lucene50_0.tim did not match. expected > checksum is 3690299979 and actual is checksum 411724865. expected length is > 4392270 and actual length is 4396498 > 2020-04-27 07:26:45.588 WARN > (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) > [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] > o.a.s.h.IndexFetcher File _1em_Lucene50_0.doc did not match. expected > checksum is 937224718 and actual is checksum 1176441880. expected length is > 110 and actual length is 110 > 2020-04-27 07:26:45.733 WARN > (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) > [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] > o.a.s.h.IndexFetcher File _1em_Lucene50_0.tim did not match. expected > checksum is 1429458386 and actual is checksum 4155623862. expected length is > 2213909 and actual length is 2213939 > 2020-04-27 07:26:45.754 ERROR > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.ReplicationHandler Index fetch failed > :org.apache.solr.common.SolrException: Unable to download _1kr_r.liv > completely. Downloaded 0!=587 > at > org.apache.solr.handler.IndexFetcher$FileFetcher.cleanup(IndexFetcher.java:1673) > at > org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1556) > at > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1531) > at > org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008) > at > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566) > at > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345) > at > org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420) > at > org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225) > at > org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626) > at > org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308) > at > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292) > at > com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > > 2020-04-27 07:26:45.755 ERROR > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.c.RecoveryStrategy Error while trying to > recover:org.apache.solr.common.SolrException: Replication for recovery failed. > at > org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:228) > at > org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626) > at > org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308) > at > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292) > at > com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > > 2020-04-27 07:26:45.755 ERROR > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.c.RecoveryStrategy Recovery failed - trying again... (1) > 2020-04-27 07:26:45.780 WARN > (recoveryExecutor-4-thread-648-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard6_replica_n20 s:shard6 c:mycollection r:core_node23) > [c:mycollection s:shard6 r:core_node23 x:mycollection_shard6_replica_n20] > o.a.s.h.IndexFetcher File _1gk_Lucene70_0.dvm did not match. expected > checksum is 1474431645 and actual is checksum 1271168044. expected length is > 275570 and actual length is 274506 > 2020-04-27 07:26:45.834 WARN > (recoveryExecutor-4-thread-648-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard6_replica_n20 s:shard6 c:mycollection r:core_node23) > [c:mycollection s:shard6 r:core_node23 x:mycollection_shard6_replica_n20] > o.a.s.h.IndexFetcher File _1gk.fdx did not match. expected checksum is > 4129522944 and actual is checksum 4080519415. expected length is 6626 and > actual length is 6626 > 2020-04-27 07:26:45.843 WARN > (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) > [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] > o.a.s.h.IndexFetcher File _1em.si did not match. expected checksum is > 2493366746 and actual is checksum 31587394. expected length is 546 and actual > length is 546 > 2020-04-27 07:26:45.882 WARN > (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) > [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] > o.a.s.h.IndexFetcher File _1mv.fdx did not match. expected checksum is > 1392813257 and actual is checksum 3455988143. expected length is 103 and > actual length is 111 > 2020-04-27 07:26:45.882 WARN > (recoveryExecutor-4-thread-648-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard6_replica_n20 s:shard6 c:mycollection r:core_node23) > [c:mycollection s:shard6 r:core_node23 x:mycollection_shard6_replica_n20] > o.a.s.h.IndexFetcher File _1gk_Lucene50_0.tip did not match. expected > checksum is 2301226889 and actual is checksum 29696594. expected length is > 86459 and actual length is 85786 > 2020-04-27 07:26:45.882 WARN > (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) > [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] > o.a.s.h.IndexFetcher File _1em_Lucene70_0.dvm did not match. expected > checksum is 112676849 and actual is checksum 354222195. expected length is > 286474 and actual length is 286546 > 2020-04-27 07:26:45.911 WARN > (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) > [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] > o.a.s.h.IndexFetcher File _1mv.fnm did not match. expected checksum is > 2203892485 and actual is checksum 1165086749. expected length is 107460 and > actual length is 106580 > 2020-04-27 07:26:45.911 WARN > (recoveryExecutor-4-thread-648-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard6_replica_n20 s:shard6 c:mycollection r:core_node23) > [c:mycollection s:shard6 r:core_node23 x:mycollection_shard6_replica_n20] > o.a.s.h.IndexFetcher File _1gk_2z.liv did not match. expected checksum is > 3175370827 and actual is checksum 520836483. expected length is 51396 and > actual length is 51404 > 2020-04-27 07:26:45.912 WARN > (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) > [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] > o.a.s.h.IndexFetcher File _1em.fdx did not match. expected checksum is > 967260408 and actual is checksum 969726862. expected length is 3180 and > actual length is 3180 > 2020-04-27 07:26:45.914 WARN > (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) > [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] > o.a.s.h.IndexFetcher File _1mv.fdt did not match. expected checksum is > 719562508 and actual is checksum 956062095. expected length is 16717 and > actual length is 21454 > 2020-04-27 07:26:45.915 WARN > (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) > [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] > o.a.s.h.IndexFetcher File _1em_Lucene50_0.tip did not match. expected > checksum is 3833337361 and actual is checksum 493358852. expected length is > 43996 and actual length is 44026 > 2020-04-27 07:26:45.916 WARN > (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) > [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] > o.a.s.h.IndexFetcher File _1mv_Lucene50_0.doc did not match. expected > checksum is 1533628328 and actual is checksum 3735455395. expected length is > 110 and actual length is 110 > 2020-04-27 07:26:45.917 WARN > (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) > [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] > o.a.s.h.IndexFetcher File _1em_Lucene70_0.dvd did not match. expected > checksum is 75152145 and actual is checksum 1665090946. expected length is > 497241188 and actual length is 497254814 > 2020-04-27 07:26:45.918 WARN > (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) > [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] > o.a.s.h.IndexFetcher File _1mv_Lucene50_0.tim did not match. expected > checksum is 1349491281 and actual is checksum 3808036883. expected length is > 25153 and actual length is 32322 > 2020-04-27 07:26:45.920 WARN > (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) > [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] > o.a.s.h.IndexFetcher File _1mv_Lucene50_0.tip did not match. expected > checksum is 3820733560 and actual is checksum 2819887160. expected length is > 571 and actual length is 725 > 2020-04-27 07:26:46.034 WARN > (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) > [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] > o.a.s.h.IndexFetcher File _1mv_Lucene70_0.dvm did not match. expected > checksum is 1547030244 and actual is checksum 1122196351. expected length is > 324313 and actual length is 323553 > 2020-04-27 07:26:46.170 WARN > (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) > [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] > o.a.s.h.IndexFetcher File _1mv.si did not match. expected checksum is > 751850967 and actual is checksum 1026527313. expected length is 508 and > actual length is 508 > and actual length is 319133 > 2020-04-27 07:27:30.567 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1mr_Lucene70_0.dvd did not match. expected > checksum is 3435029878 and actual is checksum 1202327807. expected length is > 3337719 and actual length is 3375106 > 2020-04-27 07:27:30.578 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1mr.fdx did not match. expected checksum is > 2264443519 and actual is checksum 2299249501. expected length is 104 and > actual length is 102 > 2020-04-27 07:27:30.580 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1mr.si did not match. expected checksum is > 1184027188 and actual is checksum 2810465617. expected length is 508 and > actual length is 508 > 2020-04-27 07:27:30.622 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1mv.fdx did not match. expected checksum is > 2246788287 and actual is checksum 2682862227. expected length is 92 and > actual length is 96 > 2020-04-27 07:27:30.625 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1mv.fnm did not match. expected checksum is > 3227520043 and actual is checksum 2778425164. expected length is 107137 and > actual length is 107137 > 2020-04-27 07:27:30.627 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1mv.fdt did not match. expected checksum is > 1964200539 and actual is checksum 3997289541. expected length is 7398 and > actual length is 9042 > 2020-04-27 07:27:30.630 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1mv_Lucene50_0.doc did not match. expected > checksum is 2165407577 and actual is checksum 2563543943. expected length is > 110 and actual length is 110 > 2020-04-27 07:27:30.632 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1mv_Lucene50_0.tim did not match. expected > checksum is 1427836076 and actual is checksum 225462388. expected length is > 11162 and actual length is 13623 > 2020-04-27 07:27:30.634 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1mv_Lucene50_0.tip did not match. expected > checksum is 2998276982 and actual is checksum 1966428087. expected length is > 296 and actual length is 353 > 2020-04-27 07:27:30.637 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1mv_Lucene70_0.dvm did not match. expected > checksum is 2717609594 and actual is checksum 673086945. expected length is > 315491 and actual length is 312531 > 2020-04-27 07:27:30.640 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1mv.si did not match. expected checksum is > 2885031416 and actual is checksum 762513018. expected length is 508 and > actual length is 508 > 2020-04-27 07:27:30.643 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1mv_Lucene70_0.dvd did not match. expected > checksum is 602612626 and actual is checksum 2826680041. expected length is > 1292092 and actual length is 1655171 > 2020-04-27 07:27:30.669 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1mx_Lucene50_0.tip did not match. expected > checksum is 401385741 and actual is checksum 3373755803. expected length is > 621 and actual length is 715 > 2020-04-27 07:27:30.671 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1mx.si did not match. expected checksum is > 3708405765 and actual is checksum 1057460545. expected length is 508 and > actual length is 508 > 2020-04-27 07:27:30.674 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1mx_Lucene70_0.dvm did not match. expected > checksum is 2755907323 and actual is checksum 1931314059. expected length is > 327125 and actual length is 318737 > 2020-04-27 07:27:30.677 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1mx.fnm did not match. expected checksum is > 77505520 and actual is checksum 2602905677. expected length is 107240 and > actual length is 107460 > 2020-04-27 07:27:30.680 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1mx_Lucene50_0.doc did not match. expected > checksum is 4204960097 and actual is checksum 2276943505. expected length is > 110 and actual length is 110 > 2020-04-27 07:27:30.682 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1mx.fdt did not match. expected checksum is > 3414743450 and actual is checksum 3890346633. expected length is 18415 and > actual length is 21922 > 2020-04-27 07:27:30.685 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1mx_Lucene50_0.tim did not match. expected > checksum is 2879829129 and actual is checksum 790212338. expected length is > 27860 and actual length is 33235 > 2020-04-27 07:27:30.687 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1mx.fdx did not match. expected checksum is > 3356980932 and actual is checksum 1544608917. expected length is 108 and > actual length is 112 > 2020-04-27 07:27:30.690 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1mx_Lucene70_0.dvd did not match. expected > checksum is 1979425912 and actual is checksum 584894070. expected length is > 3726110 and actual length is 4689653 > 2020-04-27 07:27:30.700 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1mz.si did not match. expected checksum is > 3754169108 and actual is checksum 527001228. expected length is 508 and > actual length is 508 > 2020-04-27 07:27:30.702 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1mz_Lucene70_0.dvd did not match. expected > checksum is 3981710860 and actual is checksum 1749011618. expected length is > 1938123 and actual length is 33694 > 2020-04-27 07:27:30.710 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1mz.fdx did not match. expected checksum is > 1515379146 and actual is checksum 3212722427. expected length is 98 and > actual length is 84 > 2020-04-27 07:27:30.712 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1mz.fnm did not match. expected checksum is > 1819400085 and actual is checksum 3372088644. expected length is 107020 and > actual length is 103887 > 2020-04-27 07:27:30.715 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1mz_Lucene50_0.doc did not match. expected > checksum is 1401562358 and actual is checksum 2224139078. expected length is > 110 and actual length is 110 > 2020-04-27 07:27:30.718 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1mz_Lucene50_0.tim did not match. expected > checksum is 3386701421 and actual is checksum 1434467863. expected length is > 15944 and actual length is 685 > 2020-04-27 07:27:30.720 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1mz.fdt did not match. expected checksum is > 4259543754 and actual is checksum 3203989793. expected length is 10616 and > actual length is 439 > 2020-04-27 07:27:30.722 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1mz_Lucene50_0.tip did not match. expected > checksum is 725882371 and actual is checksum 3650374401. expected length is > 383 and actual length is 114 > 2020-04-27 07:27:30.725 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1mz_Lucene70_0.dvm did not match. expected > checksum is 3717717087 and actual is checksum 3712473868. expected length is > 303729 and actual length is 117582 > 2020-04-27 07:27:30.728 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n0.fdx did not match. expected checksum is > 3506718582 and actual is checksum 342046803. expected length is 93 and actual > length is 104 > 2020-04-27 07:27:30.731 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n0.si did not match. expected checksum is > 2920971663 and actual is checksum 2142622028. expected length is 508 and > actual length is 508 > 2020-04-27 07:27:30.733 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n0_Lucene50_0.doc did not match. expected > checksum is 723455769 and actual is checksum 2731538349. expected length is > 110 and actual length is 110 > 2020-04-27 07:27:30.735 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n0_Lucene50_0.tim did not match. expected > checksum is 2624342764 and actual is checksum 1283943229. expected length is > 11424 and actual length is 26688 > 2020-04-27 07:27:30.738 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n0_Lucene50_0.tip did not match. expected > checksum is 978416968 and actual is checksum 1807767534. expected length is > 330 and actual length is 597 > 2020-04-27 07:27:30.740 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n0_Lucene70_0.dvm did not match. expected > checksum is 3599774596 and actual is checksum 2764190632. expected length is > 318017 and actual length is 311617 > 2020-04-27 07:27:30.743 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n0_Lucene70_0.dvd did not match. expected > checksum is 3141314417 and actual is checksum 3847788889. expected length is > 1258075 and actual length is 3539994 > 2020-04-27 07:27:30.749 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n0.fnm did not match. expected checksum is > 1404317797 and actual is checksum 2471843701. expected length is 105295 and > actual length is 107020 > 2020-04-27 07:27:30.752 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n0.fdt did not match. expected checksum is > 3412288246 and actual is checksum 922481525. expected length is 7580 and > actual length is 17733 > 2020-04-27 07:27:30.754 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n1_Lucene50_0.doc did not match. expected > checksum is 2675540020 and actual is checksum 1452209674. expected length is > 110 and actual length is 110 > 2020-04-27 07:27:30.757 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n1_Lucene50_0.tim did not match. expected > checksum is 665797990 and actual is checksum 2134373365. expected length is > 10715 and actual length is 3418 > 2020-04-27 07:27:30.759 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n1.fdt did not match. expected checksum is > 2006427884 and actual is checksum 527297847. expected length is 7178 and > actual length is 2326 > 2020-04-27 07:27:30.761 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n1.fnm did not match. expected checksum is > 200552893 and actual is checksum 3290522141. expected length is 106580 and > actual length is 105206 > 2020-04-27 07:27:30.764 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n1_Lucene50_0.tip did not match. expected > checksum is 1511033560 and actual is checksum 3689083691. expected length is > 278 and actual length is 154 > 2020-04-27 07:27:30.767 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n1_Lucene70_0.dvm did not match. expected > checksum is 1435050119 and actual is checksum 6864243. expected length is > 323217 and actual length is 338063 > 2020-04-27 07:27:30.770 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n1.si did not match. expected checksum is > 1107038327 and actual is checksum 3068870234. expected length is 508 and > actual length is 508 > 2020-04-27 07:27:30.772 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n1.fdx did not match. expected checksum is > 2853806777 and actual is checksum 459866651. expected length is 93 and actual > length is 87 > 2020-04-27 07:27:30.775 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n1_Lucene70_0.dvd did not match. expected > checksum is 1356537478 and actual is checksum 2594306289. expected length is > 1179215 and actual length is 249458 > 2020-04-27 07:27:30.780 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n2.fnm did not match. expected checksum is > 1785795479 and actual is checksum 65014613. expected length is 106580 and > actual length is 106800 > 2020-04-27 07:27:30.783 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n2_Lucene50_0.tip did not match. expected > checksum is 1824666616 and actual is checksum 2857527783. expected length is > 507 and actual length is 637 > 2020-04-27 07:27:30.785 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n2.fdt did not match. expected checksum is > 1384446542 and actual is checksum 3981590407. expected length is 13686 and > actual length is 18516 > 2020-04-27 07:27:30.788 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n2_Lucene70_0.dvm did not match. expected > checksum is 3380549662 and actual is checksum 2002436369. expected length is > 317777 and actual length is 319213 > 2020-04-27 07:27:30.791 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n2_Lucene50_0.doc did not match. expected > checksum is 1594954666 and actual is checksum 1892723425. expected length is > 110 and actual length is 110 > 2020-04-27 07:27:30.794 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n2_Lucene50_0.tim did not match. expected > checksum is 305271241 and actual is checksum 854278263. expected length is > 20522 and actual length is 27836 > 2020-04-27 07:27:30.796 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n2.si did not match. expected checksum is > 2220527769 and actual is checksum 2016379681. expected length is 508 and > actual length is 508 > 2020-04-27 07:27:30.799 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n2.fdx did not match. expected checksum is > 1648054590 and actual is checksum 137025071. expected length is 100 and > actual length is 105 > 2020-04-27 07:27:30.801 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n2_Lucene70_0.dvd did not match. expected > checksum is 262113808 and actual is checksum 327562225. expected length is > 2731808 and actual length is 3909012 > 2020-04-27 07:27:30.809 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n3_Lucene70_0.dvd did not match. expected > checksum is 2157667750 and actual is checksum 3043505208. expected length is > 36906 and actual length is 163443 > 2020-04-27 07:27:30.812 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n3.fdt did not match. expected checksum is > 3456088243 and actual is checksum 1715362430. expected length is 453 and > actual length is 1636 > 2020-04-27 07:27:30.814 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n3.fnm did not match. expected checksum is > 4028738113 and actual is checksum 4245480574. expected length is 104876 and > actual length is 105515 > 2020-04-27 07:27:30.817 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n3_Lucene50_0.doc did not match. expected > checksum is 3120632072 and actual is checksum 1945528118. expected length is > 110 and actual length is 110 > 2020-04-27 07:27:30.819 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n3_Lucene50_0.tip did not match. expected > checksum is 3303166970 and actual is checksum 1740028553. expected length is > 114 and actual length is 137 > 2020-04-27 07:27:30.822 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n3_Lucene50_0.tim did not match. expected > checksum is 126700369 and actual is checksum 162509958. expected length is > 700 and actual length is 2398 > 2020-04-27 07:27:30.825 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n3.fdx did not match. expected checksum is > 3157051968 and actual is checksum 137679212. expected length is 84 and actual > length is 86 > 2020-04-27 07:27:30.827 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n3_Lucene70_0.dvm did not match. expected > checksum is 3969090178 and actual is checksum 2364978843. expected length is > 127576 and actual length is 266581 > 2020-04-27 07:27:30.830 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n3.si did not match. expected checksum is > 3730892058 and actual is checksum 2734980133. expected length is 508 and > actual length is 508 > 2020-04-27 07:27:30.832 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n4.fnm did not match. expected checksum is > 3259266514 and actual is checksum 2051283039. expected length is 104855 and > actual length is 106819 > 2020-04-27 07:27:30.835 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n4_Lucene50_0.tim did not match. expected > checksum is 380153843 and actual is checksum 580580220. expected length is > 1822 and actual length is 1737 > 2020-04-27 07:27:30.838 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n4_Lucene50_0.doc did not match. expected > checksum is 4094454161 and actual is checksum 421930789. expected length is > 110 and actual length is 110 > 2020-04-27 07:27:30.840 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n4.fdt did not match. expected checksum is > 911575783 and actual is checksum 2090965125. expected length is 1244 and > actual length is 1201 > 2020-04-27 07:27:30.842 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n4_Lucene50_0.tip did not match. expected > checksum is 1279063865 and actual is checksum 765869778. expected length is > 128 and actual length is 130 > 2020-04-27 07:27:30.845 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n4_Lucene70_0.dvm did not match. expected > checksum is 1689040450 and actual is checksum 627554920. expected length is > 223473 and actual length is 230157 > 2020-04-27 07:27:30.848 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n4.fdx did not match. expected checksum is > 1278110969 and actual is checksum 1444534251. expected length is 86 and > actual length is 86 > 2020-04-27 07:27:30.850 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n4_Lucene70_0.dvd did not match. expected > checksum is 1992192696 and actual is checksum 3455180678. expected length is > 119844 and actual length is 117860 > 2020-04-27 07:27:30.853 WARN > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.h.IndexFetcher File _1n4.si did not match. expected checksum is > 2776236859 and actual is checksum 2572132550. expected length is 508 and > actual length is 508 > 2020-04-27 07:27:35.866 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1lu.si did not match. expected checksum is > 2573424453 and actual is checksum 3612352961. expected length is 546 and > actual length is 437 > 2020-04-27 07:27:35.868 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1kr.cfe did not match. expected checksum is > 2592320603 and actual is checksum 1374033039. expected length is 289 and > actual length is 289 > 2020-04-27 07:27:35.872 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1kr.si did not match. expected checksum is > 2911257988 and actual is checksum 2011202433. expected length is 437 and > actual length is 437 > 2020-04-27 07:27:35.874 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1kr.cfs did not match. expected checksum is > 1304802713 and actual is checksum 454030092. expected length is 7598425 and > actual length is 49265169 > 2020-04-27 07:27:37.234 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1mo.si did not match. expected checksum is > 1422458436 and actual is checksum 703149874. expected length is 437 and > actual length is 437 > 2020-04-27 07:27:38.117 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1mo.cfs did not match. expected checksum is > 1892273984 and actual is checksum 2087644984. expected length is 50710602 and > actual length is 10402264 > 2020-04-27 07:27:38.532 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1mo.cfe did not match. expected checksum is > 3864394245 and actual is checksum 1846889307. expected length is 289 and > actual length is 289 > 2020-04-27 07:27:38.575 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1m9.fdx did not match. expected checksum is > 660375856 and actual is checksum 571052523. expected length is 114 and actual > length is 84 > 2020-04-27 07:27:38.590 WARN > (recoveryExecutor-2639-thread-1-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard2_replica_n4 s:shard2 c:mycollection r:core_node7) > [c:mycollection s:shard2 r:core_node7 x:mycollection_shard2_replica_n4] > o.a.s.u.UpdateLog Log replay finished. recoveryInfo=RecoveryInfo{adds=31744 > deletes=0 deleteByQuery=0 errors=0 positionOfStart=2720536} > 2020-04-27 07:27:39.259 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1m9_Lucene70_0.dvm did not match. expected > checksum is 421440641 and actual is checksum 2075160524. expected length is > 337361 and actual length is 141820 > 2020-04-27 07:27:44.881 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1m9_Lucene70_0.dvd did not match. expected > checksum is 47189924 and actual is checksum 3640637009. expected length is > 5235629 and actual length is 48679 > 2020-04-27 07:27:45.335 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1m9_Lucene50_0.tip did not match. expected > checksum is 2305624674 and actual is checksum 704932512. expected length is > 693 and actual length is 115 > 2020-04-27 07:27:45.939 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1m9.fdt did not match. expected checksum is > 1157089232 and actual is checksum 2388254324. expected length is 23435 and > actual length is 673 > 2020-04-27 07:27:45.941 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1m9.si did not match. expected checksum is > 2811885732 and actual is checksum 2854129172. expected length is 508 and > actual length is 508 > 2020-04-27 07:27:45.943 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1m9_Lucene50_0.doc did not match. expected > checksum is 1925626873 and actual is checksum 1921615605. expected length is > 110 and actual length is 110 > 2020-04-27 07:27:45.945 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1m9_Lucene50_0.tim did not match. expected > checksum is 3627283142 and actual is checksum 3796132842. expected length is > 34596 and actual length is 958 > 2020-04-27 07:27:45.947 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1m9.fnm did not match. expected checksum is > 1291024098 and actual is checksum 2700484857. expected length is 107020 and > actual length is 104209 > 2020-04-27 07:27:45.949 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1mm_Lucene50_0.tip did not match. expected > checksum is 2785033346 and actual is checksum 1911387906. expected length is > 816 and actual length is 760 > 2020-04-27 07:27:45.951 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1mm_Lucene70_0.dvd did not match. expected > checksum is 942596692 and actual is checksum 3039336477. expected length is > 5557044 and actual length is 4847564 > 2020-04-27 07:27:45.974 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1mm.fdx did not match. expected checksum is > 143536399 and actual is checksum 2962535892. expected length is 116 and > actual length is 111 > 2020-04-27 07:27:45.976 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1mm_Lucene70_0.dvm did not match. expected > checksum is 3884436961 and actual is checksum 1345841996. expected length is > 339801 and actual length is 336809 > 2020-04-27 07:27:45.980 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1mm.fdt did not match. expected checksum is > 3902419774 and actual is checksum 3791519088. expected length is 23306 and > actual length is 21047 > 2020-04-27 07:27:45.981 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1mm.si did not match. expected checksum is > 45470103 and actual is checksum 235515609. expected length is 508 and actual > length is 508 > 2020-04-27 07:27:45.983 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1mm_Lucene50_0.doc did not match. expected > checksum is 408594907 and actual is checksum 2650340971. expected length is > 110 and actual length is 110 > 2020-04-27 07:27:45.985 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1mm_Lucene50_0.tim did not match. expected > checksum is 761408352 and actual is checksum 685575313. expected length is > 35256 and actual length is 31644 > 2020-04-27 07:27:45.987 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1mm.fnm did not match. expected checksum is > 811204953 and actual is checksum 1244308773. expected length is 107460 and > actual length is 107460 > 2020-04-27 07:27:45.989 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1mr.fnm did not match. expected checksum is > 3813441918 and actual is checksum 2133371346. expected length is 107460 and > actual length is 106360 > 2020-04-27 07:27:45.991 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1mr.fdt did not match. expected checksum is > 4263416408 and actual is checksum 3722461284. expected length is 16698 and > actual length is 21392 > 2020-04-27 07:27:45.993 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1mr_Lucene50_0.doc did not match. expected > checksum is 632422377 and actual is checksum 501799146. expected length is > 110 and actual length is 110 > 2020-04-27 07:27:45.994 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1mr_Lucene50_0.tim did not match. expected > checksum is 3170165892 and actual is checksum 4152937488. expected length is > 25090 and actual length is 32328 > 2020-04-27 07:27:46.000 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1mr_Lucene50_0.tip did not match. expected > checksum is 2679120655 and actual is checksum 3243853760. expected length is > 553 and actual length is 726 > 2020-04-27 07:27:46.001 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1mr_Lucene70_0.dvm did not match. expected > checksum is 2991034554 and actual is checksum 1980969905. expected length is > 321785 and actual length is 322117 > 2020-04-27 07:27:46.004 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1mr_Lucene70_0.dvd did not match. expected > checksum is 2117172713 and actual is checksum 4064616498. expected length is > 3479018 and actual length is 4488695 > 2020-04-27 07:27:46.017 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1mr.fdx did not match. expected checksum is > 1613761705 and actual is checksum 4154864516. expected length is 106 and > actual length is 111 > 2020-04-27 07:27:46.019 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1mr.si did not match. expected checksum is > 3059265880 and actual is checksum 531516038. expected length is 508 and > actual length is 508 > 2020-04-27 07:27:46.021 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1mx_Lucene50_0.tip did not match. expected > checksum is 1804496408 and actual is checksum 2903373400. expected length is > 627 and actual length is 621 > 2020-04-27 07:27:46.023 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1mx.si did not match. expected checksum is > 2241359534 and actual is checksum 4198491027. expected length is 508 and > actual length is 508 > 2020-04-27 07:27:46.024 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1mx_Lucene70_0.dvm did not match. expected > checksum is 3097209578 and actual is checksum 1124925868. expected length is > 312617 and actual length is 323465 > 2020-04-27 07:27:46.027 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1mx.fnm did not match. expected checksum is > 1909140548 and actual is checksum 2397297432. expected length is 106580 and > actual length is 107460 > 2020-04-27 07:27:46.029 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1mx_Lucene50_0.doc did not match. expected > checksum is 971818792 and actual is checksum 478139506. expected length is > 110 and actual length is 110 > 2020-04-27 07:27:46.031 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1mx.fdt did not match. expected checksum is > 1304288778 and actual is checksum 1544273325. expected length is 18170 and > actual length is 17756 > 2020-04-27 07:27:46.033 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1mx_Lucene50_0.tim did not match. expected > checksum is 4033259458 and actual is checksum 4016593800. expected length is > 27532 and actual length is 26636 > 2020-04-27 07:27:46.034 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1mx.fdx did not match. expected checksum is > 385809858 and actual is checksum 3497071885. expected length is 107 and > actual length is 104 > 2020-04-27 07:27:46.036 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1mx_Lucene70_0.dvd did not match. expected > checksum is 3567656601 and actual is checksum 2800338830. expected length is > 3622054 and actual length is 3792928 > 2020-04-27 07:27:46.051 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1n0.fdx did not match. expected checksum is > 2282222236 and actual is checksum 3183044106. expected length is 98 and > actual length is 108 > 2020-04-27 07:27:46.053 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1n0.si did not match. expected checksum is > 150588833 and actual is checksum 3492244992. expected length is 508 and > actual length is 508 > 2020-04-27 07:27:46.055 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1n0_Lucene50_0.doc did not match. expected > checksum is 1457615503 and actual is checksum 4089018914. expected length is > 110 and actual length is 110 > 2020-04-27 07:27:46.125 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1n0_Lucene50_0.tim did not match. expected > checksum is 3866656431 and actual is checksum 2014845387. expected length is > 16763 and actual length is 28481 > 2020-04-27 07:27:46.128 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1n0_Lucene50_0.tip did not match. expected > checksum is 3008320208 and actual is checksum 808227383. expected length is > 419 and actual length is 659 > 2020-04-27 07:27:46.130 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1n0_Lucene70_0.dvm did not match. expected > checksum is 2437844680 and actual is checksum 1756584754. expected length is > 310713 and actual length is 315009 > 2020-04-27 07:27:46.133 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1n0_Lucene70_0.dvd did not match. expected > checksum is 3036548497 and actual is checksum 343807338. expected length is > 2245614 and actual length is 4011325 > 2020-04-27 07:27:46.146 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1n0.fnm did not match. expected checksum is > 4233343814 and actual is checksum 2984252206. expected length is 106580 and > actual length is 107460 > 2020-04-27 07:27:46.148 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1n0.fdt did not match. expected checksum is > 2222747552 and actual is checksum 3771791266. expected length is 11198 and > actual length is 18927 > 2020-04-27 07:27:46.150 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1n2.fnm did not match. expected checksum is > 2062419431 and actual is checksum 885929423. expected length is 107460 and > actual length is 105632 > 2020-04-27 07:27:46.152 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1n2_Lucene50_0.tip did not match. expected > checksum is 1574973164 and actual is checksum 209201565. expected length is > 516 and actual length is 138 > 2020-04-27 07:27:46.154 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1n2.fdt did not match. expected checksum is > 3606049617 and actual is checksum 3605002010. expected length is 14213 and > actual length is 1590 > 2020-04-27 07:27:46.156 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1n2_Lucene70_0.dvm did not match. expected > checksum is 842753199 and actual is checksum 3527006266. expected length is > 316073 and actual length is 279943 > 2020-04-27 07:27:46.159 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1n2_Lucene50_0.doc did not match. expected > checksum is 2130892015 and actual is checksum 2031251178. expected length is > 110 and actual length is 110 > 2020-04-27 07:27:46.161 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1n2_Lucene50_0.tim did not match. expected > checksum is 1435858498 and actual is checksum 3731425121. expected length is > 21284 and actual length is 2331 > 2020-04-27 07:27:46.163 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1n2.si did not match. expected checksum is > 20126402 and actual is checksum 1544152005. expected length is 508 and actual > length is 508 > 2020-04-27 07:27:46.165 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1n2.fdx did not match. expected checksum is > 1651289033 and actual is checksum 1035172250. expected length is 100 and > actual length is 86 > 2020-04-27 07:27:46.166 WARN > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.h.IndexFetcher File _1n2_Lucene70_0.dvd did not match. expected > checksum is 3180610843 and actual is checksum 2027612627. expected length is > 2818109 and actual length is 159550 > 2020-04-27 07:27:52.815 WARN > (recoveryExecutor-2624-thread-1-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.u.UpdateLog Starting log replay > tlog{file=/data/dnb/cloud/node1/solr/mycollection_shard14_replica_n52/data/tlog/tlog.0000000000000000972 > refcount=2} active=true starting pos=1471697 inSortedOrder=false > 2020-04-27 07:27:53.592 WARN > (recoveryExecutor-2619-thread-1-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.u.UpdateLog Starting log replay > tlog{file=/data/dnb/cloud/node1/solr/mycollection_shard13_replica_n48/data/tlog/tlog.0000000000000000972 > refcount=2} active=true starting pos=211445345 inSortedOrder=false > 2020-04-27 07:27:56.965 WARN > (recoveryExecutor-2597-thread-1-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard1_replica_n1 s:shard1 c:mycollection r:core_node3) > [c:mycollection s:shard1 r:core_node3 x:mycollection_shard1_replica_n1] > o.a.s.u.UpdateLog Log replay finished. recoveryInfo=RecoveryInfo{adds=34889 > deletes=0 deleteByQuery=0 errors=0 positionOfStart=1404348} > 2020-04-27 07:27:57.329 WARN > (recoveryExecutor-2619-thread-1-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > o.a.s.u.UpdateLog Log replay finished. recoveryInfo=RecoveryInfo{adds=5597 > deletes=0 deleteByQuery=0 errors=0 positionOfStart=211445345} > 2020-04-27 07:28:03.537 WARN > (recoveryExecutor-2654-thread-1-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) > [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] > o.a.s.u.UpdateLog Log replay finished. recoveryInfo=RecoveryInfo{adds=35707 > deletes=0 deleteByQuery=0 errors=0 positionOfStart=2740945} > 2020-04-27 07:28:07.353 WARN > (recoveryExecutor-2629-thread-1-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard15_replica_n56 s:shard15 c:mycollection r:core_node59) > [c:mycollection s:shard15 r:core_node59 x:mycollection_shard15_replica_n56] > o.a.s.u.UpdateLog Log replay finished. recoveryInfo=RecoveryInfo{adds=35950 > deletes=0 deleteByQuery=0 errors=0 positionOfStart=3081052} > 2020-04-27 07:28:19.721 WARN > (recoveryExecutor-2624-thread-1-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > o.a.s.u.UpdateLog Log replay finished. recoveryInfo=RecoveryInfo{adds=38066 > deletes=0 deleteByQuery=0 errors=0 positionOfStart=1471697} > 2020-04-27 07:33:26.278 WARN > (updateExecutor-3-thread-827-processing-n:mysolrsever.com:6010_solr) > [c:mycollection s:shard10 r:core_node39 x:mycollection_shard10_replica_n36] > o.a.s.c.RecoveryStrategy Stopping recovery for > core=[mycollection_shard10_replica_n36] coreNodeName=[core_node39] > 2020-04-27 07:33:30.370 WARN > (recoveryExecutor-4-thread-833-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard10_replica_n36 s:shard10 c:mycollection r:core_node39) > [c:mycollection s:shard10 r:core_node39 x:mycollection_shard10_replica_n36] > o.a.s.h.IndexFetcher File _1mx.si did not match. expected checksum is > 1480542754 and actual is checksum 2732545715. expected length is 437 and > actual length is 508 > 2020-04-27 07:33:37.918 WARN > (recoveryExecutor-4-thread-833-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard10_replica_n36 s:shard10 c:mycollection r:core_node39) > [c:mycollection s:shard10 r:core_node39 x:mycollection_shard10_replica_n36] > o.a.s.h.IndexFetcher File _1mx.si did not match. expected checksum is > 1480542754 and actual is checksum 2732545715. expected length is 437 and > actual length is 508 > 2020-04-27 07:33:37.945 WARN > (recoveryExecutor-4-thread-833-processing-n:mysolrsever.com:6010_solr > x:mycollection_shard10_replica_n36 s:shard10 c:mycollection r:core_node39) > [c:mycollection s:shard10 r:core_node39 x:mycollection_shard10_replica_n36] > o.a.s.h.IndexFetcher File _1me.si did not match. expected checksum is > 204474971 and actual is checksum 3417943085. expected length is 437 and > actual length is 546 > 2020-04-27 07:34:07.322 WARN > (zkConnectionManagerCallback-6-thread-1-processing-n:mysolrsever.com:6010_solr-SendThread(zoo-prd-n1:2181)) > [ ] o.a.z.ClientCnxn Client session timed out, have not heard from server > in 10775ms for sessionid 0x171a6fb51310008 > 2020-04-27 07:34:07.426 WARN > (zkConnectionManagerCallback-6-thread-1-processing-n:mysolrsever.com:6010_solr-EventThread) > [ ] o.a.s.c.c.ConnectionManager Watcher > org.apache.solr.common.cloud.ConnectionManager@51b6bf14 name: > ZooKeeperConnection Watcher:myzoo01:2181,myzoo02:2181,myzoo03:2181 got event > WatchedEvent state:Disconnected type:None path:null path: null type: None > 2020-04-27 07:34:07.426 WARN > (zkConnectionManagerCallback-6-thread-1-processing-n:mysolrsever.com:6010_solr-EventThread) > [ ] o.a.s.c.c.ConnectionManager Watcher > org.apache.solr.common.cloud.ConnectionManager@51b6bf14 name: > ZooKeeperConnection Watcher:myzoo01:2181,myzoo02:2181,myzoo03:2181 got event > WatchedEvent state:Disconnected type:None path:null path: null type: None > 2020-04-27 07:34:07.426 WARN > (zkConnectionManagerCallback-6-thread-1-processing-n:mysolrsever.com:6010_solr-EventThread) > [ ] o.a.s.c.c.ConnectionManager zkClient has disconnected > 2020-04-28 07:53:30.910 WARN > (zkConnectionManagerCallback-6-thread-1-processing-n:mysolrsever.com:6010_solr-SendThread(zoo-prd-n2:2181)) > [ ] o.a.z.ClientCnxn Client session timed out, have not heard from server > in 10528ms for sessionid 0x171a6fb51310008 > 2020-04-28 07:53:31.015 WARN > (zkConnectionManagerCallback-6-thread-1-processing-n:mysolrsever.com:6010_solr-EventThread) > [ ] o.a.s.c.c.ConnectionManager Watcher > org.apache.solr.common.cloud.ConnectionManager@51b6bf14 name: > ZooKeeperConnection Watcher:myzoo01:2181,myzoo02:2181,myzoo03:2181 got event > WatchedEvent state:Disconnected type:None path:null path: null type: None > 2020-04-28 07:53:31.015 WARN > (zkConnectionManagerCallback-6-thread-1-processing-n:mysolrsever.com:6010_solr-EventThread) > [ ] o.a.s.c.c.ConnectionManager zkClient has disconnected
