You can add something like this to SOLR_OPTS -DzkClientTimeout=30000
in your init script or adjust solr.xml <int
name="zkClientTimeout">${zkClientTimeout:15000}</int>On Wed, Apr 29, 2020 at 5:41 PM Sethuraman, Ganesh <sethuram...@dnb.com.invalid> wrote: > > 3 Zookeeper ensemble are all in 3 separate boxes (EC2 instances). Each have > separate transactional logs directory (separate EBS volume, separate disk), > as this was zookeeper best practices. > > It feels ZK timeout is more of symptom and Solr slowness is the cause. Having > said that, do you increase the timeout setting in Solr or Zookeeper, if you > can share parameters it will certainly help > > Regards > Ganesh > > -----Original Message----- > From: matthew sporleder <msporle...@gmail.com> > Sent: Wednesday, April 29, 2020 11:47 AM > To: solr-user@lucene.apache.org > Subject: Re: SolrCloud degraded during backup and batch CSV update > > CAUTION: This email originated from outside of D&B. Please do not click links > or open attachments unless you recognize the sender and know the content is > safe. > > > FWIW I've had some luck with strategy 3 (increase zk timeout) when you > overwhelm the connection to zk or the disk on zk. > > Is zk on the same boxes as solr? > > On Tue, Apr 28, 2020 at 10:15 PM Sethuraman, Ganesh > <sethuram...@dnb.com.invalid> wrote: > > > > Hi > > > > We are using SolrCloud 7.2.1 with 3 node Zookeeper ensemble. We have 92 > > collection each on avg. having 8 shards and 2 replica with 2 EC2 nodes, > > with JVM size of 18GB (G1 GC). We need your help with the Issue we faced > > today: The issue is SolrCloud server went into a degraded collections (for > > few collections) when the Solr backup and the Solr batch CSV update load > > happened at the same time as backup. The CSV data load was about ~5 GB per > > shard/replica. We think this happened after zkClient disconnect happened as > > noted below. We had to restart Solr to bring it back to normal. > > > > > > 1. Is it not suggested to run backup and Solr batch CSV update large > > load at the same time? > > 2. In the past we have seen two CSV batch update load in parallel causes > > issues, is this also not suggested (this issue is not related to that)? > > 3. Do you think we should increase Zookeeper timeout? > > 4. How do we know if we need to up the JVM Max memory, and by how much? > > 5. We also see that once the Solr goes into degraded collection and > > recovery failed, it NEVER get back to normal, even after when there is no > > load. Is this a bug? > > > > The GC information and Solr Log below > > > > https://nam03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgceasy.io%2Fmy-gc-report.jsp%3Fp%3Dc2hhcmVkLzIwMjAvMDQvMjkvLS0wMl9zb2xyX2djLmxvZy56aXAtLTEtNDAtMzE%3D%26channel%3DWEB&data=02%7C01%7CSethuramanG%40dnb.com%7C8e47fdc157ce4bf8425c08d7ec548d6c%7C19e2b708bf12437597198dec42771b3e%7C0%7C0%7C637237720241486583&sdata=MlstokkgBAX7joUpljJnQZjrbQ7cZZAoSPfQebx2q5I%3D&reserved=0 > > > > > > 2020-04-27 07:34:07.322 WARN > > (zkConnectionManagerCallback-6-thread-1-processing-n:mysolrsever.com:6010_solr-SendThread(zoo-prd-n1:2181)) > > [ ] o.a.z.ClientCnxn Client session timed out, have not heard from > > server in 10775ms for sessionid 0x171a6fb51310008 > > .... > > 2020-04-27 07:34:07.426 WARN > > (zkConnectionManagerCallback-6-thread-1-processing-n:mysolrsever.com:6010_solr-EventThread) > > [ ] o.a.s.c.c.ConnectionManager zkClient has disconnected > > > > > > > > > > SOLR Log Below (Curtailed WARN log) > > -------------------- > > 2020-04-27 07:26:45.402 WARN > > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > > o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of > > 587 bytes) > > java.io.EOFException > > at > > org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168) > > at > > org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160) > > at > > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579) > > at > > org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545) > > at > > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526) > > at > > org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008) > > at > > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566) > > at > > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345) > > at > > org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420) > > at > > org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225) > > at > > org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626) > > at > > org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308) > > at > > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292) > > at > > com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) > > at > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > > at > > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188) > > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > at java.lang.Thread.run(Thread.java:748) > > 2020-04-27 07:26:45.405 WARN > > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > > o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of > > 587 bytes) > > java.io.EOFException > > at > > org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168) > > at > > org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160) > > at > > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579) > > at > > org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545) > > at > > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526) > > at > > org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008) > > at > > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566) > > at > > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345) > > at > > org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420) > > at > > org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225) > > at > > org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626) > > at > > org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308) > > at > > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292) > > at > > com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) > > at > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > > at > > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188) > > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > at java.lang.Thread.run(Thread.java:748) > > 2020-04-27 07:26:45.406 WARN > > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > > o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of > > 587 bytes) > > java.io.EOFException > > at > > org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168) > > at > > org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160) > > at > > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579) > > at > > org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545) > > at > > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526) > > at > > org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008) > > at > > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566) > > at > > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345) > > at > > org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420) > > at > > org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225) > > at > > org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626) > > at > > org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308) > > at > > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292) > > at > > com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) > > at > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > > at > > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188) > > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > at java.lang.Thread.run(Thread.java:748) > > 2020-04-27 07:26:45.408 WARN > > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > > o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of > > 587 bytes) > > java.io.EOFException > > at > > org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168) > > at > > org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160) > > at > > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579) > > at > > org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545) > > at > > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526) > > at > > org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008) > > at > > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566) > > at > > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345) > > at > > org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420) > > at > > org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225) > > at > > org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626) > > at > > org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308) > > at > > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292) > > at > > com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) > > at > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > > at > > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188) > > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > at java.lang.Thread.run(Thread.java:748) > > 2020-04-27 07:26:45.410 WARN > > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > > o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of > > 587 bytes) > > java.io.EOFException > > at > > org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168) > > at > > org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160) > > at > > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579) > > at > > org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545) > > at > > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526) > > at > > org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008) > > at > > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566) > > at > > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345) > > at > > org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420) > > at > > org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225) > > at > > org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626) > > at > > org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308) > > at > > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292) > > at > > com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) > > at > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > > at > > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188) > > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > at java.lang.Thread.run(Thread.java:748) > > 2020-04-27 07:26:45.412 WARN > > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > > o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of > > 587 bytes) > > java.io.EOFException > > at > > org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168) > > at > > org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160) > > at > > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579) > > at > > org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545) > > at > > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526) > > at > > org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008) > > at > > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566) > > at > > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345) > > at > > org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420) > > at > > org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225) > > at > > org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626) > > at > > org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308) > > at > > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292) > > at > > com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) > > at > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > > at > > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188) > > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > at java.lang.Thread.run(Thread.java:748) > > 2020-04-27 07:26:45.412 ERROR > > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > > o.a.s.h.IndexFetcher Error fetching file, doing one > > retry...:org.apache.solr.common.SolrException: Unable to download > > _1kr_r.liv completely. Downloaded 0!=587 > > at > > org.apache.solr.handler.IndexFetcher$FileFetcher.cleanup(IndexFetcher.java:1673) > > at > > org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1556) > > at > > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1526) > > at > > org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008) > > at > > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566) > > at > > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345) > > at > > org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420) > > at > > org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225) > > at > > org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626) > > at > > org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308) > > at > > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292) > > at > > com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) > > at > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > > at > > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188) > > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > at java.lang.Thread.run(Thread.java:748) > > > > 2020-04-27 07:26:45.414 WARN > > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > > o.a.s.h.IndexFetcher Error in fetching file: _1kr_r.liv (downloaded 0 of > > 587 bytes) > > java.io.EOFException > > at > > org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:168) > > at > > org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:160) > > at > > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchPackets(IndexFetcher.java:1579) > > at > > org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1545) > > at > > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1531) > > at > > org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008) > > at > > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566) > > at > > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345) > > at > > org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420) > > at > > org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225) > > at > > org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626) > > at > > org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308) > > at > > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292) > > at > > com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) > > at > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > > at > > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188) > > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > at java.lang.Thread.run(Thread.java:748) > > 2020-04-27 07:26:45.414 ERROR > > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > > o.a.s.h.IndexFetcher Error deleting file: _1kr_r.liv > > java.nio.file.NoSuchFileException: > > /data/dnb/cloud/node1/solr/mycollection_shard13_replica_n48/data/index.20200427072258517/_1kr_r.liv > > at > > sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) > > at > > sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) > > at > > sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) > > at > > sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:244) > > at > > sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103) > > at java.nio.file.Files.delete(Files.java:1126) > > at > > org.apache.lucene.store.FSDirectory.privateDeleteFile(FSDirectory.java:373) > > at > > org.apache.lucene.store.FSDirectory.deleteFile(FSDirectory.java:335) > > at > > org.apache.lucene.store.NRTCachingDirectory.deleteFile(NRTCachingDirectory.java:119) > > at > > org.apache.solr.handler.IndexFetcher$DirectoryFile.delete(IndexFetcher.java:1757) > > at > > org.apache.solr.handler.IndexFetcher$FileFetcher.cleanup(IndexFetcher.java:1667) > > at > > org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1556) > > at > > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1531) > > at > > org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008) > > at > > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566) > > at > > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345) > > at > > org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420) > > at > > org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225) > > at > > org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626) > > at > > org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308) > > at > > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292) > > at > > com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) > > at > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > > at > > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188) > > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > at java.lang.Thread.run(Thread.java:748) > > 2020-04-27 07:26:45.449 WARN > > (recoveryExecutor-4-thread-699-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard2_replica_n4 s:shard2 c:mycollection r:core_node7) > > [c:mycollection s:shard2 r:core_node7 x:mycollection_shard2_replica_n4] > > o.a.s.h.IndexFetcher File _1kq.cfs did not match. expected checksum is > > 3934327071 and actual is checksum 3095753427. expected length is 33235371 > > and actual length is 54180642 > > 2020-04-27 07:26:45.449 WARN > > (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) > > [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] > > o.a.s.h.IndexFetcher File _1em.fnm did not match. expected checksum is > > 138861107 and actual is checksum 188997082. expected length is 107460 and > > actual length is 107460 > > 2020-04-27 07:26:45.457 WARN > > (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) > > [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] > > o.a.s.h.IndexFetcher File _1mt_Lucene50_0.tim did not match. expected > > checksum is 2947323803 and actual is checksum 979431341. expected length is > > 32176 and actual length is 26089 > > 2020-04-27 07:26:45.521 WARN > > (recoveryExecutor-4-thread-648-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard6_replica_n20 s:shard6 c:mycollection r:core_node23) > > [c:mycollection s:shard6 r:core_node23 x:mycollection_shard6_replica_n20] > > o.a.s.h.IndexFetcher File _1gk_Lucene50_0.doc did not match. expected > > checksum is 2087731087 and actual is checksum 3540174119. expected length > > is 110 and actual length is 110 > > 2020-04-27 07:26:45.521 WARN > > (recoveryExecutor-4-thread-657-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard1_replica_n1 s:shard1 c:mycollection r:core_node3) > > [c:mycollection s:shard1 r:core_node3 x:mycollection_shard1_replica_n1] > > o.a.s.h.IndexFetcher File _1mo.cfs did not match. expected checksum is > > 777834759 and actual is checksum 289576304. expected length is 59980991 and > > actual length is 6236424 > > 2020-04-27 07:26:45.521 WARN > > (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) > > [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] > > o.a.s.h.IndexFetcher File _1mt.fdx did not match. expected checksum is > > 1321210176 and actual is checksum 224636564. expected length is 114 and > > actual length is 106 > > 2020-04-27 07:26:45.521 WARN > > (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) > > [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] > > o.a.s.h.IndexFetcher File _1em.fdt did not match. expected checksum is > > 3790133021 and actual is checksum 2516047051. expected length is 1477755 > > and actual length is 1477850 > > 2020-04-27 07:26:45.578 WARN > > (recoveryExecutor-4-thread-648-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard6_replica_n20 s:shard6 c:mycollection r:core_node23) > > [c:mycollection s:shard6 r:core_node23 x:mycollection_shard6_replica_n20] > > o.a.s.h.IndexFetcher File _1gk_Lucene50_0.tim did not match. expected > > checksum is 3690299979 and actual is checksum 411724865. expected length is > > 4392270 and actual length is 4396498 > > 2020-04-27 07:26:45.588 WARN > > (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) > > [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] > > o.a.s.h.IndexFetcher File _1em_Lucene50_0.doc did not match. expected > > checksum is 937224718 and actual is checksum 1176441880. expected length is > > 110 and actual length is 110 > > 2020-04-27 07:26:45.733 WARN > > (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) > > [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] > > o.a.s.h.IndexFetcher File _1em_Lucene50_0.tim did not match. expected > > checksum is 1429458386 and actual is checksum 4155623862. expected length > > is 2213909 and actual length is 2213939 > > 2020-04-27 07:26:45.754 ERROR > > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > > o.a.s.h.ReplicationHandler Index fetch failed > > :org.apache.solr.common.SolrException: Unable to download _1kr_r.liv > > completely. Downloaded 0!=587 > > at > > org.apache.solr.handler.IndexFetcher$FileFetcher.cleanup(IndexFetcher.java:1673) > > at > > org.apache.solr.handler.IndexFetcher$FileFetcher.fetch(IndexFetcher.java:1556) > > at > > org.apache.solr.handler.IndexFetcher$FileFetcher.fetchFile(IndexFetcher.java:1531) > > at > > org.apache.solr.handler.IndexFetcher.downloadIndexFiles(IndexFetcher.java:1008) > > at > > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:566) > > at > > org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:345) > > at > > org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:420) > > at > > org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:225) > > at > > org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626) > > at > > org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308) > > at > > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292) > > at > > com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) > > at > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > > at > > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188) > > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > at java.lang.Thread.run(Thread.java:748) > > > > 2020-04-27 07:26:45.755 ERROR > > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > > o.a.s.c.RecoveryStrategy Error while trying to > > recover:org.apache.solr.common.SolrException: Replication for recovery > > failed. > > at > > org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:228) > > at > > org.apache.solr.cloud.RecoveryStrategy.doSyncOrReplicateRecovery(RecoveryStrategy.java:626) > > at > > org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:308) > > at > > org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:292) > > at > > com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176) > > at > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > > at > > org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:188) > > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > at java.lang.Thread.run(Thread.java:748) > > > > 2020-04-27 07:26:45.755 ERROR > > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > > o.a.s.c.RecoveryStrategy Recovery failed - trying again... (1) > > 2020-04-27 07:26:45.780 WARN > > (recoveryExecutor-4-thread-648-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard6_replica_n20 s:shard6 c:mycollection r:core_node23) > > [c:mycollection s:shard6 r:core_node23 x:mycollection_shard6_replica_n20] > > o.a.s.h.IndexFetcher File _1gk_Lucene70_0.dvm did not match. expected > > checksum is 1474431645 and actual is checksum 1271168044. expected length > > is 275570 and actual length is 274506 > > 2020-04-27 07:26:45.834 WARN > > (recoveryExecutor-4-thread-648-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard6_replica_n20 s:shard6 c:mycollection r:core_node23) > > [c:mycollection s:shard6 r:core_node23 x:mycollection_shard6_replica_n20] > > o.a.s.h.IndexFetcher File _1gk.fdx did not match. expected checksum is > > 4129522944 and actual is checksum 4080519415. expected length is 6626 and > > actual length is 6626 > > 2020-04-27 07:26:45.843 WARN > > (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) > > [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] > > o.a.s.h.IndexFetcher File _1em.si did not match. expected checksum is > > 2493366746 and actual is checksum 31587394. expected length is 546 and > > actual length is 546 > > 2020-04-27 07:26:45.882 WARN > > (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) > > [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] > > o.a.s.h.IndexFetcher File _1mv.fdx did not match. expected checksum is > > 1392813257 and actual is checksum 3455988143. expected length is 103 and > > actual length is 111 > > 2020-04-27 07:26:45.882 WARN > > (recoveryExecutor-4-thread-648-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard6_replica_n20 s:shard6 c:mycollection r:core_node23) > > [c:mycollection s:shard6 r:core_node23 x:mycollection_shard6_replica_n20] > > o.a.s.h.IndexFetcher File _1gk_Lucene50_0.tip did not match. expected > > checksum is 2301226889 and actual is checksum 29696594. expected length is > > 86459 and actual length is 85786 > > 2020-04-27 07:26:45.882 WARN > > (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) > > [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] > > o.a.s.h.IndexFetcher File _1em_Lucene70_0.dvm did not match. expected > > checksum is 112676849 and actual is checksum 354222195. expected length is > > 286474 and actual length is 286546 > > 2020-04-27 07:26:45.911 WARN > > (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) > > [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] > > o.a.s.h.IndexFetcher File _1mv.fnm did not match. expected checksum is > > 2203892485 and actual is checksum 1165086749. expected length is 107460 and > > actual length is 106580 > > 2020-04-27 07:26:45.911 WARN > > (recoveryExecutor-4-thread-648-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard6_replica_n20 s:shard6 c:mycollection r:core_node23) > > [c:mycollection s:shard6 r:core_node23 x:mycollection_shard6_replica_n20] > > o.a.s.h.IndexFetcher File _1gk_2z.liv did not match. expected checksum is > > 3175370827 and actual is checksum 520836483. expected length is 51396 and > > actual length is 51404 > > 2020-04-27 07:26:45.912 WARN > > (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) > > [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] > > o.a.s.h.IndexFetcher File _1em.fdx did not match. expected checksum is > > 967260408 and actual is checksum 969726862. expected length is 3180 and > > actual length is 3180 > > 2020-04-27 07:26:45.914 WARN > > (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) > > [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] > > o.a.s.h.IndexFetcher File _1mv.fdt did not match. expected checksum is > > 719562508 and actual is checksum 956062095. expected length is 16717 and > > actual length is 21454 > > 2020-04-27 07:26:45.915 WARN > > (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) > > [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] > > o.a.s.h.IndexFetcher File _1em_Lucene50_0.tip did not match. expected > > checksum is 3833337361 and actual is checksum 493358852. expected length is > > 43996 and actual length is 44026 > > 2020-04-27 07:26:45.916 WARN > > (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) > > [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] > > o.a.s.h.IndexFetcher File _1mv_Lucene50_0.doc did not match. expected > > checksum is 1533628328 and actual is checksum 3735455395. expected length > > is 110 and actual length is 110 > > 2020-04-27 07:26:45.917 WARN > > (recoveryExecutor-4-thread-649-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) > > [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] > > o.a.s.h.IndexFetcher File _1em_Lucene70_0.dvd did not match. expected > > checksum is 75152145 and actual is checksum 1665090946. expected length is > > 497241188 and actual length is 497254814 > > 2020-04-27 07:26:45.918 WARN > > (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) > > [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] > > o.a.s.h.IndexFetcher File _1mv_Lucene50_0.tim did not match. expected > > checksum is 1349491281 and actual is checksum 3808036883. expected length > > is 25153 and actual length is 32322 > > 2020-04-27 07:26:45.920 WARN > > (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) > > [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] > > o.a.s.h.IndexFetcher File _1mv_Lucene50_0.tip did not match. expected > > checksum is 3820733560 and actual is checksum 2819887160. expected length > > is 571 and actual length is 725 > > 2020-04-27 07:26:46.034 WARN > > (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) > > [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] > > o.a.s.h.IndexFetcher File _1mv_Lucene70_0.dvm did not match. expected > > checksum is 1547030244 and actual is checksum 1122196351. expected length > > is 324313 and actual length is 323553 > > 2020-04-27 07:26:46.170 WARN > > (recoveryExecutor-4-thread-644-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard9_replica_n32 s:shard9 c:mycollection r:core_node35) > > [c:mycollection s:shard9 r:core_node35 x:mycollection_shard9_replica_n32] > > o.a.s.h.IndexFetcher File _1mv.si did not match. expected checksum is > > 751850967 and actual is checksum 1026527313. expected length is 508 and > > actual length is 508 > > and actual length is 319133 > > 2020-04-27 07:27:30.567 WARN > > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > > o.a.s.h.IndexFetcher File _1mr_Lucene70_0.dvd did not match. expected > > checksum is 3435029878 and actual is checksum 1202327807. expected length > > is 3337719 and actual length is 3375106 > > 2020-04-27 07:27:30.578 WARN > > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > > o.a.s.h.IndexFetcher File _1mr.fdx did not match. expected checksum is > > 2264443519 and actual is checksum 2299249501. expected length is 104 and > > actual length is 102 > > 2020-04-27 07:27:30.580 WARN > > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > > o.a.s.h.IndexFetcher File _1mr.si did not match. expected checksum is > > 1184027188 and actual is checksum 2810465617. expected length is 508 and > > actual length is 508 > > 2020-04-27 07:27:30.622 WARN > > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > > o.a.s.h.IndexFetcher File _1mv.fdx did not match. expected checksum is > > 2246788287 and actual is checksum 2682862227. expected length is 92 and > > actual length is 96 > > 2020-04-27 07:27:30.625 WARN > > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > > o.a.s.h.IndexFetcher File _1mv.fnm did not match. expected checksum is > > 3227520043 and actual is checksum 2778425164. expected length is 107137 and > > actual length is 107137 > > 2020-04-27 07:27:30.627 WARN > > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > > o.a.s.h.IndexFetcher File _1mv.fdt did not match. expected checksum is > > 1964200539 and actual is checksum 3997289541. expected length is 7398 and > > actual length is 9042 > > 2020-04-27 07:27:30.630 WARN > > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > > o.a.s.h.IndexFetcher File _1mv_Lucene50_0.doc did not match. expected > > checksum is 2165407577 and actual is checksum 2563543943. expected length > > is 110 and actual length is 110 > > 2020-04-27 07:27:30.632 WARN > > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > > o.a.s.h.IndexFetcher File _1mv_Lucene50_0.tim did not match. expected > > checksum is 1427836076 and actual is checksum 225462388. expected length is > > 11162 and actual length is 13623 > > 2020-04-27 07:27:30.634 WARN > > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > > o.a.s.h.IndexFetcher File _1mv_Lucene50_0.tip did not match. expected > > checksum is 2998276982 and actual is checksum 1966428087. expected length > > is 296 and actual length is 353 > > 2020-04-27 07:27:30.637 WARN > > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > > o.a.s.h.IndexFetcher File _1mv_Lucene70_0.dvm did not match. expected > > checksum is 2717609594 and actual is checksum 673086945. expected length is > > 315491 and actual length is 312531 > > 2020-04-27 07:27:30.640 WARN > > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > > o.a.s.h.IndexFetcher File _1mv.si did not match. expected checksum is > > 2885031416 and actual is checksum 762513018. expected length is 508 and > > actual length is 508 > > 2020-04-27 07:27:30.643 WARN > > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > > o.a.s.h.IndexFetcher File _1mv_Lucene70_0.dvd did not match. expected > > checksum is 602612626 and actual is checksum 2826680041. expected length is > > 1292092 and actual length is 1655171 > > 2020-04-27 07:27:30.669 WARN > > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > > o.a.s.h.IndexFetcher File _1mx_Lucene50_0.tip did not match. expected > > checksum is 401385741 and actual is checksum 3373755803. expected length is > > 621 and actual length is 715 > > 2020-04-27 07:27:30.671 WARN > > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > > o.a.s.h.IndexFetcher File _1mx.si did not match. expected checksum is > > 3708405765 and actual is checksum 1057460545. expected length is 508 and > > actual length is 508 > > 2020-04-27 07:27:30.674 WARN > > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > > o.a.s.h.IndexFetcher File _1mx_Lucene70_0.dvm did not match. expected > > checksum is 2755907323 and actual is checksum 1931314059. expected length > > is 327125 and actual length is 318737 > > 2020-04-27 07:27:30.677 WARN > > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > > o.a.s.h.IndexFetcher File _1mx.fnm did not match. expected checksum is > > 77505520 and actual is checksum 2602905677. expected length is 107240 and > > actual length is 107460 > > 2020-04-27 07:27:30.680 WARN > > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > > o.a.s.h.IndexFetcher File _1mx_Lucene50_0.doc did not match. expected > > checksum is 4204960097 and actual is checksum 2276943505. expected length > > is 110 and actual length is 110 > > 2020-04-27 07:27:30.682 WARN > > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > > o.a.s.h.IndexFetcher File _1mx.fdt did not match. expected checksum is > > 3414743450 and actual is checksum 3890346633. expected length is 18415 and > > actual length is 21922 > > 2020-04-27 07:27:30.685 WARN > > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > > o.a.s.h.IndexFetcher File _1mx_Lucene50_0.tim did not match. expected > > checksum is 2879829129 and actual is checksum 790212338. expected length is > > 27860 and actual length is 33235 > > 2020-04-27 07:27:30.687 WARN > > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > > o.a.s.h.IndexFetcher File _1mx.fdx did not match. expected checksum is > > 3356980932 and actual is checksum 1544608917. expected length is 108 and > > actual length is 112 > > 2020-04-27 07:27:30.690 WARN > > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > > o.a.s.h.IndexFetcher File _1mx_Lucene70_0.dvd did not match. expected > > checksum is 1979425912 and actual is checksum 584894070. expected length is > > 3726110 and actual length is 4689653 > > 2020-04-27 07:27:30.700 WARN > > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > > o.a.s.h.IndexFetcher File _1mz.si did not match. expected checksum is > > 3754169108 and actual is checksum 527001228. expected length is 508 and > > actual length is 508 > > 2020-04-27 07:27:30.702 WARN > > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > > o.a.s.h.IndexFetcher File _1mz_Lucene70_0.dvd did not match. expected > > checksum is 3981710860 and actual is checksum 1749011618. expected length > > is 1938123 and actual length is 33694 > > 2020-04-27 07:27:30.710 WARN > > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > > o.a.s.h.IndexFetcher File _1mz.fdx did not match. expected checksum is > > 1515379146 and actual is checksum 3212722427. expected length is 98 and > > actual length is 84 > > 2020-04-27 07:27:30.712 WARN > > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > > o.a.s.h.IndexFetcher File _1mz.fnm did not match. expected checksum is > > 1819400085 and actual is checksum 3372088644. expected length is 107020 and > > actual length is 103887 > > 2020-04-27 07:27:30.715 WARN > > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > > o.a.s.h.IndexFetcher File _1mz_Lucene50_0.doc did not match. expected > > checksum is 1401562358 and actual is checksum 2224139078. expected length > > is 110 and actual length is 110 > > 2020-04-27 07:27:30.718 WARN > > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > > o.a.s.h.IndexFetcher File _1mz_Lucene50_0.tim did not match. expected > > checksum is 3386701421 and actual is checksum 1434467863. expected length > > is 15944 and actual length is 685 > > 2020-04-27 07:27:30.720 WARN > > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > > o.a.s.h.IndexFetcher File _1mz.fdt did not match. expected checksum is > > 4259543754 and actual is checksum 3203989793. expected length is 10616 and > > actual length is 439 > > 2020-04-27 07:27:30.722 WARN > > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > > o.a.s.h.IndexFetcher File _1mz_Lucene50_0.tip did not match. expected > > checksum is 725882371 and actual is checksum 3650374401. expected length is > > 383 and actual length is 114 > > 2020-04-27 07:27:30.725 WARN > > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > > o.a.s.h.IndexFetcher File _1mz_Lucene70_0.dvm did not match. expected > > checksum is 3717717087 and actual is checksum 3712473868. expected length > > is 303729 and actual length is 117582 > > 2020-04-27 07:27:30.728 WARN > > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > > o.a.s.h.IndexFetcher File _1n0.fdx did not match. expected checksum is > > 3506718582 and actual is checksum 342046803. expected length is 93 and > > actual length is 104 > > 2020-04-27 07:27:30.731 WARN > > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > > o.a.s.h.IndexFetcher File _1n0.si did not match. expected checksum is > > 2920971663 and actual is checksum 2142622028. expected length is 508 and > > actual length is 508 > > 2020-04-27 07:27:30.733 WARN > > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > > o.a.s.h.IndexFetcher File _1n0_Lucene50_0.doc did not match. expected > > checksum is 723455769 and actual is checksum 2731538349. expected length is > > 110 and actual length is 110 > > 2020-04-27 07:27:30.735 WARN > > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > > o.a.s.h.IndexFetcher File _1n0_Lucene50_0.tim did not match. expected > > checksum is 2624342764 and actual is checksum 1283943229. expected length > > is 11424 and actual length is 26688 > > 2020-04-27 07:27:30.738 WARN > > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > > o.a.s.h.IndexFetcher File _1n0_Lucene50_0.tip did not match. expected > > checksum is 978416968 and actual is checksum 1807767534. expected length is > > 330 and actual length is 597 > > 2020-04-27 07:27:30.740 WARN > > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > > o.a.s.h.IndexFetcher File _1n0_Lucene70_0.dvm did not match. expected > > checksum is 3599774596 and actual is checksum 2764190632. expected length > > is 318017 and actual length is 311617 > > 2020-04-27 07:27:30.743 WARN > > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > > o.a.s.h.IndexFetcher File _1n0_Lucene70_0.dvd did not match. expected > > checksum is 3141314417 and actual is checksum 3847788889. expected length > > is 1258075 and actual length is 3539994 > > 2020-04-27 07:27:30.749 WARN > > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > > o.a.s.h.IndexFetcher File _1n0.fnm did not match. expected checksum is > > 1404317797 and actual is checksum 2471843701. expected length is 105295 and > > actual length is 107020 > > 2020-04-27 07:27:30.752 WARN > > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > > o.a.s.h.IndexFetcher File _1n0.fdt did not match. expected checksum is > > 3412288246 and actual is checksum 922481525. expected length is 7580 and > > actual length is 17733 > > 2020-04-27 07:27:30.754 WARN > > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > > o.a.s.h.IndexFetcher File _1n1_Lucene50_0.doc did not match. expected > > checksum is 2675540020 and actual is checksum 1452209674. expected length > > is 110 and actual length is 110 > > 2020-04-27 07:27:30.757 WARN > > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > > o.a.s.h.IndexFetcher File _1n1_Lucene50_0.tim did not match. expected > > checksum is 665797990 and actual is checksum 2134373365. expected length is > > 10715 and actual length is 3418 > > 2020-04-27 07:27:30.759 WARN > > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > > o.a.s.h.IndexFetcher File _1n1.fdt did not match. expected checksum is > > 2006427884 and actual is checksum 527297847. expected length is 7178 and > > actual length is 2326 > > 2020-04-27 07:27:30.761 WARN > > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > > o.a.s.h.IndexFetcher File _1n1.fnm did not match. expected checksum is > > 200552893 and actual is checksum 3290522141. expected length is 106580 and > > actual length is 105206 > > 2020-04-27 07:27:30.764 WARN > > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > > o.a.s.h.IndexFetcher File _1n1_Lucene50_0.tip did not match. expected > > checksum is 1511033560 and actual is checksum 3689083691. expected length > > is 278 and actual length is 154 > > 2020-04-27 07:27:30.767 WARN > > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > > o.a.s.h.IndexFetcher File _1n1_Lucene70_0.dvm did not match. expected > > checksum is 1435050119 and actual is checksum 6864243. expected length is > > 323217 and actual length is 338063 > > 2020-04-27 07:27:30.770 WARN > > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > > o.a.s.h.IndexFetcher File _1n1.si did not match. expected checksum is > > 1107038327 and actual is checksum 3068870234. expected length is 508 and > > actual length is 508 > > 2020-04-27 07:27:30.772 WARN > > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > > o.a.s.h.IndexFetcher File _1n1.fdx did not match. expected checksum is > > 2853806777 and actual is checksum 459866651. expected length is 93 and > > actual length is 87 > > 2020-04-27 07:27:30.775 WARN > > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > > o.a.s.h.IndexFetcher File _1n1_Lucene70_0.dvd did not match. expected > > checksum is 1356537478 and actual is checksum 2594306289. expected length > > is 1179215 and actual length is 249458 > > 2020-04-27 07:27:30.780 WARN > > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > > o.a.s.h.IndexFetcher File _1n2.fnm did not match. expected checksum is > > 1785795479 and actual is checksum 65014613. expected length is 106580 and > > actual length is 106800 > > 2020-04-27 07:27:30.783 WARN > > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > > o.a.s.h.IndexFetcher File _1n2_Lucene50_0.tip did not match. expected > > checksum is 1824666616 and actual is checksum 2857527783. expected length > > is 507 and actual length is 637 > > 2020-04-27 07:27:30.785 WARN > > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > > o.a.s.h.IndexFetcher File _1n2.fdt did not match. expected checksum is > > 1384446542 and actual is checksum 3981590407. expected length is 13686 and > > actual length is 18516 > > 2020-04-27 07:27:30.788 WARN > > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > > o.a.s.h.IndexFetcher File _1n2_Lucene70_0.dvm did not match. expected > > checksum is 3380549662 and actual is checksum 2002436369. expected length > > is 317777 and actual length is 319213 > > 2020-04-27 07:27:30.791 WARN > > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > > o.a.s.h.IndexFetcher File _1n2_Lucene50_0.doc did not match. expected > > checksum is 1594954666 and actual is checksum 1892723425. expected length > > is 110 and actual length is 110 > > 2020-04-27 07:27:30.794 WARN > > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > > o.a.s.h.IndexFetcher File _1n2_Lucene50_0.tim did not match. expected > > checksum is 305271241 and actual is checksum 854278263. expected length is > > 20522 and actual length is 27836 > > 2020-04-27 07:27:30.796 WARN > > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > > o.a.s.h.IndexFetcher File _1n2.si did not match. expected checksum is > > 2220527769 and actual is checksum 2016379681. expected length is 508 and > > actual length is 508 > > 2020-04-27 07:27:30.799 WARN > > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > > o.a.s.h.IndexFetcher File _1n2.fdx did not match. expected checksum is > > 1648054590 and actual is checksum 137025071. expected length is 100 and > > actual length is 105 > > 2020-04-27 07:27:30.801 WARN > > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > > o.a.s.h.IndexFetcher File _1n2_Lucene70_0.dvd did not match. expected > > checksum is 262113808 and actual is checksum 327562225. expected length is > > 2731808 and actual length is 3909012 > > 2020-04-27 07:27:30.809 WARN > > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > > o.a.s.h.IndexFetcher File _1n3_Lucene70_0.dvd did not match. expected > > checksum is 2157667750 and actual is checksum 3043505208. expected length > > is 36906 and actual length is 163443 > > 2020-04-27 07:27:30.812 WARN > > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > > o.a.s.h.IndexFetcher File _1n3.fdt did not match. expected checksum is > > 3456088243 and actual is checksum 1715362430. expected length is 453 and > > actual length is 1636 > > 2020-04-27 07:27:30.814 WARN > > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > > o.a.s.h.IndexFetcher File _1n3.fnm did not match. expected checksum is > > 4028738113 and actual is checksum 4245480574. expected length is 104876 and > > actual length is 105515 > > 2020-04-27 07:27:30.817 WARN > > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > > o.a.s.h.IndexFetcher File _1n3_Lucene50_0.doc did not match. expected > > checksum is 3120632072 and actual is checksum 1945528118. expected length > > is 110 and actual length is 110 > > 2020-04-27 07:27:30.819 WARN > > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > > o.a.s.h.IndexFetcher File _1n3_Lucene50_0.tip did not match. expected > > checksum is 3303166970 and actual is checksum 1740028553. expected length > > is 114 and actual length is 137 > > 2020-04-27 07:27:30.822 WARN > > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > > o.a.s.h.IndexFetcher File _1n3_Lucene50_0.tim did not match. expected > > checksum is 126700369 and actual is checksum 162509958. expected length is > > 700 and actual length is 2398 > > 2020-04-27 07:27:30.825 WARN > > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > > o.a.s.h.IndexFetcher File _1n3.fdx did not match. expected checksum is > > 3157051968 and actual is checksum 137679212. expected length is 84 and > > actual length is 86 > > 2020-04-27 07:27:30.827 WARN > > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > > o.a.s.h.IndexFetcher File _1n3_Lucene70_0.dvm did not match. expected > > checksum is 3969090178 and actual is checksum 2364978843. expected length > > is 127576 and actual length is 266581 > > 2020-04-27 07:27:30.830 WARN > > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > > o.a.s.h.IndexFetcher File _1n3.si did not match. expected checksum is > > 3730892058 and actual is checksum 2734980133. expected length is 508 and > > actual length is 508 > > 2020-04-27 07:27:30.832 WARN > > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > > o.a.s.h.IndexFetcher File _1n4.fnm did not match. expected checksum is > > 3259266514 and actual is checksum 2051283039. expected length is 104855 and > > actual length is 106819 > > 2020-04-27 07:27:30.835 WARN > > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > > o.a.s.h.IndexFetcher File _1n4_Lucene50_0.tim did not match. expected > > checksum is 380153843 and actual is checksum 580580220. expected length is > > 1822 and actual length is 1737 > > 2020-04-27 07:27:30.838 WARN > > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > > o.a.s.h.IndexFetcher File _1n4_Lucene50_0.doc did not match. expected > > checksum is 4094454161 and actual is checksum 421930789. expected length is > > 110 and actual length is 110 > > 2020-04-27 07:27:30.840 WARN > > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > > o.a.s.h.IndexFetcher File _1n4.fdt did not match. expected checksum is > > 911575783 and actual is checksum 2090965125. expected length is 1244 and > > actual length is 1201 > > 2020-04-27 07:27:30.842 WARN > > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > > o.a.s.h.IndexFetcher File _1n4_Lucene50_0.tip did not match. expected > > checksum is 1279063865 and actual is checksum 765869778. expected length is > > 128 and actual length is 130 > > 2020-04-27 07:27:30.845 WARN > > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > > o.a.s.h.IndexFetcher File _1n4_Lucene70_0.dvm did not match. expected > > checksum is 1689040450 and actual is checksum 627554920. expected length is > > 223473 and actual length is 230157 > > 2020-04-27 07:27:30.848 WARN > > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > > o.a.s.h.IndexFetcher File _1n4.fdx did not match. expected checksum is > > 1278110969 and actual is checksum 1444534251. expected length is 86 and > > actual length is 86 > > 2020-04-27 07:27:30.850 WARN > > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > > o.a.s.h.IndexFetcher File _1n4_Lucene70_0.dvd did not match. expected > > checksum is 1992192696 and actual is checksum 3455180678. expected length > > is 119844 and actual length is 117860 > > 2020-04-27 07:27:30.853 WARN > > (recoveryExecutor-4-thread-653-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > > o.a.s.h.IndexFetcher File _1n4.si did not match. expected checksum is > > 2776236859 and actual is checksum 2572132550. expected length is 508 and > > actual length is 508 > > 2020-04-27 07:27:35.866 WARN > > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > > o.a.s.h.IndexFetcher File _1lu.si did not match. expected checksum is > > 2573424453 and actual is checksum 3612352961. expected length is 546 and > > actual length is 437 > > 2020-04-27 07:27:35.868 WARN > > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > > o.a.s.h.IndexFetcher File _1kr.cfe did not match. expected checksum is > > 2592320603 and actual is checksum 1374033039. expected length is 289 and > > actual length is 289 > > 2020-04-27 07:27:35.872 WARN > > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > > o.a.s.h.IndexFetcher File _1kr.si did not match. expected checksum is > > 2911257988 and actual is checksum 2011202433. expected length is 437 and > > actual length is 437 > > 2020-04-27 07:27:35.874 WARN > > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > > o.a.s.h.IndexFetcher File _1kr.cfs did not match. expected checksum is > > 1304802713 and actual is checksum 454030092. expected length is 7598425 and > > actual length is 49265169 > > 2020-04-27 07:27:37.234 WARN > > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > > o.a.s.h.IndexFetcher File _1mo.si did not match. expected checksum is > > 1422458436 and actual is checksum 703149874. expected length is 437 and > > actual length is 437 > > 2020-04-27 07:27:38.117 WARN > > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > > o.a.s.h.IndexFetcher File _1mo.cfs did not match. expected checksum is > > 1892273984 and actual is checksum 2087644984. expected length is 50710602 > > and actual length is 10402264 > > 2020-04-27 07:27:38.532 WARN > > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > > o.a.s.h.IndexFetcher File _1mo.cfe did not match. expected checksum is > > 3864394245 and actual is checksum 1846889307. expected length is 289 and > > actual length is 289 > > 2020-04-27 07:27:38.575 WARN > > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > > o.a.s.h.IndexFetcher File _1m9.fdx did not match. expected checksum is > > 660375856 and actual is checksum 571052523. expected length is 114 and > > actual length is 84 > > 2020-04-27 07:27:38.590 WARN > > (recoveryExecutor-2639-thread-1-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard2_replica_n4 s:shard2 c:mycollection r:core_node7) > > [c:mycollection s:shard2 r:core_node7 x:mycollection_shard2_replica_n4] > > o.a.s.u.UpdateLog Log replay finished. recoveryInfo=RecoveryInfo{adds=31744 > > deletes=0 deleteByQuery=0 errors=0 positionOfStart=2720536} > > 2020-04-27 07:27:39.259 WARN > > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > > o.a.s.h.IndexFetcher File _1m9_Lucene70_0.dvm did not match. expected > > checksum is 421440641 and actual is checksum 2075160524. expected length is > > 337361 and actual length is 141820 > > 2020-04-27 07:27:44.881 WARN > > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > > o.a.s.h.IndexFetcher File _1m9_Lucene70_0.dvd did not match. expected > > checksum is 47189924 and actual is checksum 3640637009. expected length is > > 5235629 and actual length is 48679 > > 2020-04-27 07:27:45.335 WARN > > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > > o.a.s.h.IndexFetcher File _1m9_Lucene50_0.tip did not match. expected > > checksum is 2305624674 and actual is checksum 704932512. expected length is > > 693 and actual length is 115 > > 2020-04-27 07:27:45.939 WARN > > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > > o.a.s.h.IndexFetcher File _1m9.fdt did not match. expected checksum is > > 1157089232 and actual is checksum 2388254324. expected length is 23435 and > > actual length is 673 > > 2020-04-27 07:27:45.941 WARN > > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > > o.a.s.h.IndexFetcher File _1m9.si did not match. expected checksum is > > 2811885732 and actual is checksum 2854129172. expected length is 508 and > > actual length is 508 > > 2020-04-27 07:27:45.943 WARN > > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > > o.a.s.h.IndexFetcher File _1m9_Lucene50_0.doc did not match. expected > > checksum is 1925626873 and actual is checksum 1921615605. expected length > > is 110 and actual length is 110 > > 2020-04-27 07:27:45.945 WARN > > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > > o.a.s.h.IndexFetcher File _1m9_Lucene50_0.tim did not match. expected > > checksum is 3627283142 and actual is checksum 3796132842. expected length > > is 34596 and actual length is 958 > > 2020-04-27 07:27:45.947 WARN > > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > > o.a.s.h.IndexFetcher File _1m9.fnm did not match. expected checksum is > > 1291024098 and actual is checksum 2700484857. expected length is 107020 and > > actual length is 104209 > > 2020-04-27 07:27:45.949 WARN > > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > > o.a.s.h.IndexFetcher File _1mm_Lucene50_0.tip did not match. expected > > checksum is 2785033346 and actual is checksum 1911387906. expected length > > is 816 and actual length is 760 > > 2020-04-27 07:27:45.951 WARN > > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > > o.a.s.h.IndexFetcher File _1mm_Lucene70_0.dvd did not match. expected > > checksum is 942596692 and actual is checksum 3039336477. expected length is > > 5557044 and actual length is 4847564 > > 2020-04-27 07:27:45.974 WARN > > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > > o.a.s.h.IndexFetcher File _1mm.fdx did not match. expected checksum is > > 143536399 and actual is checksum 2962535892. expected length is 116 and > > actual length is 111 > > 2020-04-27 07:27:45.976 WARN > > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > > o.a.s.h.IndexFetcher File _1mm_Lucene70_0.dvm did not match. expected > > checksum is 3884436961 and actual is checksum 1345841996. expected length > > is 339801 and actual length is 336809 > > 2020-04-27 07:27:45.980 WARN > > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > > o.a.s.h.IndexFetcher File _1mm.fdt did not match. expected checksum is > > 3902419774 and actual is checksum 3791519088. expected length is 23306 and > > actual length is 21047 > > 2020-04-27 07:27:45.981 WARN > > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > > o.a.s.h.IndexFetcher File _1mm.si did not match. expected checksum is > > 45470103 and actual is checksum 235515609. expected length is 508 and > > actual length is 508 > > 2020-04-27 07:27:45.983 WARN > > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > > o.a.s.h.IndexFetcher File _1mm_Lucene50_0.doc did not match. expected > > checksum is 408594907 and actual is checksum 2650340971. expected length is > > 110 and actual length is 110 > > 2020-04-27 07:27:45.985 WARN > > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > > o.a.s.h.IndexFetcher File _1mm_Lucene50_0.tim did not match. expected > > checksum is 761408352 and actual is checksum 685575313. expected length is > > 35256 and actual length is 31644 > > 2020-04-27 07:27:45.987 WARN > > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > > o.a.s.h.IndexFetcher File _1mm.fnm did not match. expected checksum is > > 811204953 and actual is checksum 1244308773. expected length is 107460 and > > actual length is 107460 > > 2020-04-27 07:27:45.989 WARN > > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > > o.a.s.h.IndexFetcher File _1mr.fnm did not match. expected checksum is > > 3813441918 and actual is checksum 2133371346. expected length is 107460 and > > actual length is 106360 > > 2020-04-27 07:27:45.991 WARN > > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > > o.a.s.h.IndexFetcher File _1mr.fdt did not match. expected checksum is > > 4263416408 and actual is checksum 3722461284. expected length is 16698 and > > actual length is 21392 > > 2020-04-27 07:27:45.993 WARN > > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > > o.a.s.h.IndexFetcher File _1mr_Lucene50_0.doc did not match. expected > > checksum is 632422377 and actual is checksum 501799146. expected length is > > 110 and actual length is 110 > > 2020-04-27 07:27:45.994 WARN > > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > > o.a.s.h.IndexFetcher File _1mr_Lucene50_0.tim did not match. expected > > checksum is 3170165892 and actual is checksum 4152937488. expected length > > is 25090 and actual length is 32328 > > 2020-04-27 07:27:46.000 WARN > > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > > o.a.s.h.IndexFetcher File _1mr_Lucene50_0.tip did not match. expected > > checksum is 2679120655 and actual is checksum 3243853760. expected length > > is 553 and actual length is 726 > > 2020-04-27 07:27:46.001 WARN > > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > > o.a.s.h.IndexFetcher File _1mr_Lucene70_0.dvm did not match. expected > > checksum is 2991034554 and actual is checksum 1980969905. expected length > > is 321785 and actual length is 322117 > > 2020-04-27 07:27:46.004 WARN > > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > > o.a.s.h.IndexFetcher File _1mr_Lucene70_0.dvd did not match. expected > > checksum is 2117172713 and actual is checksum 4064616498. expected length > > is 3479018 and actual length is 4488695 > > 2020-04-27 07:27:46.017 WARN > > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > > o.a.s.h.IndexFetcher File _1mr.fdx did not match. expected checksum is > > 1613761705 and actual is checksum 4154864516. expected length is 106 and > > actual length is 111 > > 2020-04-27 07:27:46.019 WARN > > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > > o.a.s.h.IndexFetcher File _1mr.si did not match. expected checksum is > > 3059265880 and actual is checksum 531516038. expected length is 508 and > > actual length is 508 > > 2020-04-27 07:27:46.021 WARN > > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > > o.a.s.h.IndexFetcher File _1mx_Lucene50_0.tip did not match. expected > > checksum is 1804496408 and actual is checksum 2903373400. expected length > > is 627 and actual length is 621 > > 2020-04-27 07:27:46.023 WARN > > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > > o.a.s.h.IndexFetcher File _1mx.si did not match. expected checksum is > > 2241359534 and actual is checksum 4198491027. expected length is 508 and > > actual length is 508 > > 2020-04-27 07:27:46.024 WARN > > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > > o.a.s.h.IndexFetcher File _1mx_Lucene70_0.dvm did not match. expected > > checksum is 3097209578 and actual is checksum 1124925868. expected length > > is 312617 and actual length is 323465 > > 2020-04-27 07:27:46.027 WARN > > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > > o.a.s.h.IndexFetcher File _1mx.fnm did not match. expected checksum is > > 1909140548 and actual is checksum 2397297432. expected length is 106580 and > > actual length is 107460 > > 2020-04-27 07:27:46.029 WARN > > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > > o.a.s.h.IndexFetcher File _1mx_Lucene50_0.doc did not match. expected > > checksum is 971818792 and actual is checksum 478139506. expected length is > > 110 and actual length is 110 > > 2020-04-27 07:27:46.031 WARN > > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > > o.a.s.h.IndexFetcher File _1mx.fdt did not match. expected checksum is > > 1304288778 and actual is checksum 1544273325. expected length is 18170 and > > actual length is 17756 > > 2020-04-27 07:27:46.033 WARN > > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > > o.a.s.h.IndexFetcher File _1mx_Lucene50_0.tim did not match. expected > > checksum is 4033259458 and actual is checksum 4016593800. expected length > > is 27532 and actual length is 26636 > > 2020-04-27 07:27:46.034 WARN > > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > > o.a.s.h.IndexFetcher File _1mx.fdx did not match. expected checksum is > > 385809858 and actual is checksum 3497071885. expected length is 107 and > > actual length is 104 > > 2020-04-27 07:27:46.036 WARN > > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > > o.a.s.h.IndexFetcher File _1mx_Lucene70_0.dvd did not match. expected > > checksum is 3567656601 and actual is checksum 2800338830. expected length > > is 3622054 and actual length is 3792928 > > 2020-04-27 07:27:46.051 WARN > > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > > o.a.s.h.IndexFetcher File _1n0.fdx did not match. expected checksum is > > 2282222236 and actual is checksum 3183044106. expected length is 98 and > > actual length is 108 > > 2020-04-27 07:27:46.053 WARN > > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > > o.a.s.h.IndexFetcher File _1n0.si did not match. expected checksum is > > 150588833 and actual is checksum 3492244992. expected length is 508 and > > actual length is 508 > > 2020-04-27 07:27:46.055 WARN > > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > > o.a.s.h.IndexFetcher File _1n0_Lucene50_0.doc did not match. expected > > checksum is 1457615503 and actual is checksum 4089018914. expected length > > is 110 and actual length is 110 > > 2020-04-27 07:27:46.125 WARN > > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > > o.a.s.h.IndexFetcher File _1n0_Lucene50_0.tim did not match. expected > > checksum is 3866656431 and actual is checksum 2014845387. expected length > > is 16763 and actual length is 28481 > > 2020-04-27 07:27:46.128 WARN > > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > > o.a.s.h.IndexFetcher File _1n0_Lucene50_0.tip did not match. expected > > checksum is 3008320208 and actual is checksum 808227383. expected length is > > 419 and actual length is 659 > > 2020-04-27 07:27:46.130 WARN > > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > > o.a.s.h.IndexFetcher File _1n0_Lucene70_0.dvm did not match. expected > > checksum is 2437844680 and actual is checksum 1756584754. expected length > > is 310713 and actual length is 315009 > > 2020-04-27 07:27:46.133 WARN > > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > > o.a.s.h.IndexFetcher File _1n0_Lucene70_0.dvd did not match. expected > > checksum is 3036548497 and actual is checksum 343807338. expected length is > > 2245614 and actual length is 4011325 > > 2020-04-27 07:27:46.146 WARN > > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > > o.a.s.h.IndexFetcher File _1n0.fnm did not match. expected checksum is > > 4233343814 and actual is checksum 2984252206. expected length is 106580 and > > actual length is 107460 > > 2020-04-27 07:27:46.148 WARN > > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > > o.a.s.h.IndexFetcher File _1n0.fdt did not match. expected checksum is > > 2222747552 and actual is checksum 3771791266. expected length is 11198 and > > actual length is 18927 > > 2020-04-27 07:27:46.150 WARN > > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > > o.a.s.h.IndexFetcher File _1n2.fnm did not match. expected checksum is > > 2062419431 and actual is checksum 885929423. expected length is 107460 and > > actual length is 105632 > > 2020-04-27 07:27:46.152 WARN > > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > > o.a.s.h.IndexFetcher File _1n2_Lucene50_0.tip did not match. expected > > checksum is 1574973164 and actual is checksum 209201565. expected length is > > 516 and actual length is 138 > > 2020-04-27 07:27:46.154 WARN > > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > > o.a.s.h.IndexFetcher File _1n2.fdt did not match. expected checksum is > > 3606049617 and actual is checksum 3605002010. expected length is 14213 and > > actual length is 1590 > > 2020-04-27 07:27:46.156 WARN > > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > > o.a.s.h.IndexFetcher File _1n2_Lucene70_0.dvm did not match. expected > > checksum is 842753199 and actual is checksum 3527006266. expected length is > > 316073 and actual length is 279943 > > 2020-04-27 07:27:46.159 WARN > > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > > o.a.s.h.IndexFetcher File _1n2_Lucene50_0.doc did not match. expected > > checksum is 2130892015 and actual is checksum 2031251178. expected length > > is 110 and actual length is 110 > > 2020-04-27 07:27:46.161 WARN > > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > > o.a.s.h.IndexFetcher File _1n2_Lucene50_0.tim did not match. expected > > checksum is 1435858498 and actual is checksum 3731425121. expected length > > is 21284 and actual length is 2331 > > 2020-04-27 07:27:46.163 WARN > > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > > o.a.s.h.IndexFetcher File _1n2.si did not match. expected checksum is > > 20126402 and actual is checksum 1544152005. expected length is 508 and > > actual length is 508 > > 2020-04-27 07:27:46.165 WARN > > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > > o.a.s.h.IndexFetcher File _1n2.fdx did not match. expected checksum is > > 1651289033 and actual is checksum 1035172250. expected length is 100 and > > actual length is 86 > > 2020-04-27 07:27:46.166 WARN > > (recoveryExecutor-4-thread-697-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > > o.a.s.h.IndexFetcher File _1n2_Lucene70_0.dvd did not match. expected > > checksum is 3180610843 and actual is checksum 2027612627. expected length > > is 2818109 and actual length is 159550 > > 2020-04-27 07:27:52.815 WARN > > (recoveryExecutor-2624-thread-1-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > > o.a.s.u.UpdateLog Starting log replay > > tlog{file=/data/dnb/cloud/node1/solr/mycollection_shard14_replica_n52/data/tlog/tlog.0000000000000000972 > > refcount=2} active=true starting pos=1471697 inSortedOrder=false > > 2020-04-27 07:27:53.592 WARN > > (recoveryExecutor-2619-thread-1-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > > o.a.s.u.UpdateLog Starting log replay > > tlog{file=/data/dnb/cloud/node1/solr/mycollection_shard13_replica_n48/data/tlog/tlog.0000000000000000972 > > refcount=2} active=true starting pos=211445345 inSortedOrder=false > > 2020-04-27 07:27:56.965 WARN > > (recoveryExecutor-2597-thread-1-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard1_replica_n1 s:shard1 c:mycollection r:core_node3) > > [c:mycollection s:shard1 r:core_node3 x:mycollection_shard1_replica_n1] > > o.a.s.u.UpdateLog Log replay finished. recoveryInfo=RecoveryInfo{adds=34889 > > deletes=0 deleteByQuery=0 errors=0 positionOfStart=1404348} > > 2020-04-27 07:27:57.329 WARN > > (recoveryExecutor-2619-thread-1-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard13_replica_n48 s:shard13 c:mycollection r:core_node51) > > [c:mycollection s:shard13 r:core_node51 x:mycollection_shard13_replica_n48] > > o.a.s.u.UpdateLog Log replay finished. recoveryInfo=RecoveryInfo{adds=5597 > > deletes=0 deleteByQuery=0 errors=0 positionOfStart=211445345} > > 2020-04-27 07:28:03.537 WARN > > (recoveryExecutor-2654-thread-1-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard5_replica_n16 s:shard5 c:mycollection r:core_node19) > > [c:mycollection s:shard5 r:core_node19 x:mycollection_shard5_replica_n16] > > o.a.s.u.UpdateLog Log replay finished. recoveryInfo=RecoveryInfo{adds=35707 > > deletes=0 deleteByQuery=0 errors=0 positionOfStart=2740945} > > 2020-04-27 07:28:07.353 WARN > > (recoveryExecutor-2629-thread-1-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard15_replica_n56 s:shard15 c:mycollection r:core_node59) > > [c:mycollection s:shard15 r:core_node59 x:mycollection_shard15_replica_n56] > > o.a.s.u.UpdateLog Log replay finished. recoveryInfo=RecoveryInfo{adds=35950 > > deletes=0 deleteByQuery=0 errors=0 positionOfStart=3081052} > > 2020-04-27 07:28:19.721 WARN > > (recoveryExecutor-2624-thread-1-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard14_replica_n52 s:shard14 c:mycollection r:core_node55) > > [c:mycollection s:shard14 r:core_node55 x:mycollection_shard14_replica_n52] > > o.a.s.u.UpdateLog Log replay finished. recoveryInfo=RecoveryInfo{adds=38066 > > deletes=0 deleteByQuery=0 errors=0 positionOfStart=1471697} > > 2020-04-27 07:33:26.278 WARN > > (updateExecutor-3-thread-827-processing-n:mysolrsever.com:6010_solr) > > [c:mycollection s:shard10 r:core_node39 x:mycollection_shard10_replica_n36] > > o.a.s.c.RecoveryStrategy Stopping recovery for > > core=[mycollection_shard10_replica_n36] coreNodeName=[core_node39] > > 2020-04-27 07:33:30.370 WARN > > (recoveryExecutor-4-thread-833-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard10_replica_n36 s:shard10 c:mycollection r:core_node39) > > [c:mycollection s:shard10 r:core_node39 x:mycollection_shard10_replica_n36] > > o.a.s.h.IndexFetcher File _1mx.si did not match. expected checksum is > > 1480542754 and actual is checksum 2732545715. expected length is 437 and > > actual length is 508 > > 2020-04-27 07:33:37.918 WARN > > (recoveryExecutor-4-thread-833-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard10_replica_n36 s:shard10 c:mycollection r:core_node39) > > [c:mycollection s:shard10 r:core_node39 x:mycollection_shard10_replica_n36] > > o.a.s.h.IndexFetcher File _1mx.si did not match. expected checksum is > > 1480542754 and actual is checksum 2732545715. expected length is 437 and > > actual length is 508 > > 2020-04-27 07:33:37.945 WARN > > (recoveryExecutor-4-thread-833-processing-n:mysolrsever.com:6010_solr > > x:mycollection_shard10_replica_n36 s:shard10 c:mycollection r:core_node39) > > [c:mycollection s:shard10 r:core_node39 x:mycollection_shard10_replica_n36] > > o.a.s.h.IndexFetcher File _1me.si did not match. expected checksum is > > 204474971 and actual is checksum 3417943085. expected length is 437 and > > actual length is 546 > > 2020-04-27 07:34:07.322 WARN > > (zkConnectionManagerCallback-6-thread-1-processing-n:mysolrsever.com:6010_solr-SendThread(zoo-prd-n1:2181)) > > [ ] o.a.z.ClientCnxn Client session timed out, have not heard from > > server in 10775ms for sessionid 0x171a6fb51310008 > > 2020-04-27 07:34:07.426 WARN > > (zkConnectionManagerCallback-6-thread-1-processing-n:mysolrsever.com:6010_solr-EventThread) > > [ ] o.a.s.c.c.ConnectionManager Watcher > > org.apache.solr.common.cloud.ConnectionManager@51b6bf14 name: > > ZooKeeperConnection Watcher:myzoo01:2181,myzoo02:2181,myzoo03:2181 got > > event WatchedEvent state:Disconnected type:None path:null path: null type: > > None > > 2020-04-27 07:34:07.426 WARN > > (zkConnectionManagerCallback-6-thread-1-processing-n:mysolrsever.com:6010_solr-EventThread) > > [ ] o.a.s.c.c.ConnectionManager Watcher > > org.apache.solr.common.cloud.ConnectionManager@51b6bf14 name: > > ZooKeeperConnection Watcher:myzoo01:2181,myzoo02:2181,myzoo03:2181 got > > event WatchedEvent state:Disconnected type:None path:null path: null type: > > None > > 2020-04-27 07:34:07.426 WARN > > (zkConnectionManagerCallback-6-thread-1-processing-n:mysolrsever.com:6010_solr-EventThread) > > [ ] o.a.s.c.c.ConnectionManager zkClient has disconnected > > 2020-04-28 07:53:30.910 WARN > > (zkConnectionManagerCallback-6-thread-1-processing-n:mysolrsever.com:6010_solr-SendThread(zoo-prd-n2:2181)) > > [ ] o.a.z.ClientCnxn Client session timed out, have not heard from > > server in 10528ms for sessionid 0x171a6fb51310008 > > 2020-04-28 07:53:31.015 WARN > > (zkConnectionManagerCallback-6-thread-1-processing-n:mysolrsever.com:6010_solr-EventThread) > > [ ] o.a.s.c.c.ConnectionManager Watcher > > org.apache.solr.common.cloud.ConnectionManager@51b6bf14 name: > > ZooKeeperConnection Watcher:myzoo01:2181,myzoo02:2181,myzoo03:2181 got > > event WatchedEvent state:Disconnected type:None path:null path: null type: > > None > > 2020-04-28 07:53:31.015 WARN > > (zkConnectionManagerCallback-6-thread-1-processing-n:mysolrsever.com:6010_solr-EventThread) > > [ ] o.a.s.c.c.ConnectionManager zkClient has disconnected
