Hi Hitesh, This list is for conversations regarding Cassandra development. Can you subscribe and post this to us...@cassandra.apache.org instead? You will get a much wider audience when you do.
On Tue, Apr 10, 2018 at 10:45 PM, hitesh dua <hiteshd...@gmail.com> wrote: > Hi, > My Compression strategy in Production was *LZ4 Compression. *But I modified > it to *Deflate *using alter command > > For compression change, we have to use *nodetool Upgradesstables *to > forcefully upgrade the compression strategy on all sstables > > But once upgradesstabloes command completed on all the 5 nodes in the > cluster, My requests started to fail, both read and write > > Replication Factor - 3 > Read Consistency - 1 > Write Consistency - 1 > FYI - I am also using lightweight transaction which uses PAXOS > Cassandra Version 3.10 > > I am now facing Following Errors in my debug.log file and some of my > requests have started to fail : > > Debug.log > > ERROR [ReadRepairStage:82952] 2018-04-09 19:05:20,669 >>> CassandraDaemon.java:229 - Exception in thread >>> Thread[ReadRepairStage:82952,5,main] >> >> org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out >>> - received only 0 responses. >> >> at >> org.apache.cassandra.service.DataResolver$RepairMergeListener.close(DataResolver.java:171) >>> ~[apache-cassandra-3.10.jar:3.10] >> >> at org.apache.cassandra.db.partitions.UnfilteredPartitionIterat >>> ors$2.close(UnfilteredPartitionIterators.java:182) >>> ~[apache-cassandra-3.10.jar:3.10] >> >> at org.apache.cassandra.db.transform.BaseIterator.close(BaseIterator.java:82) >>> ~[apache-cassandra-3.10.jar:3.10] >> >> at >> org.apache.cassandra.service.DataResolver.compareResponses(DataResolver.java:89) >>> ~[apache-cassandra-3.10.jar:3.10] >> >> at org.apache.cassandra.service.AsyncRepairCallback$1.runMayThr >>> ow(AsyncRepairCallback.java:50) ~[apache-cassandra-3.10.jar:3.10] >> >> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) >>> ~[apache-cassandra-3.10.jar:3.10] >> >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >>> ~[na:1.8.0_144] >> >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >>> ~[na:1.8.0_144] >> >> at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$ >>> threadLocalDeallocator$0(NamedThreadFactory.java:79) >>> ~[apache-cassandra-3.10.jar:3.10] >> >> at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_144] >> >> DEBUG [ReadRepairStage:82953] 2018-04-09 19:05:22,932 >>> ReadCallback.java:242 - Digest mismatch: >> >> org.apache.cassandra.service.DigestMismatchException: Mismatch for key >>> DecoratedKey(-2666936192316364820, 5756f5b8e7b341afa22cef22c5d33260) >>> (d29a0e2a05f81315f0945dee5a210060 vs d41d8cd98f00b204e9800998ecf8427e) >> >> at >> org.apache.cassandra.service.DigestResolver.compareResponses(DigestResolver.java:92) >>> ~[apache-cassandra-3.10.jar:3.10] >> >> at >> org.apache.cassandra.service.ReadCallback$AsyncRepairRunner.run(ReadCallback.java:233) >>> ~[apache-cassandra-3.10.jar:3.10] >> >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >>> [na:1.8.0_144] >> >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >>> [na:1.8.0_144] >> >> at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$ >>> threadLocalDeallocator$0(NamedThreadFactory.java:79) >>> [apache-cassandra-3.10.jar:3.10] >> >> at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_144] >> >> INFO [HintsDispatcher:767] 2018-04-09 19:05:24,874 >>> HintsDispatchExecutor.java:283 - Finished hinted handoff of file >>> 68c7c130-6cf8-4864-bde8-1819f238045c-1523315072851-1.hints to endpoint >>> 68c7c130-6cf8-4864-bde8-1819f238045c, partially >> >> DEBUG [ReadRepairStage:82950] 2018-04-09 19:05:24,932 >>> DataResolver.java:169 - Timeout while read-repairing after receiving all 1 >>> data and digest responses >> >> ERROR [ReadRepairStage:82950] 2018-04-09 19:05:24,933 >>> CassandraDaemon.java:229 - Exception in thread >>> Thread[ReadRepairStage:82950,5,main] >> >> org.apache.cassandra.exceptions.ReadTimeoutException: Operation timed out >>> - received only 0 responses. >> >> at >> org.apache.cassandra.service.DataResolver$RepairMergeListener.close(DataResolver.java:171) >>> ~[apache-cassandra-3.10.jar:3.10] >> >> at org.apache.cassandra.db.partitions.UnfilteredPartitionIterat >>> ors$2.close(UnfilteredPartitionIterators.java:182) >>> ~[apache-cassandra-3.10.jar:3.10] >> >> at org.apache.cassandra.db.transform.BaseIterator.close(BaseIterator.java:82) >>> ~[apache-cassandra-3.10.jar:3.10] >> >> at >> org.apache.cassandra.service.DataResolver.compareResponses(DataResolver.java:89) >>> ~[apache-cassandra-3.10.jar:3.10] >> >> at org.apache.cassandra.service.AsyncRepairCallback$1.runMayThr >>> ow(AsyncRepairCallback.java:50) ~[apache-cassandra-3.10.jar:3.10] >> >> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) >>> ~[apache-cassandra-3.10.jar:3.10] >> >> at >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >>> ~[na:1.8.0_144] >> >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >>> ~[na:1.8.0_144] >> >> at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$ >>> threadLocalDeallocator$0(NamedThreadFactory.java:79) >>> ~[apache-cassandra-3.10.jar:3.10] >> >> at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_144] >> >> > nodetool info shows > > Gossip active : true >> >> Thrift active : false >> >> Native Transport active: true >> >> Load : 280.43 GiB >> >> Generation No : 1514537104 >> >> Uptime (seconds) : 8810363 >> >> Heap Memory (MB) : 1252.06 / 3970.00 >> >> Off Heap Memory (MB) : 573.33 >> >> Data Center : dc1 >> >> Rack : rack1 >> >> *Exceptions : 18987* >> >> Key Cache : entries 351612, size 99.86 MiB, capacity 100 MiB, >>> 11144584 hits, 21126425 requests, 0.528 recent hit rate, 14400 save period >>> in seconds >> >> > Hope anyone can help me with this. > > Thanks, > Hitesh dua > hiteshd...@gmail.com --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org For additional commands, e-mail: dev-h...@cassandra.apache.org
