Re: RC1 - Counters

2021-05-05 Thread Joe Obernberger
You are correct.  Thank you!  One of the machines in the cluster is a Centos 8 box that doesn't come with NTP - they only included chronyd...which doesn't seem to work with our existing NTP server.  Ugh,  I finally gave up and just set the clock manually and it now works. Cheers! -Joe On 5/5/

Re: RC1 - Counters

2021-05-05 Thread Bowen Song
This sounds like the clock on your Cassandra servers are not in sync. Can you please ensure all Cassandra servers have their clock synced (usually via NTP) and retry this? On 05/05/2021 14:42, Joe Obernberger wrote: Want to add - I am seeing this in the log: INFO  [ScheduledTasks:1] 2021-05-0

Re: RC1 - Counters

2021-05-05 Thread Jeff Jirsa
NTP is running and clocks are sync'd? No large pauses? On Wed, May 5, 2021 at 6:44 AM Joe Obernberger wrote: > Want to add - I am seeing this in the log: > INFO [ScheduledTasks:1] 2021-05-05 09:36:05,022 > MessagingMetrics.java:206 - COUNTER_MUTATION_RSP messages were dropped > in last 5000 ms:

Re: RC1 - Counters

2021-05-05 Thread Joe Obernberger
Want to add - I am seeing this in the log: INFO  [ScheduledTasks:1] 2021-05-05 09:36:05,022 MessagingMetrics.java:206 - COUNTER_MUTATION_RSP messages were dropped in last 5000 ms: 0 internal and 1 cross node. Mean internal dropped latency: 0 ms and Mean cross-node dropped latency: 21356 ms -j

RC1 - Counters

2021-05-05 Thread Joe Obernberger
I'm seeing some odd behavior with RC1 and counters - from cqlsh: cqlsh> select * from doc.seq;  id   | doccount --+--    DS |    1  DS_1 |  844 (2 rows) cqlsh> update doc.seq set doccount=doccount+1 where id='DS_1'; OperationTimedOut: errors={'172.16.100.208:9042': 'Client r