Re: Elevated response times from all nodes in a data center at the same time.

2019-10-16 Thread Reid Pinchback
;user@cassandra.apache.org" Date: Wednesday, October 16, 2019 at 11:36 AM To: "user@cassandra.apache.org" Cc: Bill Walters Subject: Re: Elevated response times from all nodes in a data center at the same time. Message from External Sender It's possible the queries you're norm

Re: Elevated response times from all nodes in a data center at the same time.

2019-10-16 Thread Jon Haddad
nk about anything else, I hope some of those ideas will > help you diagnose the problem. Once it is diagnosed, we should be able to > reason about how we can fix it. > > C*heers, > --- > Alain Rodriguez - al...@thelastpickle.com > France / Spain > > The

Re: Elevated response times from all nodes in a data center at the same time.

2019-10-16 Thread Alain RODRIGUEZ
have compactions fronting the p99’s. If so, then > go back to looking at the I/O. Disbelieve any metrics not captured at a > high resolution for a time window around the compactions, like 100ms. You > could be hitting I/O stalls where reads are blocked by the flushing of > write

Re: Elevated response times from all nodes in a data center at the same time.

2019-10-15 Thread Reid Pinchback
writes. It’s short-lived when it happens, and per-minute metrics won’t provide breadcrumbs. From: Bill Walters Date: Monday, October 14, 2019 at 7:10 PM To: Subject: Elevated response times from all nodes in a data center at the same time. Hi Everyone, Need some suggestions regarding a peculiar

Elevated response times from all nodes in a data center at the same time.

2019-10-14 Thread Bill Walters
Hi Everyone, Need some suggestions regarding a peculiar issue we started facing in our production cluster for the last couple of days. Here are our Production environment details. AWS Regions: us-east-1 and us-west-2. Deployed over 3 availability zone in each region. No of Nodes: 24 Data Centers