Do you have a lot of compactions going on? It sounds like you might've built up a huge backlog. Is your throttling configured properly?
> On 11 May 2017, at 18:50, varun saluja <saluj...@gmail.com> wrote: > > Hi Experts, > > Seeking your help on a production issue. We were running high write > intensive job on our 3 node cassandra cluster V 2.1.7. > > TPS on nodes were high. Job ran for more than 2 days and thereafter, loadavg > on 1 of the node increased to very high number like loadavg : 29. > > System log reports: > > INFO [ScheduledTasks:1] 2017-05-11 22:11:04,466 MessagingService.java:888 - > 839 MUTATION messages dropped in last 5000ms > INFO [ScheduledTasks:1] 2017-05-11 22:11:04,466 MessagingService.java:888 - > 2 READ messages dropped in last 5000ms > INFO [ScheduledTasks:1] 2017-05-11 22:11:04,466 MessagingService.java:888 - > 1 REQUEST_RESPONSE messages dropped in last 5000ms > > The job was stopped due to heavy load. But sill after 12 hours , we can see > mutation drops messages and sudden increase on avgload > > Are these hintedhandoff mutations? Can we stop these. > Strangely this behaviour is seen only on 2 nodes. Node 1 does not show any > load or any such activity. > > Due to heavy load and GC , there are intermittent gossip failures among node. > Can you someone Please help. > > PS: Load job was stopped on cluster. Everything ran fine for few hours and > and Later issue started again like mutation messages drops. > > Thanks and Regards, > Varun Saluja > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > For additional commands, e-mail: dev-h...@cassandra.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org For additional commands, e-mail: dev-h...@cassandra.apache.org
