Re: Lots of deletions results in death by GC

2014-02-06 Thread Ben Hood
On Wed, Feb 5, 2014 at 2:52 AM, srmore wrote: > Dropped messages are the sign that Cassandra is taking heavy that's the load > shedding mechanism. I would love to see some sort of back-pressure > implemented. +1 for back pressure in general with Cassandra

Re: Lots of deletions results in death by GC

2014-02-05 Thread Benedict Elliott Smith
g and fixed it. Now > I just need to figure out where to go from here: do I wait, use the dev > branch or work around. > > Robert > > From: Benedict Elliott Smith > Reply-To: > Date: Wednesday, February 5, 2014 at 8:32 AM > > To: > Subject: Re: Lots of deletions r

Re: Lots of deletions results in death by GC

2014-02-05 Thread Robert Wille
: Date: Wednesday, February 5, 2014 at 8:32 AM To: Subject: Re: Lots of deletions results in death by GC I believe there is a bug, and I have filed a ticket for it: https://issues.apache.org/jira/browse/CASSANDRA-6655 I will have a patch uploaded shortly, but it's just missed the 2.0.5 re

Re: Lots of deletions results in death by GC

2014-02-05 Thread Benedict Elliott Smith
ode. Okay, not >> permanently, but until a manual flush occurs. >> >> If anyone has any further thoughts, I'd love to hear them. I'm quite at >> the end of my rope. >> >> Thanks in advance >> >> Robert >> >> From: Nate McC

Re: Lots of deletions results in death by GC

2014-02-05 Thread Robert Wille
Yes. It¹s kind of an unusual workload. An insertion phase followed by a deletion phase, generally not overlapping. From: Benedict Elliott Smith Reply-To: Date: Tuesday, February 4, 2014 at 5:29 PM To: Subject: Re: Lots of deletions results in death by GC Is it possible you are generating

Re: Lots of deletions results in death by GC

2014-02-04 Thread srmore
ove to hear them. I'm quite at > the end of my rope. > > Thanks in advance > > Robert > > From: Nate McCall > Reply-To: > Date: Saturday, February 1, 2014 at 9:25 AM > To: Cassandra Users > Subject: Re: Lots of deletions results in death by GC > > What

Re: Lots of deletions results in death by GC

2014-02-04 Thread Benedict Elliott Smith
> > Thanks in advance > > Robert > > From: Nate McCall > Reply-To: > Date: Saturday, February 1, 2014 at 9:25 AM > To: Cassandra Users > Subject: Re: Lots of deletions results in death by GC > > What's the output of 'nodetool tpstats' while this is

Re: Lots of deletions results in death by GC

2014-02-04 Thread Robert Coli
On Tue, Feb 4, 2014 at 4:10 PM, Robert Wille wrote: > I would almost think that this is caused by high load on the server, but > I've never seen CPU utilization go above about two of my eight available > cores. If high load triggers this problem, then that is very disconcerting. > That means that

Re: Lots of deletions results in death by GC

2014-02-04 Thread Robert Wille
Robert From: Nate McCall Reply-To: Date: Saturday, February 1, 2014 at 9:25 AM To: Cassandra Users Subject: Re: Lots of deletions results in death by GC What's the output of 'nodetool tpstats' while this is happening? Specifically is Flush Writer "All time blocke

Re: Lots of deletions results in death by GC

2014-02-01 Thread Nate McCall
What's the output of 'nodetool tpstats' while this is happening? Specifically is Flush Writer "All time blocked" increasing? If so, play around with turning up memtable_flush_writers and memtable_flush_queue_size and see if that helps. On Sat, Feb 1, 2014 at 9:03 AM, Robert Wille wrote: > A few

Lots of deletions results in death by GC

2014-02-01 Thread Robert Wille
A few days ago I posted about an issue I¹m having where GC takes a long time (20-30 seconds), and it happens repeatedly and basically no work gets done. I¹ve done further investigation, and I now believe that I know the cause. If I do a lot of deletes, it creates memory pressure until the memtables