Re: Long GC due to promotion failures

2014-01-22 Thread Lee Mighdoll
On Wed, Jan 22, 2014 at 11:35 AM, John Watson wrote: > I thought PrintFLSStatistics was necessary for determining heap > fragmentation? Or is it possible to see that without it as well? > I've found that easier parsing is more important than tracking indicators of fragmentation. Perm-Gen stays

Re: Long GC due to promotion failures

2014-01-22 Thread John Watson
LCS does create a lot of SSTables unfortunately. The nodes are keeping up on compactions though. This started after starting to read from a CF that has tombstones in its rows. What's even more concerning, is it's continuing even after stopping reads and dropping that CF. On Wed, Jan 22, 2014 at

Re: Long GC due to promotion failures

2014-01-22 Thread John Watson
I thought PrintFLSStatistics was necessary for determining heap fragmentation? Or is it possible to see that without it as well? Perm-Gen stays steady, but I'll enable it anyway to see if it has any affect. Thanks, John On Wed, Jan 22, 2014 at 8:34 AM, Lee Mighdoll wrote: > I don't recommend P

Re: Long GC due to promotion failures

2014-01-22 Thread Lee Mighdoll
I don't recommend PrintFLSStatistics=1, it makes the gc logs hard to mechanically parse. Because of that, I can't easily tell whether you're in the same situation we found. But just in case, try setting +CMSClassUnloadingEnabled. There's an issue related to JMX in DSE that prevents effective old

Re: Long GC due to promotion failures

2014-01-22 Thread Jason Wee
SSTable count: 365 Your sstable counts are too many... don't know what is the best count should be but for my experience, anything below 20 are good. Is your compaction running? I read on a few blog on how should we read cfhistograms, but never really understood fully. Anyone care to explain usi

Long GC due to promotion failures

2014-01-21 Thread John Watson
Pretty reliable, at some point, nodes will have super long GCs. Followed by https://issues.apache.org/jira/browse/CASSANDRA-6592 Lovely log messages: 9030.798: [ParNew (0: promotion failure size = 4194306) (2: promotion failure size = 4194306) (4: promotion failure size = 4194306) (promotion