Recommendations for Tuning BloomFilter values

Hello, I am trying to understand pros and cons of tuning(increasing) the bloom filter value, to optimize the read performance. are there any guidelines on increasing the value. I am using LeveledCompactionStrategy, so the default bloom_filter_fp_chance is 0.1

Re: disabling bloomfilter not working? memory numbers don't add up?

ser@cassandra.apache.org>" mailto:user@cassandra.apache.org>> Date: Thursday, February 28, 2013 11:30 AM To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" mailto:user@cassandra.apache.org>> Subject: Re: disabling bloomfilter not working? memory numbers d

Re: disabling bloomfilter not working? memory numbers don't add up?

> 1. Can I stop the node, delete the *Filter.db files and restart the node(is > this safe)??? No. > 2. Why do I have 5 gig being eaten up by cassandra? "nodetool info" memory > 5.2Gig, key cache:11 meg and row cache 0 bytes. All bloomfilters are also > small <1meg. If this is the Heap memo

Re: disabling bloomfilter not working? memory numbers don't add up?

H, my upgrade completed and then I added node back in and ran my repair. What is weird is that my nreldata column family still shows 156Meg of memory still(down from 2 gig though!!) in use and a false positive ratio of .99576 when I have the filter completely disabled(ie. Set to 1.0). I se

Re: disabling bloomfilter not working? or did I do this wrong?

Hmmm, ok, that makes sense. I suspect the same is true with leveled compaction as well? Thanks, Dean On 2/25/13 6:47 AM, "Edward Capriolo" wrote: >Mostly but not 100%. You have a bloom filter for each sstable, so >"going to disk" means finding the row in each sstable if you end up >skipping so

Re: disabling bloomfilter not working? or did I do this wrong?

pickle.com>> > Reply-To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" > mailto:user@cassandra.apache.org>> > Date: Sunday, February 24, 2013 7:09 PM > To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" > mailto:

Re: disabling bloomfilter not working? or did I do this wrong?

.org>" mailto:user@cassandra.apache.org>> Date: Sunday, February 24, 2013 7:09 PM To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" mailto:user@cassandra.apache.org>> Subject: Re: disabling bloomfilter not working? or did I do this wrong? Yeah, disabling compl

Re: disabling bloomfilter not working? or did I do this wrong?

Yeah, disabling completely is probably not great. There is some wriggle room between disabled and "less memory" Did I link to this bloom filter calculator ? http://hur.st/bloomfilter also https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/utils/BloomCalcula

Re: disabling bloomfilter not working? or did I do this wrong?

> 5 hour wait left for my upgradesstables(waited 4 hours already). I will > check the bloomfilter after that. > > Out of curiosity, if I had much wider rows (ie. < 900k) per row, will > compaction run faster(e…upgradesstables) at all or would it basically > run at the same

Re: disabling bloomfilter not working? or did I do this wrong?

Thanks, but I found out it is still running. It looks like I have about a 5 hour wait left for my upgradesstables(waited 4 hours already). I will check the bloomfilter after that. Out of curiosity, if I had much wider rows (ie. < 900k) per row, will compaction run faster(er

Re: disabling bloomfilter not working? or did I do this wrong?

> Bloom Filter Space Used: 2318392048 Just to be sane do a quick check of the -Filter.db files on disk for this CF. If they are very small try a restart on the node. > Number of Keys (estimate): 1249133696 Hey a billion rows on a node, what an age we live in :) Cheers - Aaron M

disabling bloomfilter not working? or did I do this wrong?

So in the cli, I ran update column family nreldata with bloom_filter_fp_chance=1.0; Then I ran nodetool upgradesstables databus5 nreldata; But my bloom filter size is still around 2gig(and I want to free up this heap) According to nodetool cfstats command… Column Family: nreldata SSTable

Re: BloomFilter

d in the JVM heap, from 1.2 onwards they are stored off heap. > 3) What is the ratio of the RAM/Disk per node ? What is the max disk size > recommended for 1 node ? If I have 10 TB of data per node, how much RAM will > the bloomfilter consume ? If you are using a spinning disk (HDD)

BloomFilter

) What is the ratio of the RAM/Disk per node ? What is the max disk size recommended for 1 node ? If I have 10 TB of data per node, how much RAM will the bloomfilter consume ? Thanks, kanwar

Re: add bloomfilter results to nodetool?

On Tue, Oct 11, 2011 at 12:19 PM, Yang wrote: > I find the info about bloomfilter very helpful, could we add that to NodeCmd ? Feel free to create a ticket and tag it 'lhf' -Brandon

add bloomfilter results to nodetool?

I find the info about bloomfilter very helpful, could we add that to NodeCmd ? Thanks Yang

Re: BloomFilter is taking too much memory

More insight for this sstable: the ArrayList for IndexSummary has 644195 entries, so total number of entries for this sstable is: 644195*128=~82mil. The problem is that the total bits for its BloomFilter (long[19400551] inside BitSet) is 19400551*64=1241635264, which means each key is taking

Re: BloomFilter is taking too much memory

BloomFilter is not redundant, because it stores information about _all_ keys while the index summary stores every 1/128 key. On Tue, May 4, 2010 at 3:47 PM, Weijun Li wrote: > Hello, > > We stored about 47mil keys in one Cassandra node and what a memory dump > shows for one of the S

BloomFilter is taking too much memory

Hello, We stored about 47mil keys in one Cassandra node and what a memory dump shows for one of the SStableReader: SSTableReader: 386MB. Among this 386MB, IndexSummary takes about 231MB but BloomFilter takes 155MB with an embedded huge array long[19.4mil]. It seems that BloomFilter is