On 9/13/2013 12:03 AM, Per Steffensen wrote:
What is it that will fill my heap? I am trying to avoid the FieldCache. For now, I am actually not doing any searches - focus on indexing for now - and certainly not group/facet/sort searches that will use the FieldCache.
I don't know what makes up the heap when you have lots of documents. I am not really using any RAM hungry features and I wouldn't be able to get away with a 4GB heap on my Solr servers. Uncollectable (and collectable) RAM usage is heaviest during indexing. I sort on one or two fields and we don't use facets.
Here's a screenshot of my index status page showing how big my indexes are on each machine, it's a couple of months old now. These machines have a 6GB heap, and I don't dare make it any smaller, or I'll get OOM errors during indexing. They have 64GB total RAM.
https://dl.dropboxusercontent.com/u/97770508/statuspagescreenshot.png
More RAM will probably help, but only for a while. I want billions of documents in my collections - and also on each machine. Currently we are aiming 15 billion documents per month (500 million per day) and keep at least two years of data in the system. Currently we use one collection for each month, so when the system has been running for two years it will be 24 collections with 15 billion documents each. Indexing will only go on in the collection corresponding to the "current" month, but searching will (potentially) be across all 24 collections. The documents are very small. I know that 6 machines will not do in the long run - currently this is only testing - but number of machines should not be higher than about 20-40. In general it is a problem if Solr/Lucene will not perform fairly well if data does not fit RAM - then it cannot really be used for "big data". I would have to buy hundreds or even thousands of machines with 64GB+ RAM. That is not realistic.
To lower your overall RAM requirements, use SSD, and store as little data as possible - only the id used to retrieve data from another source, ideally. That will lower your RAM requirements. You'll probably still want 10-25% of your index size for the disk cache. With regular disks, that's 50-100%.
Put your OS and Solr itself on regular disks in RAID1 and your Solr data on the SSD. Due to the eventual decay caused by writes, SSD will eventually die, so be ready for SSD failures to take out shard replicas. So far I'm not aware of any RAID solutions that offer TRIM support, and without TRIM support, an SSD eventually has performance problems. Without RAID, a failure will take out that replica. That's one of the points of SolrCloud - having replicas so single failures don't bring down your index.
If you can't use SSD or get tons of RAM, you're going to have performance problems. Solr (and any other Lucene-based search product) does really well with super-large indexes if you have the system resources available. If you don't, it sucks.
Thanks, Shawn
