On 2/14/2014 5:28 AM, Annette Newton wrote: > Solr Version: 4.3.1 > Number Shards: 10 > Replicas: 1 > Heap size: 15GB > Machine RAM: 30GB > Zookeeper timeout: 45 seconds > > We are continuing the fight to keep our solr setup functioning. As a > result of this we have made significant changes to our schema to reduce the > amount of data we write. I setup a new cluster to reindex our data, > initially I ran the import with no replicas, and achieved quite impressive > results. Our peak was 60,000 new documents per minute, no shard loses, no > outages due to garbage collection (which is an issue we see in production), > at the end of the load the index stood at 97,000,000 documents and 20GB per > shard. During the highest insertion rate I would say that querying > suffered, but that is not of concern right now.
Solr 4.3.1 has a number of problems when it comes to large clouds. Upgrading to 4.6.1 would be strongly advisable, but that's only something to try after looking into the rest of what I have to say. If I read what you've written correctly, you are running all this on one machine. To put it bluntly, this isn't going to work well unless you put a LOT more memory into that machine. For good performance, Solr relies on the OS disk cache, because reading from the disk is VERY expensive in terms of time. The OS will automatically use RAM that's not being used for other purposes for the disk cache, so that it can avoid reading off the disk as much as possible. http://wiki.apache.org/solr/SolrPerformanceProblems Below is a summary of what that Wiki page says, with your numbers as I understand them. If I am misunderstanding your numbers, then this advice may need adjustment. Note that when I see "one replica" I take that to mean replicationFactor=1, so there is only one copy of the index. If you actually mean that you have *two* copies, then you have twice as much data as I've indicated below, and your requirements will be even larger: With ten shards that are each 20GB in size, your total index size is 200GB. With 15 GB of heap, your ideal memory size for that server would be 215GB -- the 15GB heap plus enough extra to fit the entire 200GB index into RAM. In reality you probably don't need that much, but it's likely that you would need at least half the index to fit into RAM at any one moment, which adds up to 115GB. If you're prepared to deal with moderate-to-severe performance problems, you **MIGHT** be able to get away with only 25% of the index fitting into RAM, which still requires 65GB of RAM, but with SolrCloud, such performance problems usually mean that the cloud won't be stable, so it's not advisable to even try it. One of the bits of advice on the wiki page is to split your index into shards and put it on more machines, which drops the memory requirements for each machine. You're already using a multi-shard SolrCloud, so you probably just need more hardware. If you had one 20GB shard on a machine with 30GB of RAM, you could probably use a heap size of 4-8GB per machine and have plenty of RAM left over to cache the index very well. You could most likely add another 50% to the index size and still be OK. Thanks, Shawn
