On 10/29/2012 3:26 PM, shreejay wrote:
I am trying to run two SolrCloud with 3 and 2 shards respectively (lets say Cloud3shards and Clouds2Shards). All servers are identical with 18GB Ram (16GB assigned for Java).
This bit right here sets off warning bells right away. You're only leaving 2GB of RAM for the OS to cache your index, which you later say is 50GB. It's impossible for me to give you a precise figure, but I would expect that with an index that size, you'd want to have at least 20GB of free memory, and if you can have 50GB or more of free memory after the OS and Java take their chunk, Solr would have truly excellent performance. As it is now, your performance will be terrible, which probably explains all your issues.
It seems highly unlikely that you would have queries complex enough that you actually do need to allocate 16GB of RAM to Java. Also, requesting large numbers of documents (the 5000 and 20000 numbers you mentioned) is slow, and compounded in a cloud (distributed) index. Solr is optimized for a small number of results.
First recommendation for fixing things: get more memory. 32GB would be a good starting point, 64GB would be better, so that the entire index will be able to fit in OS cache memory. If you expect your index to grow at all, plan accordingly.
Second recommendation, whether or not you get more actual memory: Lower the memory that Java is using, and configure some alternate memory management options for Java. Solr does have caching capability, but it is highly specialized. For general index caching, the OS does a far better job, and it needs free memory in order to accomplish it. Here's some commandline options for Java that I passed along to someone else on this list:
-Xmx4096M -Xms4096M -XX:NewRatio=1 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled
http://www.petefreitag.com/articles/gctuning/