On 10/21/2013 2:11 PM, neoman wrote: > We are using solr 4.4 version production with 4 shards. This is our memory > settings. > -d64 -server -Xms8192m -Xmx12288m -XX:MaxPermSize=256m \ > -XX:NewRatio=1 -XX:SurvivorRatio=6 \ > -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode > -XX:CMSIncrementalDutyCycleMin=0 \ > -XX:CMSIncrementalDutyCycle=10 -XX:+CMSIncrementalPacing \ > -XX:+PrintTenuringDistribution -XX:+PrintGCDetails -XX:+PrintGCTimeStamps > -XX:+PrintHeapAtGC \ > -XX:+CMSClassUnloadingEnabled -XX:+DisableExplicitGC \ > -XX:+UseLargePages \ > -XX:+UseParNewGC \ > -XX:ConcGCThreads=10 \ > -XX:ParallelGCThreads=10 \ > -XX:MaxGCPauseMillis=30000 \ > I notice in production that, the old generation becomes full and no amount > of garbage collection will free up the memory > This is similar to the issue discussed in this link. > http://grokbase.com/t/lucene/solr-user/12bwydq5jr/permanently-full-old-generation > Did anyone have this problem? Can you please point anything wrong with the > GC configuration?
Your GC configuration is very likely fine. There are two possibilities that seem likely to me, and it might be a combination of both: 1) Your index size, configuration, and/or query characteristics require lots of memory. 2) You are experiencing the SolrCloud update deadlocks that have been become fairly commonplace, and the blocked threads have resulted in too much memory being used. For the first problem, you can either increase your max heap or figure out how to reduce memory requirements. Can you share your solrconfig.xml and details about your index, like number of documents, total index size, index size per shard replica, number of servers, replication factor, total RAM installed per server, etc? http://wiki.apache.org/solr/SolrPerformanceProblems#Java_Heap For the second problem, upgrading to a newer version may be the only option. Version 4.5.0 has made the situation better, but some users are still having problems. Version 4.5.1 is due out soon, but some of the fixes might require running a dev 4.6 snapshot. I think there might be some deadlock issues that don't have fixes committed yet. Thanks, Shawn
