Thanks for responding Shawn. Annette is away until Monday so I am looking into this in the meantime. Looking at the times of the Full GC entries at the end of the log, I think they are collections we started manually through jconsole to try and reduce the size of the old generation. This only seemed to have an effect when we reloaded the core first though.
It is my understanding that the eden size is deliberately smaller to keep the ParNew collection time down. If it takes too long then the node is flagged as down. On 29 November 2012 15:28, Shawn Heisey <s...@elyograg.org> wrote: > > My jvm settings: > > > > > > -Xmx8192M -Xms8192M -XX:+CMSScavengeBeforeRemark -XX:NewRatio=2 > > -XX:+CMSParallelRemarkEnabled -XX:+UseParNewGC -XX:+UseConcMarkSweepGC > > -XX:+AggressiveOpts -XX:CMSInitiatingOccupancyFraction=70 > > -XX:+UseCMSInitiatingOccupancyOnly -XX:-CMSIncrementalPacing > > -XX:CMSIncrementalDutyCycle=75 > > > > I turned off IncrementalPacing, and enabled > > CMSInitiatingOccupancyFraction, > > after issues with nodes being reported as down due to large Garbage > > collection pauses. The problem with the memory profile was visible > before > > the drop down to 1.2GB (this was when I reloaded the core), my concern > was > > that the collection of the old generation didn't seem to free any of the > > heap, and we went from occasionally collecting to always collecting the > > old > > gen. > > > > Please see the attached gc log. > > I am on the train for my morning commute, so I have some time, but no > access to the log or graph. > > Confession time: GC logs make me go glassy eyed and babble incoherently, > but I did take a look at it. I saw 18 CMS collections and three entries > near the end that saif Full GC. It looks like these collections take 6 to > 8 seconds. That is pretty nasty, but probably unavoidable, so the goal is > to make them happen extremely infrequently - do young generation > collections instead. > > The thing that seems to make GC less of a problem for solr is maximizing > the young generation memory pool. Based on the available info, I would > start with making NewRatio 1 instead of 2. This will increase the eden > size and decrease the old gen size. You may even want to use an explicit > -Xmn of 6144. If that doesn't help, you might actually need 6GB or so of > old gen heap, so try increasing the overall heap size to 9 or 10 GB and > going back to a NewRatio of 2. > > Thanks, > Shawn >
