Re: Permanently Full Old Generation...

2012-11-30 Thread Walter Underwood
We are running 1.6 update 37. That was released on the same day as your version, so it should have the same bug fixes. We use these options in production, it is very stable: export CATALINA_OPTS="$CATALINA_OPTS -d64" export CATALINA_OPTS="$CATALINA_OPTS -Xms4096m -Xmx6144m" export CATALINA_OPTS=

Re: Permanently Full Old Generation...

2012-11-30 Thread Andy Kershaw
We are currently operating at reduced load which is why the ParNew collections are not a problem. I don't know how long they were taking before though. Thanks for the warning about index formats. Our JVM is: Java(TM) SE Runtime Environment (build 1.7.0_09-b05) Java HotSpot(TM) 64-Bit Server VM (b

Re: Permanently Full Old Generation...

2012-11-29 Thread Walter Underwood
Several suggestions. 1. Adjust the traffic load for about 75% CPU. When you hit 100%, you are already in an overload state and the variance of the response times goes way up. You'll have very noisy benchmark data. 2. Do not force manual GCs during a benchmark. 3. Do not force merge (optimise).

Re: Permanently Full Old Generation...

2012-11-29 Thread Shawn Heisey
On 11/29/2012 10:44 AM, Andy Kershaw wrote: Annette is away until Monday so I am looking into this in the meantime. Looking at the times of the Full GC entries at the end of the log, I think they are collections we started manually through jconsole to try and reduce the size of the old generation

Re: Permanently Full Old Generation...

2012-11-29 Thread Andy Kershaw
Thanks for responding Shawn. Annette is away until Monday so I am looking into this in the meantime. Looking at the times of the Full GC entries at the end of the log, I think they are collections we started manually through jconsole to try and reduce the size of the old generation. This only seem

RE: Permanently Full Old Generation...

2012-11-29 Thread Shawn Heisey
> My jvm settings: > > > -Xmx8192M -Xms8192M -XX:+CMSScavengeBeforeRemark -XX:NewRatio=2 > -XX:+CMSParallelRemarkEnabled -XX:+UseParNewGC -XX:+UseConcMarkSweepGC > -XX:+AggressiveOpts -XX:CMSInitiatingOccupancyFraction=70 > -XX:+UseCMSInitiatingOccupancyOnly -XX:-CMSIncrementalPacing > -XX:CMSIncre

Re: Permanently Full Old Generation...

2012-11-28 Thread Shawn Heisey
On 11/28/2012 9:06 AM, Annette Newton wrote: We are seeing strange gc behaviour after running solr cloud under quite heavy insert load for a period of time. The old generation becomes full and no amount of garbage collection will free up the memory. I have attached a memory profile, as you can

RE: Permanently Full Old Generation...

2012-11-28 Thread Annette Newton
lass/0x73ae8db40> 3456 96768 class [[C <http://localhost:7000/class/0x73b60cf18> This sample of the heap dump I took when we encountered the problem previously.. -Original Message- From: Jack Krupansky [mailto:j...@basetechnology.com] Sent: 28 Novemb

Re: Permanently Full Old Generation...

2012-11-28 Thread Jack Krupansky
Have you done a Java heap dump to see what the most common objects are? -- Jack Krupansky From: Annette Newton Sent: Wednesday, November 28, 2012 11:06 AM To: solr-user@lucene.apache.org Cc: Andy Kershaw Subject: Permanently Full Old Generation... Hi, I’m hoping someone can help me with