On 8/14/14, 11:22 AM, "Shawn Heisey" <s...@elyograg.org> wrote:
>On 8/14/2014 7:46 AM, Scott Rankin wrote: >> I¹m running a Solr setup and am getting occasional periods where memory >> usage and GC just spike out of nowhere (unrelated to traffic). I¹m >>hoping >> someone can shed some light. Here¹s the setup: >> >> - Solr 4.3.1, Oracle JDK 1.7.0_51 64 bit on CentOS 6.5 >> - We have 2 Solr servers, one acting as a master that receives all the >> writes, and one that is replicating from the master and handles all the >> reads >> - JVM parameters are: >> >> -Djava.io.tmpdir=/usr/local/solr/solr-tomcat/temp >> -Dcatalina.home=/usr/local/solr/solr-tomcat >> -Dcatalina.base=/usr/local/solr/solr-tomcat >> -Djava.endorsed.dirs=/usr/local/solr/solr-tomcat/endorsed >> -XX:+DisableExplicitGC >> -XX:GCTimeRatio=9 >> -XX:MaxGCPauseMillis=1500 >> -XX:+UseParallelGC >> -Xss512k >> -XX:MaxPermSize=256m >> -Xmx3096m >> -Xms64m >> >>-Dlog4j.configuration=file:///usr/local/solr/solr-tomcat/conf/solr-log4j. >>pr >> operties >> -Dsolr.solr.home=/usr/local/solr/solr-home >> -Duser.timezone=GMT >> -javaagent:/usr/local/solr/solr-tomcat/newrelic/newrelic.jar >> -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager >> >>-Djava.util.logging.config.file=/usr/local/solr/solr-tomcat/conf/logging. >>pr >> operties >> >> >> We have one active core, the index is about 2.17 GB with around 6 >>million >> documents. >> >> The issue that we see is that every so often, heap memory will spike, GC >> percentage will go to 100%, and we¹ll see OOM errors. There¹s no change >> in traffic patterns as far as we can tell. According to New Relic, it¹s >> the Old Gen that¹s running out of space. The OOM stack trace is below. >> I¹d be very grateful for any help you can offer! >> >> Thanks, >> Scott >> >> ERROR - 2014-08-13 22:45:53.252; org.apache.solr.common.SolrException; >> null:java.lang.RuntimeException: java.lang.OutOfMemoryError: Java heap >> space > >When you actually get anOutOfMemoryErrorexception, it means that the >program is trying to allocate more memory than you have told it it's >allowed to allocate. In your case, you've limited it to 3096 >megabytes. This is not the same thing as garbage collection, although >the low memory situation will usually lead to a LOT of GC activity. > >This specific section of the following wiki page lists some of the >reasons that your Solr install might be using a lot of heap memory. >Later it describes some things you can do to reduce heap requirements: > >http://wiki.apache.org/solr/SolrPerformanceProblems#Java_Heap > >It's a good idea to read the entire wiki page, including the parts that >come before the Java Heap section that is directly linked above, and the >other wiki pages linked at the top of that page. > >Thanks, >Shawn > My question was actually more about what in Solr might cause the server to suddenly go from a very consistent heap size of 300-400 MB to over 2 GB in a matter of minutes with no changes in traffic. I get why the VM is crashing, I just don’t know why Solr is suddenly going nuts. This email message contains information that Corporate Reimbursement Services, Inc. considers confidential and/or proprietary, or may later designate as confidential and proprietary. It is intended only for use of the individual or entity named above and should not be forwarded to any other persons or entities without the express consent of Corporate Reimbursement Services, Inc., nor should it be used for any purpose other than in the course of any potential or actual business relationship with Corporate Reimbursement Services, Inc. If the reader of this message is not the intended recipient, or the employee or agent responsible to deliver it to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify sender immediately and destroy the original message. Internal Revenue Service regulations require that certain types of written advice include a disclaimer. To the extent the preceding message contains advice relating to a Federal tax issue, unless expressly stated otherwise the advice is not intended or written to be used, and it cannot be used by the recipient or any other taxpayer, for the purpose of avoiding Federal tax penalties, and was not written to support the promotion or marketing of any transaction or matter discussed herein.
