Hello all, I¹m running a Solr setup and am getting occasional periods where memory usage and GC just spike out of nowhere (unrelated to traffic). I¹m hoping someone can shed some light. Here¹s the setup:
- Solr 4.3.1, Oracle JDK 1.7.0_51 64 bit on CentOS 6.5 - We have 2 Solr servers, one acting as a master that receives all the writes, and one that is replicating from the master and handles all the reads - JVM parameters are: -Djava.io.tmpdir=/usr/local/solr/solr-tomcat/temp -Dcatalina.home=/usr/local/solr/solr-tomcat -Dcatalina.base=/usr/local/solr/solr-tomcat -Djava.endorsed.dirs=/usr/local/solr/solr-tomcat/endorsed -XX:+DisableExplicitGC -XX:GCTimeRatio=9 -XX:MaxGCPauseMillis=1500 -XX:+UseParallelGC -Xss512k -XX:MaxPermSize=256m -Xmx3096m -Xms64m -Dlog4j.configuration=file:///usr/local/solr/solr-tomcat/conf/solr-log4j.pr operties -Dsolr.solr.home=/usr/local/solr/solr-home -Duser.timezone=GMT -javaagent:/usr/local/solr/solr-tomcat/newrelic/newrelic.jar -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.util.logging.config.file=/usr/local/solr/solr-tomcat/conf/logging.pr operties We have one active core, the index is about 2.17 GB with around 6 million documents. The issue that we see is that every so often, heap memory will spike, GC percentage will go to 100%, and we¹ll see OOM errors. There¹s no change in traffic patterns as far as we can tell. According to New Relic, it¹s the Old Gen that¹s running out of space. The OOM stack trace is below. I¹d be very grateful for any help you can offer! Thanks, Scott ERROR - 2014-08-13 22:45:53.252; org.apache.solr.common.SolrException; null:java.lang.RuntimeException: java.lang.OutOfMemoryError: Java heap space at org.apache.solr.servlet.SolrDispatchFilter.sendError(SolrDispatchFilter.jav a:670) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java :380) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java :155) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applicatio nFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterC hain.java:210) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.j ava:222) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.j ava:123) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:17 1) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99 ) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.jav a:118) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Proc essor.java:1023) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(Abstra ctProtocol.java:589) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java :310) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1 145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java: 615) at java.lang.Thread.run(Thread.java:744) Caused by: java.lang.OutOfMemoryError: Java heap space at org.apache.lucene.search.FieldCacheImpl$DoubleCache$1.visitTerm(FieldCacheI mpl.java:1041) at org.apache.lucene.search.FieldCacheImpl$Uninvert.uninvert(FieldCacheImpl.ja va:312) at org.apache.lucene.search.FieldCacheImpl$DoubleCache.createValue(FieldCacheI mpl.java:1057) at org.apache.lucene.search.FieldCacheImpl$Cache.get(FieldCacheImpl.java:212) at org.apache.lucene.search.FieldCacheImpl.getDoubles(FieldCacheImpl.java:987) at org.apache.lucene.queries.function.valuesource.DoubleFieldSource.getValues( DoubleFieldSource.java:61) at org.apache.solr.search.function.distance.HaversineConstFunction.getValues(H aversineConstFunction.java:204) at org.apache.lucene.queries.function.valuesource.ReciprocalFloatFunction.getV alues(ReciprocalFloatFunction.java:65) at org.apache.lucene.queries.function.FunctionQuery$AllScorer.<init>(FunctionQ uery.java:120) at org.apache.lucene.queries.function.FunctionQuery$FunctionWeight.scorer(Func tionQuery.java:95) at org.apache.lucene.search.BooleanQuery$BooleanWeight.scorer(BooleanQuery.jav a:311) at org.apache.lucene.search.FilteredQuery$RandomAccessFilterStrategy.filteredS corer(FilteredQuery.java:538) at org.apache.lucene.search.FilteredQuery$1.scorer(FilteredQuery.java:133) at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:621) at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:297) at org.apache.solr.search.Grouping.searchWithTimeLimiter(Grouping.java:426) at org.apache.solr.search.Grouping.execute(Grouping.java:348) at org.apache.solr.handler.component.QueryComponent.process(QueryComponent.jav a:384) at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHan dler.java:208) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase .java:135) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1820) at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java: 656) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java :359) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java :155) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Applicatio nFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterC hain.java:210) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.j ava:222) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.j ava:123) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:17 1) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99 ) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.jav a:118) Scott Rankin Corporate Reimbursement Services, Inc. Phone: 617-467-1931 Email: sran...@crsinc.com This email message contains information that Corporate Reimbursement Services, Inc. considers confidential and/or proprietary, or may later designate as confidential and proprietary. It is intended only for use of the individual or entity named above and should not be forwarded to any other persons or entities without the express consent of Corporate Reimbursement Services, Inc., nor should it be used for any purpose other than in the course of any potential or actual business relationship with Corporate Reimbursement Services, Inc. If the reader of this message is not the intended recipient, or the employee or agent responsible to deliver it to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify sender immediately and destroy the original message. Internal Revenue Service regulations require that certain types of written advice include a disclaimer. To the extent the preceding message contains advice relating to a Federal tax issue, unless expressly stated otherwise the advice is not intended or written to be used, and it cannot be used by the recipient or any other taxpayer, for the purpose of avoiding Federal tax penalties, and was not written to support the promotion or marketing of any transaction or matter discussed herein.
