Well, this is probably not a rogue query. You can test this, of course by replaying all your queries on a test system. My guess is that it's just too much stuff on too small a box.
Or you could have poorly configured Solr parameters. I've seen, for instance, the filterCache sized at 1M. Which runs fine for a long time until indexing stops and the filterCache fills up. Which from your picture seems like a possibility, you have an increasing memory usage for days. I'd suggest something like jhat/jmap to find out where the memory is being used up, a review of your configs, things like that. Best Erick On Fri, Aug 2, 2013 at 7:44 AM, Grzegorz Sobczyk <gsobc...@gmail.com> wrote: > Solr parameters listed in dashboard: > -DzkHost=localhost:2181,172.27.5.121:2181,172.27.5.122:2181 > -XX:+UseConcMarkSweepGC > -Xmx3072m > -Djava.awt.headless=true > > Mem usage in last days: http://i42.tinypic.com/29z5rew.png > > It's production system and there's too many requests to detect which query > is buggy > some lines before every error (result of: zgrep -B 200 "SEVERE: > null:java.lang.RuntimeException: java.lang.OutOfMemoryError" > daemon.err.2.gz > out.log ) > https://gist.github.com/gsobczyk/65e2ab9bb13a99aec89e > > I've tried invoking recent request before exception but without success of > reproduce error > > > > On 1 August 2013 20:39, Erick Erickson <erickerick...@gmail.com> wrote: > > > What are the memory parameters you start Solr with? The Solr admin page > > will tell you how much memory the JVM has. > > > > Also, cut/paste the queries you're running when you see this. > > > > Best > > Erick > > > > > > On Thu, Aug 1, 2013 at 9:50 AM, Grzegorz Sobczyk <gsobc...@gmail.com> > > wrote: > > > > > after node starts in log I have only few requests: > > > https://gist.github.com/gsobczyk/6131503#file-solr-oom-log > > > this error occurred multiple times > > > > > > > > > > > > > > > On 1 August 2013 15:33, Rafał Kuć <r....@solr.pl> wrote: > > > > > > > Hello! > > > > > > > > The exception you've shown tells you that Solr tried to allocate an > > > > array that exceeded heap size. Do you use some custom sorts? Did you > > > > send large bulks during the time that the exception occurred? > > > > > > > > -- > > > > Regards, > > > > Rafał Kuć > > > > Sematext :: http://sematext.com/ :: Solr - Lucene - ElasticSearch > > > > > > > > > Today I found in solr logs exception: java.lang.OutOfMemoryError: > > > > Requested > > > > > array size exceeds VM limit. > > > > > At that time memory usage was ~200MB / Xmx3g > > > > > > > > > Env looks like this: > > > > > 3x standalone zK (Java 7) > > > > > 3x Solr 4.2.1 on Tomcat (Java 7) > > > > > Linux 3.2.0-4-amd64 #1 SMP Debian 3.2.46-1 x86_64 GNU/Linux > > > > > One Solr and one ZK on single host: lmsiprse01, lmsiprse02, > > lmsiprse03 > > > > > > > > > Before exception I start restarting Solr: > > > > > lmsiprse01: > > > > > [2013-08-01 05:23:43]: /etc/init.d/tomcat6-1 stop > > > > > [2013-08-01 05:25:09]: /etc/init.d/tomcat6-1 start > > > > > lmsiprse02 (leader): > > > > > 2013-08-01 05:27:21]: /etc/init.d/tomcat6-1 stop > > > > > 2013-08-01 05:29:31]: /etc/init.d/tomcat6-1 start > > > > > lmsiprse03: > > > > > [2013-08-01 05:25:48]: /etc/init.d/tomcat6-1 stop > > > > > [2013-08-01 05:26:42]: /etc/init.d/tomcat6-1 start > > > > > > > > > and error shows up at 2013 5:27:26 on lmsiprse01 > > > > > > > > > Is anybody knows what happened? > > > > > > > > > fragment of log looks: > > > > > sie 01, 2013 5:27:26 AM org.apache.solr.core.SolrCore execute > > > > > INFO: [products] webapp=/solr path=/select > > > > > > > > > > > > > > > params={facet=true&start=0&q=&facet.limit=-1&facet.field=attribute_u-typ&facet.field=attribute_u-gama-kolorystyczna&facet.field=brand_name&wt=javabin&fq=node_id:1056&version=2&rows=0} > > > > > hits=1241 status=0 QTime=33 > > > > > sie 01, 2013 5:27:26 AM org.apache.solr.common.SolrException log > > > > > SEVERE: null:java.lang.RuntimeException: > java.lang.OutOfMemoryError: > > > > > Requested array size exceeds VM limit > > > > > at > > > > > > > > > > > > > > > org.apache.solr.servlet.SolrDispatchFilter.sendError(SolrDispatchFilter.java:653) > > > > > at > > > > > > > > > > > > > > > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:366) > > > > > at > > > > > > > > > > > > > > > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:141) > > > > > at > > > > > > > > > > > > > > > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) > > > > > at > > > > > > > > > > > > > > > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) > > > > > at > > > > > > > > > > > > > > > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) > > > > > at > > > > > > > > > > > > > > > org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) > > > > > at > > > > > > > > > > > > > > > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) > > > > > at > > > > > > > > > > > > > > > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) > > > > > at > > > > > > > > > > > > > > > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) > > > > > at > > > > > > > > > > > > > > > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293) > > > > > at > > > > > > > > > > > > > > > org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859) > > > > > at > > > > > > > > > > > > > > > org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:602) > > > > > at > > > > > > > org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489) > > > > > at java.lang.Thread.run(Thread.java:724) > > > > > Caused by: java.lang.OutOfMemoryError: Requested array size exceeds > > VM > > > > limit > > > > > at > org.apache.lucene.util.PriorityQueue.<init>(PriorityQueue.java:64) > > > > > at > org.apache.lucene.util.PriorityQueue.<init>(PriorityQueue.java:37) > > > > > at > > > > > > > > > > > > > > > org.apache.solr.handler.component.ShardFieldSortedHitQueue.<init>(ShardDoc.java:113) > > > > > at > > > > > > > > > > > > > > > org.apache.solr.handler.component.QueryComponent.mergeIds(QueryComponent.java:766) > > > > > at > > > > > > > > > > > > > > > org.apache.solr.handler.component.QueryComponent.handleRegularResponses(QueryComponent.java:625) > > > > > at > > > > > > > > > > > > > > > org.apache.solr.handler.component.QueryComponent.handleResponses(QueryComponent.java:604) > > > > > at > > > > > > > > > > > > > > > org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:311) > > > > > at > > > > > > > > > > > > > > > org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135) > > > > > at org.apache.solr.core.SolrCore.execute(SolrCore.java:1817) > > > > > at > > > > > > > > > > > > > > > org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:639) > > > > > at > > > > > > > > > > > > > > > org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:345) > > > > > ... 13 more > > > > > > > > > > > > > > > > > -- > > > Grzegorz Sobczyk > > > > > > > > > -- > Grzegorz Sobczyk >
