A little more to this ... Just on chance this was a weird Jetty issue or something, I tried with the latest 9.... and the problem still occurs :-(
This is on Java 7 on debian: java version "1.7.0_21" Java(TM) SE Runtime Environment (build 1.7.0_21-b11) Java HotSpot(TM) 64-Bit Server VM (build 23.21-b01, mixed mode) Here is an example stack trace from the log 2013-06-26 19:31:33,801 [qtp632640515-62] ERROR solr.servlet.SolrDispatchFilter Q:22 - null:java.lang.RuntimeException: java.lang.OutOfMemoryError: Java heap space at org.apache.solr.servlet.SolrDispatchFilter.sendError(SolrDispatchFilter.java:670) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:380) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:155) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1423) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:450) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:138) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:564) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:213) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1083) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:379) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:175) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1017) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:136) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:258) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:109) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) at org.eclipse.jetty.server.Server.handle(Server.java:445) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:260) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:225) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.run(AbstractConnection.java:358) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:596) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:527) at java.lang.Thread.run(Thread.java:722) Caused by: java.lang.OutOfMemoryError: Java heap space On Wed, Jun 26, 2013 at 12:27 PM, Timothy Potter <thelabd...@gmail.com> wrote: > Recently upgraded to 4.3.1 but this problem has persisted for a while now ... > > I'm using the following configuration when starting Jetty: > > -XX:OnOutOfMemoryError="/home/solr/oom_killer.sh 83 %p" > > If an OOM is triggered during Solr web app initialization (such as by > me lowering -Xmx to a value that is too low to initialize Solr with), > then the script gets called and does what I expect! > > However, once the Solr webapp initializes and Solr is happily > responding to updates and queries. When an OOM occurs in this > situation, then the script doesn't actually get invoked! All I see is > the following in the stdout/stderr log of my process: > > # > # java.lang.OutOfMemoryError: Java heap space > # -XX:OnOutOfMemoryError="/home/solr/oom_killer.sh 83 %p" > # Executing /bin/sh -c "/home/solr/oom_killer.sh 83 21358"... > > The oom_killer.sh script doesn't actually get called! > > So to recap, it works if an OOM occurs during initialization but once > Solr is running, the OOM killer doesn't fire correctly. This leads me > to believe my script is fine and there's something else going wrong. > Here's the oom_killer.sh script (pretty basic): > > #!/bin/bash > SOLR_PORT=$1 > SOLR_PID=$2 > NOW=$(date +"%Y%m%d_%H%M") > ( > echo "Running OOM killer script for process $SOLR_PID for Solr on port > 89$SOLR_PORT" > kill -9 $SOLR_PID > echo "Killed process $SOLR_PID" > exec /home/solr/solr-dg/dg-solr.sh recover $SOLR_PORT & > echo "Restarted Solr on 89$SOLR_PORT after OOM" > ) | tee oom_killer-89$SOLR_PORT-$NOW.log > > Anyone see anything like this before? Suggestions on where to begin > tracking down this issue? > > Cheers, > Tim
