a really simple approach is to have the OOM generate an email e.g.
1) create a simple script (call it java_oom.sh) and drop it in your tomcat bin dir echo `date` | mail -s "Java Error: OutOfMemory - $HOSTNAME" not...@domain.com 2) configure your java options (in setenv.sh or similar) to trigger heap dump and the email script when OOM occurs # config error behaviors CATALINA_OPTS="$CATALINA_OPTS -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=$TOMCAT_DIR/temp/tomcat-dump.hprof -XX:OnError=$TOMCAT_DIR/bin/java_error.sh -XX:OnOutOfMemoryError=$TOMCAT_DIR/bin/java_oom.sh -XX:ErrorFile=$TOMCAT_DIR/temp/java_error%p.log" ________________________________________ From: Mark Miller <markrmil...@gmail.com> Sent: Tuesday, October 14, 2014 06:30 To: solr-user@lucene.apache.org Subject: Re: Recovering from Out of Mem Best is to pass the Java cmd line option that kills the process on OOM and setup a supervisor on the process to restart it. You need a somewhat recent release for this to work properly though. - Mark > On Oct 14, 2014, at 9:06 AM, Salman Akram > <salman.ak...@northbaysolutions.net> wrote: > > I know there are some suggestions to avoid OOM issue e.g. setting > appropriate Max Heap size etc. However, what's the best way to recover from > it as it goes into non-responding state? We are using Tomcat on back end. > > The scenario is that once we face OOM issue it keeps on taking queries > (doesn't give any error) but they just time out. So even though we have a > fail over system implemented but we don't have a way to distinguish if > these are real time out queries OR due to OOM. > > -- > Regards, > > Salman Akram
