Re: Recovering from Out of Mem

2014-10-21 Thread Shawn Heisey
On 10/21/2014 1:24 AM, Salman Akram wrote: > Yes so the most imp thing is what's the best way to 'know' that there is > OOM? Some script of a ping with 1-2 mins time? To touch on both your question and that posed by Toke Eskildsen: Java itself has a configuration option to call a program or scrip

Re: Recovering from Out of Mem

2014-10-21 Thread Salman Akram
ery) may just come back and you get into an > oscillating situation of restart after restart. i generally want a human > involved when error conditions which should be outliers (like OOM) are > happening > > > > From: Salman Akram >

Re: Recovering from Out of Mem

2014-10-21 Thread Toke Eskildsen
On Mon, 2014-10-20 at 16:25 +0200, Shawn Heisey wrote: > In general, once OOME happens, program operation (and in some cases the > status of the most recently indexed documents) is completely > undetermined. We can be sure that the data which has already been > written to disk will be correct, but

Re: Recovering from Out of Mem

2014-10-20 Thread Boogie Shafer
e OOM) are happening From: Salman Akram Sent: Monday, October 20, 2014 08:47 To: Solr Group Subject: Re: Recovering from Out of Mem " That's why it is considered better to crash the program and restart it for OOME." In the end aren't

Re: Recovering from Out of Mem

2014-10-20 Thread Salman Akram
" That's why it is considered better to crash the program and restart it for OOME." In the end aren't you also saying the same thing or I misunderstood something? We don't get this issue on master server (indexing). Our real concern is slave where sometimes (rare) so not an obvious heap config is

Re: Recovering from Out of Mem

2014-10-20 Thread Shawn Heisey
On 10/19/2014 11:32 PM, Ramzi Alqrainy wrote: > You can create a script to ping on Solr every 10 sec. if no response, then > restart it (Kill process id and run Solr again). > This is the fastest and easiest way to do that on windows. I wouldn't do this myself. Any temporary problem that results

Re: Recovering from Out of Mem

2014-10-19 Thread Salman Akram
u > > > just spit out an email alert and deal with service restarts and > > > troubleshooting manually > > > > > > > > > e.g. something like the following for a java_error.sh will drop an > email > > > with a timestamp > > > > > >

Re: Recovering from Out of Mem

2014-10-19 Thread Ramzi Alqrainy
You can create a script to ping on Solr every 10 sec. if no response, then restart it (Kill process id and run Solr again). This is the fastest and easiest way to do that on windows. -- View this message in context: http://lucene.472066.n3.nabble.com/Recovering-from-Out-of-Mem

Re: Recovering from Out of Mem

2014-10-17 Thread Tim Potter
restarts and > > troubleshooting manually > > > > > > e.g. something like the following for a java_error.sh will drop an email > > with a timestamp > > > > > > > > echo `date` | mail -s "Java Error: General - $HOSTNAME" > not...@domain.com > >

Re: Recovering from Out of Mem

2014-10-17 Thread Salman Akram
a Error: General - $HOSTNAME" not...@domain.com > > > > From: Tim Potter > Sent: Tuesday, October 14, 2014 07:35 > To: solr-user@lucene.apache.org > Subject: Re: Recovering from Out of Mem > > jfyi - the bin/solr script does the following: > > -XX:OnOutOfMemo

Re: Recovering from Out of Mem

2014-10-14 Thread Boogie Shafer
will drop an email with a timestamp echo `date` | mail -s "Java Error: General - $HOSTNAME" not...@domain.com From: Tim Potter Sent: Tuesday, October 14, 2014 07:35 To: solr-user@lucene.apache.org Subject: Re: Recovering from Out of Mem

Re: Recovering from Out of Mem

2014-10-14 Thread Tim Potter
mp and the email script when OOM occurs # > > > config error behaviors > > > CATALINA_OPTS="$CATALINA_OPTS -XX:+HeapDumpOnOutOfMemoryError > > > -XX:HeapDumpPath=$TOMCAT_DIR/temp/tomcat-dump.hprof > > > -XX:OnError=$TOMCAT_DIR/bin/java_error.sh > > >

Re: Recovering from Out of Mem

2014-10-14 Thread Markus Jelsma
OfMemoryError > > -XX:HeapDumpPath=$TOMCAT_DIR/temp/tomcat-dump.hprof > > -XX:OnError=$TOMCAT_DIR/bin/java_error.sh > > -XX:OnOutOfMemoryError=$TOMCAT_DIR/bin/java_oom.sh > > -XX:ErrorFile=$TOMCAT_DIR/temp/java_error%p.log" > > ________________ > > From

Re: Recovering from Out of Mem

2014-10-14 Thread Yago Riveiro
: Mark Miller > Sent: Tuesday, October 14, 2014 06:30 > To: solr-user@lucene.apache.org > Subject: Re: Recovering from Out of Mem > Best is to pass the Java cmd line option that kills the process on OOM and > setup a supervisor on the process to restart it. You need a somewhat recen

Re: Recovering from Out of Mem

2014-10-14 Thread Markus Jelsma
/java_error%p.log" > > > > > From: Mark Miller > Sent: Tuesday, October 14, 2014 06:30 > To: solr-user@lucene.apache.org > Subject: Re: Recovering from Out of Mem > > Best is to pass the Java cmd line option that kills the p

Re: Recovering from Out of Mem

2014-10-14 Thread Boogie Shafer
bin/java_oom.sh -XX:ErrorFile=$TOMCAT_DIR/temp/java_error%p.log" From: Mark Miller Sent: Tuesday, October 14, 2014 06:30 To: solr-user@lucene.apache.org Subject: Re: Recovering from Out of Mem Best is to pass the Java cmd line option that kills the process

Re: Recovering from Out of Mem

2014-10-14 Thread Mark Miller
Best is to pass the Java cmd line option that kills the process on OOM and setup a supervisor on the process to restart it. You need a somewhat recent release for this to work properly though. - Mark > On Oct 14, 2014, at 9:06 AM, Salman Akram > wrote: > > I know there are some suggestions

Recovering from Out of Mem

2014-10-14 Thread Salman Akram
I know there are some suggestions to avoid OOM issue e.g. setting appropriate Max Heap size etc. However, what's the best way to recover from it as it goes into non-responding state? We are using Tomcat on back end. The scenario is that once we face OOM issue it keeps on taking queries (doesn't gi