Hi and thanks for looking at the problem ...
Mark Miller <markrmil...@gmail.com> wrote on 01/15/2009 02:58:24 PM: > Mark Miller <markrmil...@gmail.com> > 01/15/2009 02:58 PM > > Re: Help with Solr 1.3 lockups? > > How much RAM are you giving the JVM? Thats running out of memory loading > a FieldCache, which can be a more memory intensive data structure. It > pretty much points to the JVM not having enough RAM to do what you want. > How many fields do you sort on? How many fields do you facet on? How > much RAM do you have available and how much have you given Solr? How > many documents are you working with? I'm using the stock tomcat and JVM settings. I see the VM footprint sitting at 877M right now. It hasn't locked up yet this time around. There are 2 fields we facet and 1 that we sort on. The machine has 16G of memory, and the index is currently sitting at 38G, though I haven't run an optimize in a while. There are about 1 million docs in the index, though we have 3 full copies of the data stored in different fields and processed in different ways. I do a commit every 10 docs or 3 seconds, whichever comes first. We're approximating real-time updating. The index is currently sitting on NFS, which I know isn't great for performance. I didn't think it could cause reliability issues though. > As far as rebooting a failed server, the best technique is generally > external. I would recommend a script/program on another machine that > hits the Solr instance with a simple query every now and again. If you > don't get a valid response within a reasonable amount of time, or after > a reasonable number of tries, fire off alert emails and issue a command > to that server to reboot the JVM. Or something to that effect. I suspect I'll add a watchdog, no matter what's causing the problem here. > However, you should figure out why you are running out of memory. You > don't want to use more resources than you have available if you can help it. Definitely. That's on the agenda :-) Thanks, Jerry > - Mark > > Jerome L Quinn wrote: > > Hi, all. > > > > I'm running solr 1.3 inside Tomcat 6.0.18. I'm running a modified query > > parser, tokenizer, highlighter, and have a CustomScoreQuery for dates. > > > > After some amount of time, I see solr stop responding to update requests. > > When crawling through the logs, I see the following pattern: > > > > Jan 12, 2009 7:27:42 PM org.apache.solr.update.DirectUpdateHandler2 commit > > INFO: start commit(optimize=false,waitFlush=false,waitSearcher=true) > > Jan 12, 2009 7:28:11 PM org.apache.solr.common.SolrException log > > SEVERE: Error during auto-warming of > > > key:org.apache.solr.search.queryresult...@ce0f92b9:java.lang.OutOfMemoryError > > at org.apache.lucene.index.TermBuffer.toTerm (TermBuffer.java:122) > > at org.apache.lucene.index.SegmentTermEnum.term > > (SegmentTermEnum.java:167) > > at org.apache.lucene.index.SegmentMergeInfo.next > > (SegmentMergeInfo.java:66) > > at org.apache.lucene.index.MultiSegmentReader $MultiTermEnum.next > > (MultiSegmentReader.java:492) > > at org.apache.lucene.search.FieldCacheImpl$7.createValue > > (FieldCacheImpl.java:267) > > at org.apache.lucene.search.FieldCacheImpl$Cache.get > > (FieldCacheImpl.java:72) > > at org.apache.lucene.search.FieldCacheImpl.getInts > > (FieldCacheImpl.java:245) > > at org.apache.solr.search.function.IntFieldSource.getValues > > (IntFieldSource.java:50) > > at org.apache.solr.search.function.SimpleFloatFunction.getValues > > (SimpleFloatFunction.java:41) > > at org.apache.solr.search.function.BoostedQuery $CustomScorer.<init> > > (BoostedQuery.java:111) > > at org.apache.solr.search.function.BoostedQuery $CustomScorer.<init> > > (BoostedQuery.java:97) > > at org.apache.solr.search.function.BoostedQuery > > $BoostedWeight.scorer(BoostedQuery.java:88) > > at org.apache.lucene.search.IndexSearcher.search > > (IndexSearcher.java:132) > > at org.apache.lucene.search.Searcher.search(Searcher.java:126) > > at org.apache.lucene.search.Searcher.search(Searcher.java:105) > > at org.apache.solr.search.SolrIndexSearcher.getDocListNC > > (SolrIndexSearcher.java:966) > > at org.apache.solr.search.SolrIndexSearcher.getDocListC > > (SolrIndexSearcher.java:838) > > at org.apache.solr.search.SolrIndexSearcher.access$000 > > (SolrIndexSearcher.java:56) > > at org.apache.solr.search.SolrIndexSearcher$2.regenerateItem > > (SolrIndexSearcher.java:260) > > at org.apache.solr.search.LRUCache.warm(LRUCache.java:194) > > at org.apache.solr.search.SolrIndexSearcher.warm > > (SolrIndexSearcher.java:1518) > > at org.apache.solr.core.SolrCore$3.call(SolrCore.java:1018) > > at java.util.concurrent.FutureTask$Sync.innerRun > > (FutureTask.java:314) > > at java.util.concurrent.FutureTask.run(FutureTask.java:149) > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask > > (ThreadPoolExecutor.java:896) > > at java.util.concurrent.ThreadPoolExecutor$Worker.run > > (ThreadPoolExecutor.java:918) > > at java.lang.Thread.run(Thread.java:735) > > > > Jan 12, 2009 7:28:11 PM org.apache.tomcat.util.net.JIoEndpoint$Acceptor run > > SEVERE: Socket accept failed > > Throwable occurred: java.lang.OutOfMemoryError > > at java.net.PlainSocketImpl.socketAccept(Native Method) > > at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:414) > > at java.net.ServerSocket.implAccept(ServerSocket.java:464) > > at java.net.ServerSocket.accept(ServerSocket.java:432) > > at > > org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket > > (DefaultServerSocketFactory.java:61) > > at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run > > (JIoEndpoint.java:310) > > at java.lang.Thread.run(Thread.java:735) > > > > <<<<<<<<<<<<<<<<<<<>>>>>>>>>>>>>>>>> > > << Java dumps core and heap at this point >> > > <<<<<<<<<<<<<<<<<<<>>>>>>>>>>>>>>>>> > > > > Jan 12, 2009 7:28:21 PM org.apache.solr.common.SolrException log > > SEVERE: org.apache.lucene.store.LockObtainFailedException: Lock obtain > > timed out: SingleInstanceLock: write.lock > > at org.apache.lucene.store.Lock.obtain(Lock.java:85) > > at org.apache.lucene.index.IndexWriter.init (IndexWriter.java:1140) > > at org.apache.lucene.index.IndexWriter.<init> (IndexWriter.java:938) > > at org.apache.solr.update.SolrIndexWriter.<init> > > (SolrIndexWriter.java:116) > > at org.apache.solr.update.UpdateHandler.createMainIndexWriter > > (UpdateHandler.java:122) > > at org.apache.solr.update.DirectUpdateHandler2.openWriter > > (DirectUpdateHandler2.java:167) > > at org.apache.solr.update.DirectUpdateHandler2.addDoc > > (DirectUpdateHandler2.java:221) > > at org.apache.solr.update.processor.RunUpdateProcessor.processAdd > > (RunUpdateProcessorFactory.java:59) > > at org.apache.solr.handler.XmlUpdateRequestHandler.processUpdate > > (XmlUpdateRequestHandler.java:196) > > at > > org.apache.solr.handler.XmlUpdateRequestHandler.handleRequestBody > > (XmlUpdateRequestHandler.java:123) > > at org.apache.solr.handler.RequestHandlerBase.handleRequest > > (RequestHandlerBase.java:131) > > at org.apache.solr.core.SolrCore.execute(SolrCore.java:1204) > > at org.apache.solr.servlet.SolrDispatchFilter.execute > > (SolrDispatchFilter.java:303) > > at org.apache.solr.servlet.SolrDispatchFilter.doFilter > > (SolrDispatchFilter.java:232) > > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter > > (ApplicationFilterChain.java:235) > > at org.apache.catalina.core.ApplicationFilterChain.doFilter > > (ApplicationFilterChain.java:206) > > at org.apache.catalina.core.StandardWrapperValve.invoke > > (StandardWrapperValve.java:233) > > at org.apache.catalina.core.StandardContextValve.invoke > > (StandardContextValve.java:191) > > at org.apache.catalina.core.StandardHostValve.invoke > > (StandardHostValve.java:128) > > at org.apache.catalina.valves.ErrorReportValve.invoke > > (ErrorReportValve.java:102) > > at org.apache.catalina.core.StandardEngineValve.invoke > > (StandardEngineValve.java:109) > > at org.apache.catalina.connector.CoyoteAdapter.service > > (CoyoteAdapter.java:286) > > at org.apache.coyote.http11.Http11Processor.process > > (Http11Processor.java:845) > > at org.apache.coyote.http11.Http11Protocol > > $Http11ConnectionHandler.process(Http11Protocol.java:583) > > at org.apache.tomcat.util.net.JIoEndpoint$Worker.run > > (JIoEndpoint.java:447) > > at java.lang.Thread.run(Thread.java:735) > > > > > > After this, all future updates cause the same write lock failure. > > > > I'm willing to believe my code is somehow causing Solr to run out of > > memory, though I'd like to know if anyone sees the problem on vanilla Solr. > > > > An even bigger problem is the fact that once Solr is wedged, it stays that > > way until a human notices and restarts things. The tomcat stays running > > and there's no automatic detection that will either restart Solr, or > > restart the Tomcat container. > > > > Any suggestions on either front? > > > > Thanks, > > Jerry Quinn > > > > > > >
