Java 1.5 has thread-locking bugs. Switching to Java 1.6 may cure this problem.
On Thu, Jan 15, 2009 at 10:57 AM, Jerome L Quinn <jlqu...@us.ibm.com> wrote: > > Hi, all. > > I'm running solr 1.3 inside Tomcat 6.0.18. I'm running a modified query > parser, tokenizer, highlighter, and have a CustomScoreQuery for dates. > > After some amount of time, I see solr stop responding to update requests. > When crawling through the logs, I see the following pattern: > > Jan 12, 2009 7:27:42 PM org.apache.solr.update.DirectUpdateHandler2 commit > INFO: start commit(optimize=false,waitFlush=false,waitSearcher=true) > Jan 12, 2009 7:28:11 PM org.apache.solr.common.SolrException log > SEVERE: Error during auto-warming of > key:org.apache.solr.search.queryresult...@ce0f92b9 > :java.lang.OutOfMemoryError > at org.apache.lucene.index.TermBuffer.toTerm(TermBuffer.java:122) > at org.apache.lucene.index.SegmentTermEnum.term > (SegmentTermEnum.java:167) > at org.apache.lucene.index.SegmentMergeInfo.next > (SegmentMergeInfo.java:66) > at org.apache.lucene.index.MultiSegmentReader$MultiTermEnum.next > (MultiSegmentReader.java:492) > at org.apache.lucene.search.FieldCacheImpl$7.createValue > (FieldCacheImpl.java:267) > at org.apache.lucene.search.FieldCacheImpl$Cache.get > (FieldCacheImpl.java:72) > at org.apache.lucene.search.FieldCacheImpl.getInts > (FieldCacheImpl.java:245) > at org.apache.solr.search.function.IntFieldSource.getValues > (IntFieldSource.java:50) > at org.apache.solr.search.function.SimpleFloatFunction.getValues > (SimpleFloatFunction.java:41) > at org.apache.solr.search.function.BoostedQuery$CustomScorer.<init> > (BoostedQuery.java:111) > at org.apache.solr.search.function.BoostedQuery$CustomScorer.<init> > (BoostedQuery.java:97) > at org.apache.solr.search.function.BoostedQuery > $BoostedWeight.scorer(BoostedQuery.java:88) > at org.apache.lucene.search.IndexSearcher.search > (IndexSearcher.java:132) > at org.apache.lucene.search.Searcher.search(Searcher.java:126) > at org.apache.lucene.search.Searcher.search(Searcher.java:105) > at org.apache.solr.search.SolrIndexSearcher.getDocListNC > (SolrIndexSearcher.java:966) > at org.apache.solr.search.SolrIndexSearcher.getDocListC > (SolrIndexSearcher.java:838) > at org.apache.solr.search.SolrIndexSearcher.access$000 > (SolrIndexSearcher.java:56) > at org.apache.solr.search.SolrIndexSearcher$2.regenerateItem > (SolrIndexSearcher.java:260) > at org.apache.solr.search.LRUCache.warm(LRUCache.java:194) > at org.apache.solr.search.SolrIndexSearcher.warm > (SolrIndexSearcher.java:1518) > at org.apache.solr.core.SolrCore$3.call(SolrCore.java:1018) > at java.util.concurrent.FutureTask$Sync.innerRun > (FutureTask.java:314) > at java.util.concurrent.FutureTask.run(FutureTask.java:149) > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask > (ThreadPoolExecutor.java:896) > at java.util.concurrent.ThreadPoolExecutor$Worker.run > (ThreadPoolExecutor.java:918) > at java.lang.Thread.run(Thread.java:735) > > Jan 12, 2009 7:28:11 PM org.apache.tomcat.util.net.JIoEndpoint$Acceptor run > SEVERE: Socket accept failed > Throwable occurred: java.lang.OutOfMemoryError > at java.net.PlainSocketImpl.socketAccept(Native Method) > at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:414) > at java.net.ServerSocket.implAccept(ServerSocket.java:464) > at java.net.ServerSocket.accept(ServerSocket.java:432) > at > org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket > (DefaultServerSocketFactory.java:61) > at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run > (JIoEndpoint.java:310) > at java.lang.Thread.run(Thread.java:735) > > <<<<<<<<<<<<<<<<<<<>>>>>>>>>>>>>>>>> > << Java dumps core and heap at this point >> > <<<<<<<<<<<<<<<<<<<>>>>>>>>>>>>>>>>> > > Jan 12, 2009 7:28:21 PM org.apache.solr.common.SolrException log > SEVERE: org.apache.lucene.store.LockObtainFailedException: Lock obtain > timed out: SingleInstanceLock: write.lock > at org.apache.lucene.store.Lock.obtain(Lock.java:85) > at org.apache.lucene.index.IndexWriter.init(IndexWriter.java:1140) > at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:938) > at org.apache.solr.update.SolrIndexWriter.<init> > (SolrIndexWriter.java:116) > at org.apache.solr.update.UpdateHandler.createMainIndexWriter > (UpdateHandler.java:122) > at org.apache.solr.update.DirectUpdateHandler2.openWriter > (DirectUpdateHandler2.java:167) > at org.apache.solr.update.DirectUpdateHandler2.addDoc > (DirectUpdateHandler2.java:221) > at org.apache.solr.update.processor.RunUpdateProcessor.processAdd > (RunUpdateProcessorFactory.java:59) > at org.apache.solr.handler.XmlUpdateRequestHandler.processUpdate > (XmlUpdateRequestHandler.java:196) > at > org.apache.solr.handler.XmlUpdateRequestHandler.handleRequestBody > (XmlUpdateRequestHandler.java:123) > at org.apache.solr.handler.RequestHandlerBase.handleRequest > (RequestHandlerBase.java:131) > at org.apache.solr.core.SolrCore.execute(SolrCore.java:1204) > at org.apache.solr.servlet.SolrDispatchFilter.execute > (SolrDispatchFilter.java:303) > at org.apache.solr.servlet.SolrDispatchFilter.doFilter > (SolrDispatchFilter.java:232) > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter > (ApplicationFilterChain.java:235) > at org.apache.catalina.core.ApplicationFilterChain.doFilter > (ApplicationFilterChain.java:206) > at org.apache.catalina.core.StandardWrapperValve.invoke > (StandardWrapperValve.java:233) > at org.apache.catalina.core.StandardContextValve.invoke > (StandardContextValve.java:191) > at org.apache.catalina.core.StandardHostValve.invoke > (StandardHostValve.java:128) > at org.apache.catalina.valves.ErrorReportValve.invoke > (ErrorReportValve.java:102) > at org.apache.catalina.core.StandardEngineValve.invoke > (StandardEngineValve.java:109) > at org.apache.catalina.connector.CoyoteAdapter.service > (CoyoteAdapter.java:286) > at org.apache.coyote.http11.Http11Processor.process > (Http11Processor.java:845) > at org.apache.coyote.http11.Http11Protocol > $Http11ConnectionHandler.process(Http11Protocol.java:583) > at org.apache.tomcat.util.net.JIoEndpoint$Worker.run > (JIoEndpoint.java:447) > at java.lang.Thread.run(Thread.java:735) > > > After this, all future updates cause the same write lock failure. > > I'm willing to believe my code is somehow causing Solr to run out of > memory, though I'd like to know if anyone sees the problem on vanilla Solr. > > An even bigger problem is the fact that once Solr is wedged, it stays that > way until a human notices and restarts things. The tomcat stays running > and there's no automatic detection that will either restart Solr, or > restart the Tomcat container. > > Any suggestions on either front? > > Thanks, > Jerry Quinn > >
