Hi, all.
I'm running solr 1.3 inside Tomcat 6.0.18. I'm running a modified query
parser, tokenizer, highlighter, and have a CustomScoreQuery for dates.
After some amount of time, I see solr stop responding to update requests.
When crawling through the logs, I see the following pattern:
Jan 12, 2009 7:27:42 PM org.apache.solr.update.DirectUpdateHandler2 commit
INFO: start commit(optimize=false,waitFlush=false,waitSearcher=true)
Jan 12, 2009 7:28:11 PM org.apache.solr.common.SolrException log
SEVERE: Error during auto-warming of
key:org.apache.solr.search.queryresult...@ce0f92b9:java.lang.OutOfMemoryError
at org.apache.lucene.index.TermBuffer.toTerm(TermBuffer.java:122)
at org.apache.lucene.index.SegmentTermEnum.term
(SegmentTermEnum.java:167)
at org.apache.lucene.index.SegmentMergeInfo.next
(SegmentMergeInfo.java:66)
at org.apache.lucene.index.MultiSegmentReader$MultiTermEnum.next
(MultiSegmentReader.java:492)
at org.apache.lucene.search.FieldCacheImpl$7.createValue
(FieldCacheImpl.java:267)
at org.apache.lucene.search.FieldCacheImpl$Cache.get
(FieldCacheImpl.java:72)
at org.apache.lucene.search.FieldCacheImpl.getInts
(FieldCacheImpl.java:245)
at org.apache.solr.search.function.IntFieldSource.getValues
(IntFieldSource.java:50)
at org.apache.solr.search.function.SimpleFloatFunction.getValues
(SimpleFloatFunction.java:41)
at org.apache.solr.search.function.BoostedQuery$CustomScorer.<init>
(BoostedQuery.java:111)
at org.apache.solr.search.function.BoostedQuery$CustomScorer.<init>
(BoostedQuery.java:97)
at org.apache.solr.search.function.BoostedQuery
$BoostedWeight.scorer(BoostedQuery.java:88)
at org.apache.lucene.search.IndexSearcher.search
(IndexSearcher.java:132)
at org.apache.lucene.search.Searcher.search(Searcher.java:126)
at org.apache.lucene.search.Searcher.search(Searcher.java:105)
at org.apache.solr.search.SolrIndexSearcher.getDocListNC
(SolrIndexSearcher.java:966)
at org.apache.solr.search.SolrIndexSearcher.getDocListC
(SolrIndexSearcher.java:838)
at org.apache.solr.search.SolrIndexSearcher.access$000
(SolrIndexSearcher.java:56)
at org.apache.solr.search.SolrIndexSearcher$2.regenerateItem
(SolrIndexSearcher.java:260)
at org.apache.solr.search.LRUCache.warm(LRUCache.java:194)
at org.apache.solr.search.SolrIndexSearcher.warm
(SolrIndexSearcher.java:1518)
at org.apache.solr.core.SolrCore$3.call(SolrCore.java:1018)
at java.util.concurrent.FutureTask$Sync.innerRun
(FutureTask.java:314)
at java.util.concurrent.FutureTask.run(FutureTask.java:149)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask
(ThreadPoolExecutor.java:896)
at java.util.concurrent.ThreadPoolExecutor$Worker.run
(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:735)
Jan 12, 2009 7:28:11 PM org.apache.tomcat.util.net.JIoEndpoint$Acceptor run
SEVERE: Socket accept failed
Throwable occurred: java.lang.OutOfMemoryError
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:414)
at java.net.ServerSocket.implAccept(ServerSocket.java:464)
at java.net.ServerSocket.accept(ServerSocket.java:432)
at
org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket
(DefaultServerSocketFactory.java:61)
at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run
(JIoEndpoint.java:310)
at java.lang.Thread.run(Thread.java:735)
<<<<<<<<<<<<<<<<<<<>>>>>>>>>>>>>>>>>
<< Java dumps core and heap at this point >>
<<<<<<<<<<<<<<<<<<<>>>>>>>>>>>>>>>>>
Jan 12, 2009 7:28:21 PM org.apache.solr.common.SolrException log
SEVERE: org.apache.lucene.store.LockObtainFailedException: Lock obtain
timed out: SingleInstanceLock: write.lock
at org.apache.lucene.store.Lock.obtain(Lock.java:85)
at org.apache.lucene.index.IndexWriter.init(IndexWriter.java:1140)
at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:938)
at org.apache.solr.update.SolrIndexWriter.<init>
(SolrIndexWriter.java:116)
at org.apache.solr.update.UpdateHandler.createMainIndexWriter
(UpdateHandler.java:122)
at org.apache.solr.update.DirectUpdateHandler2.openWriter
(DirectUpdateHandler2.java:167)
at org.apache.solr.update.DirectUpdateHandler2.addDoc
(DirectUpdateHandler2.java:221)
at org.apache.solr.update.processor.RunUpdateProcessor.processAdd
(RunUpdateProcessorFactory.java:59)
at org.apache.solr.handler.XmlUpdateRequestHandler.processUpdate
(XmlUpdateRequestHandler.java:196)
at
org.apache.solr.handler.XmlUpdateRequestHandler.handleRequestBody
(XmlUpdateRequestHandler.java:123)
at org.apache.solr.handler.RequestHandlerBase.handleRequest
(RequestHandlerBase.java:131)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1204)
at org.apache.solr.servlet.SolrDispatchFilter.execute
(SolrDispatchFilter.java:303)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter
(SolrDispatchFilter.java:232)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter
(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter
(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke
(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke
(StandardContextValve.java:191)
at org.apache.catalina.core.StandardHostValve.invoke
(StandardHostValve.java:128)
at org.apache.catalina.valves.ErrorReportValve.invoke
(ErrorReportValve.java:102)
at org.apache.catalina.core.StandardEngineValve.invoke
(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service
(CoyoteAdapter.java:286)
at org.apache.coyote.http11.Http11Processor.process
(Http11Processor.java:845)
at org.apache.coyote.http11.Http11Protocol
$Http11ConnectionHandler.process(Http11Protocol.java:583)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run
(JIoEndpoint.java:447)
at java.lang.Thread.run(Thread.java:735)
After this, all future updates cause the same write lock failure.
I'm willing to believe my code is somehow causing Solr to run out of
memory, though I'd like to know if anyone sees the problem on vanilla Solr.
An even bigger problem is the fact that once Solr is wedged, it stays that
way until a human notices and restarts things. The tomcat stays running
and there's no automatic detection that will either restart Solr, or
restart the Tomcat container.
Any suggestions on either front?
Thanks,
Jerry Quinn
