On Dec 23, 2009, at 10:15 AM, Jon Poulton wrote: > Hi there, > I'm looking at some problems we are having with some legacy code which uses > Solr (1.3) under the hood. We seem to get repeated OutOfMemory errors on a > 24-48 hour basis searching a relatively small index of 70,000 documents. This > may be an error in the way Solr is configured, or it may be a problem with > the way it's being invoked, I don't know, as I'm fairly new to Solr. > > The OutOfMemoryError is an unusual one: > > java.lang.OutOfMemoryError: GC overhead limit exceeded > > There are also a few stack traces in the solr logs. The first is: > > SEVERE: Error during auto-warming of > key:root_show_stop_range:[2009-12-22T08:59:37.254 TO > *]:java.lang.OutOfMemoryError: GC overhead limit exceeded > at java.util.Arrays.copyOfRange(Arrays.java:3209) > at java.lang.String.<init>(String.java:215) > at > org.apache.lucene.index.TermBuffer.toTerm(TermBuffer.java:122) > at > org.apache.lucene.index.SegmentTermEnum.term(SegmentTermEnum.java:167) > at > org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:251) > at > org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:218) > at > org.apache.lucene.index.SegmentTermDocs.seek(SegmentTermDocs.java:55) > at > org.apache.lucene.index.MultiSegmentReader$MultiTermDocs.termDocs(MultiSegmentReader.java:609) > at > org.apache.lucene.index.MultiSegmentReader$MultiTermDocs.next(MultiSegmentReader.java:560) > at > org.apache.lucene.search.RangeFilter.getDocIdSet(RangeFilter.java:268) > at > org.apache.lucene.search.ConstantScoreQuery$ConstantScorer.<init>(ConstantScoreQuery.java:116) > at > org.apache.lucene.search.ConstantScoreQuery$ConstantWeight.scorer(ConstantScoreQuery.java:81) > at > org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:131) > at org.apache.lucene.search.Searcher.search(Searcher.java:126) > at > org.apache.solr.search.SolrIndexSearcher.getDocSetNC(SolrIndexSearcher.java:601) > at > org.apache.solr.search.SolrIndexSearcher.getDocSet(SolrIndexSearcher.java:507) > at > org.apache.solr.search.SolrIndexSearcher.cacheDocSet(SolrIndexSearcher.java:482) > at > org.apache.solr.search.SolrIndexSearcher$1.regenerateItem(SolrIndexSearcher.java:224) > at org.apache.solr.search.LRUCache.warm(LRUCache.java:194) > at > org.apache.solr.search.SolrIndexSearcher.warm(SolrIndexSearcher.java:1518) > at org.apache.solr.core.SolrCore$3.call(SolrCore.java:1018) > at > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:619) > > I took a memory dump of the running application, and found that vast amounts > of memory (over 400MB) was being consumed by nine or ten large > SolrIndexSearcher objects; references to which are being held within a > LinkedHashMap in SolrCore called "infoRegistry". > > I had a quick look at the Solr 1.3.0 source code to try and figure out what > was going wrong and whether SolrCore was being used incorrectly in our own > source. It looks like whenever a new Searcher is created, it "registers" > itself with SolrCore, and this registration places a reference to the > Searcher in a LinkedHashMap (the "infoRegistry"). > > What is puzzling me is why so many SolrIndexSearcher objects are being > created, and what the conditions are for their creation and removal. The code > I can see in our own product does not use SolrIndexSearcher directly, it > simply makes calls to "execute" on SolrCore; so I would normally expect that > that class would be managing the Searcher life cycle internally.
It sounds like you are either using embedded mode or you have some custom code. Are you sure you are releasing your resources correctly? > > Does anyone have any idea as to what may be going on here? I have had a look > at the solrconfig.xml file, but it does not seem to depart significantly from > the defaults provided. > > Thanks in advance for any help. > > Jon
