Darnit - didn't finish that email. This is after running your old short doc perf test for 10,000 iterations. You see the same thing with 1000 iterations but much less pronounced eg gettin' worse with more iterations.
Mark Miller wrote: > A little before and after. The before is around may 5th'is - the after > is trunk. > > http://myhardshadow.com/memanalysis/before.png > http://myhardshadow.com/memanalysis/after.png > > Mark Miller wrote: > >> Took a peak at the checkout around the time he says he's using. >> >> CharTokenizer appears to be holding onto much large char[] arrays now >> than before. Same with snowball.Among - used to be almost nothing, now >> its largio. >> >> The new TokenStream stuff appears to be clinging. Needs to find some >> inner peace. >> >> Yonik Seeley wrote: >> >> >>> On Mon, Oct 5, 2009 at 4:54 PM, Jeff Newburn <jnewb...@zappos.com> wrote: >>> >>> >>> >>>> Ok we have done some more testing on this issue. When I only have the 1 >>>> core the reindex completes fine. However, when I added a second core with >>>> no documents it runs out of heap again. This time the heap was 322Mb of >>>> LRUCache. The 1 query that warms returns exactly 2 documents so I have no >>>> idea where the LRUCache is getting its information or what is even in >>>> there. >>>> >>>> >>>> >>> I guess the obvious thing to check would be the custom search component. >>> Does it access documents? I don't see how else the document cache >>> could self populate with so many entries (assuming it is the document >>> cache again). >>> >>> -Yonik >>> http://www.lucidimagination.com >>> >>> >>> >>> >>> >>> >>> >>>> -- >>>> Jeff Newburn >>>> Software Engineer, Zappos.com >>>> jnewb...@zappos.com - 702-943-7562 >>>> >>>> >>>> >>>> >>>> >>>>> From: Yonik Seeley <yo...@lucidimagination.com> >>>>> Reply-To: <solr-user@lucene.apache.org> >>>>> Date: Mon, 5 Oct 2009 13:32:32 -0400 >>>>> To: <solr-user@lucene.apache.org> >>>>> Subject: Re: Solr Trunk Heap Space Issues >>>>> >>>>> On Mon, Oct 5, 2009 at 1:00 PM, Jeff Newburn <jnewb...@zappos.com> wrote: >>>>> >>>>> >>>>> >>>>>> Ok I have eliminated all queries for warming and am still getting the >>>>>> heap >>>>>> space dump. Any ideas at this point what could be wrong? This seems >>>>>> like a >>>>>> huge increase in memory to go from indexing without issues to not being >>>>>> able >>>>>> to even with warming off. >>>>>> >>>>>> >>>>>> >>>>> Do you have any custom Analyzers, Tokenizers, TokenFilters? >>>>> Another change is that token streams are reused by caching in a >>>>> thread-local, so every thread in your server could potentially have a >>>>> copy of an analysis chain (token stream) per field that you have used. >>>>> This normally shouldn't be an issue since these will be small. Also, >>>>> how many unique fields do you have? >>>>> >>>>> -Yonik >>>>> http://www.lucidimagination.com >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>>> Jeff Newburn >>>>>> Software Engineer, Zappos.com >>>>>> jnewb...@zappos.com - 702-943-7562 >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> From: Jeff Newburn <jnewb...@zappos.com> >>>>>>> Reply-To: <solr-user@lucene.apache.org> >>>>>>> Date: Thu, 01 Oct 2009 08:41:18 -0700 >>>>>>> To: "solr-user@lucene.apache.org" <solr-user@lucene.apache.org> >>>>>>> Subject: Solr Trunk Heap Space Issues >>>>>>> >>>>>>> I am trying to update to the newest version of solr from trunk as of May >>>>>>> 5th. I updated and compiled from trunk as of yesterday (09/30/2009). >>>>>>> When >>>>>>> I try to do a full import I am receiving a GC heap error after changing >>>>>>> nothing in the configuration files. Why would this happen in the most >>>>>>> recent versions but not in the version from a few months ago. The stack >>>>>>> trace is below. >>>>>>> >>>>>>> Oct 1, 2009 8:34:32 AM >>>>>>> org.apache.solr.update.processor.LogUpdateProcessor >>>>>>> finish >>>>>>> INFO: {add=[166400, 166608, 166698, 166800, 166811, 167097, 167316, >>>>>>> 167353, >>>>>>> ...(83 more)]} 0 35991 >>>>>>> Oct 1, 2009 8:34:32 AM org.apache.solr.common.SolrException log >>>>>>> SEVERE: java.lang.OutOfMemoryError: GC overhead limit exceeded >>>>>>> at java.util.Arrays.copyOfRange(Arrays.java:3209) >>>>>>> at java.lang.String.<init>(String.java:215) >>>>>>> at >>>>>>> com.ctc.wstx.util.TextBuffer.contentsAsString(TextBuffer.java:384) >>>>>>> at >>>>>>> com.ctc.wstx.sr.BasicStreamReader.getText(BasicStreamReader.java:821) >>>>>>> at org.apache.solr.handler.XMLLoader.readDoc(XMLLoader.java:280) >>>>>>> at >>>>>>> org.apache.solr.handler.XMLLoader.processUpdate(XMLLoader.java:139) >>>>>>> at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:69) >>>>>>> at >>>>>>> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentSt >>>>>>> reamHandlerBase.java:54) >>>>>>> at >>>>>>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase. >>>>>>> java:131) >>>>>>> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316) >>>>>>> at >>>>>>> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:3 >>>>>>> 38) >>>>>>> at >>>>>>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java: >>>>>>> 241) >>>>>>> at >>>>>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application >>>>>>> FilterChain.java:235) >>>>>>> at >>>>>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh >>>>>>> ain.java:206) >>>>>>> at >>>>>>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.ja >>>>>>> va:233) >>>>>>> at >>>>>>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.ja >>>>>>> va:175) >>>>>>> at >>>>>>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128 >>>>>>> ) >>>>>>> at >>>>>>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102 >>>>>>> ) >>>>>>> at >>>>>>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java >>>>>>> :109) >>>>>>> at >>>>>>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286) >>>>>>> at >>>>>>> org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java: >>>>>>> 879) >>>>>>> at >>>>>>> org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(H >>>>>>> ttp11NioProtocol.java:719) >>>>>>> at >>>>>>> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java: >>>>>>> 2080) >>>>>>> at >>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja >>>>>>> va:886) >>>>>>> at >>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9 >>>>>>> 08) >>>>>>> at java.lang.Thread.run(Thread.java:619) >>>>>>> >>>>>>> Oct 1, 2009 8:40:06 AM org.apache.solr.core.SolrCore execute >>>>>>> INFO: [zeta-main] webapp=/solr path=/update params={} status=500 >>>>>>> QTime=5265 >>>>>>> Oct 1, 2009 8:40:12 AM org.apache.solr.common.SolrException log >>>>>>> SEVERE: java.lang.OutOfMemoryError: GC overhead limit exceeded >>>>>>> >>>>>>> -- >>>>>>> Jeff Newburn >>>>>>> Software Engineer, Zappos.com >>>>>>> jnewb...@zappos.com - 702-943-7562 >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> >>>> >>>> >>>> >> >> > > > -- - Mark http://www.lucidimagination.com
