Yeah - I was wondering about that ... not sure how these guys are stacking up ...
Yonik Seeley wrote: > TestIndexingPerformance? > What the heck... that's not even multi-threaded! > > -Yonik > http://www.lucidimagination.com > > > > On Tue, Oct 6, 2009 at 12:17 PM, Mark Miller <markrmil...@gmail.com> wrote: > >> Darnit - didn't finish that email. This is after running your old short >> doc perf test for 10,000 iterations. You see the same thing with 1000 >> iterations but much less pronounced eg gettin' worse with more iterations. >> >> Mark Miller wrote: >> >>> A little before and after. The before is around may 5th'is - the after >>> is trunk. >>> >>> http://myhardshadow.com/memanalysis/before.png >>> http://myhardshadow.com/memanalysis/after.png >>> >>> Mark Miller wrote: >>> >>> >>>> Took a peak at the checkout around the time he says he's using. >>>> >>>> CharTokenizer appears to be holding onto much large char[] arrays now >>>> than before. Same with snowball.Among - used to be almost nothing, now >>>> its largio. >>>> >>>> The new TokenStream stuff appears to be clinging. Needs to find some >>>> inner peace. >>>> >>>> Yonik Seeley wrote: >>>> >>>> >>>> >>>>> On Mon, Oct 5, 2009 at 4:54 PM, Jeff Newburn <jnewb...@zappos.com> wrote: >>>>> >>>>> >>>>> >>>>> >>>>>> Ok we have done some more testing on this issue. When I only have the 1 >>>>>> core the reindex completes fine. However, when I added a second core >>>>>> with >>>>>> no documents it runs out of heap again. This time the heap was 322Mb of >>>>>> LRUCache. The 1 query that warms returns exactly 2 documents so I have >>>>>> no >>>>>> idea where the LRUCache is getting its information or what is even in >>>>>> there. >>>>>> >>>>>> >>>>>> >>>>>> >>>>> I guess the obvious thing to check would be the custom search component. >>>>> Does it access documents? I don't see how else the document cache >>>>> could self populate with so many entries (assuming it is the document >>>>> cache again). >>>>> >>>>> -Yonik >>>>> http://www.lucidimagination.com >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>>> -- >>>>>> Jeff Newburn >>>>>> Software Engineer, Zappos.com >>>>>> jnewb...@zappos.com - 702-943-7562 >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>> From: Yonik Seeley <yo...@lucidimagination.com> >>>>>>> Reply-To: <solr-user@lucene.apache.org> >>>>>>> Date: Mon, 5 Oct 2009 13:32:32 -0400 >>>>>>> To: <solr-user@lucene.apache.org> >>>>>>> Subject: Re: Solr Trunk Heap Space Issues >>>>>>> >>>>>>> On Mon, Oct 5, 2009 at 1:00 PM, Jeff Newburn <jnewb...@zappos.com> >>>>>>> wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> Ok I have eliminated all queries for warming and am still getting the >>>>>>>> heap >>>>>>>> space dump. Any ideas at this point what could be wrong? This seems >>>>>>>> like a >>>>>>>> huge increase in memory to go from indexing without issues to not >>>>>>>> being able >>>>>>>> to even with warming off. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> Do you have any custom Analyzers, Tokenizers, TokenFilters? >>>>>>> Another change is that token streams are reused by caching in a >>>>>>> thread-local, so every thread in your server could potentially have a >>>>>>> copy of an analysis chain (token stream) per field that you have used. >>>>>>> This normally shouldn't be an issue since these will be small. Also, >>>>>>> how many unique fields do you have? >>>>>>> >>>>>>> -Yonik >>>>>>> http://www.lucidimagination.com >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>> Jeff Newburn >>>>>>>> Software Engineer, Zappos.com >>>>>>>> jnewb...@zappos.com - 702-943-7562 >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> From: Jeff Newburn <jnewb...@zappos.com> >>>>>>>>> Reply-To: <solr-user@lucene.apache.org> >>>>>>>>> Date: Thu, 01 Oct 2009 08:41:18 -0700 >>>>>>>>> To: "solr-user@lucene.apache.org" <solr-user@lucene.apache.org> >>>>>>>>> Subject: Solr Trunk Heap Space Issues >>>>>>>>> >>>>>>>>> I am trying to update to the newest version of solr from trunk as of >>>>>>>>> May >>>>>>>>> 5th. I updated and compiled from trunk as of yesterday (09/30/2009). >>>>>>>>> When >>>>>>>>> I try to do a full import I am receiving a GC heap error after >>>>>>>>> changing >>>>>>>>> nothing in the configuration files. Why would this happen in the most >>>>>>>>> recent versions but not in the version from a few months ago. The >>>>>>>>> stack >>>>>>>>> trace is below. >>>>>>>>> >>>>>>>>> Oct 1, 2009 8:34:32 AM >>>>>>>>> org.apache.solr.update.processor.LogUpdateProcessor >>>>>>>>> finish >>>>>>>>> INFO: {add=[166400, 166608, 166698, 166800, 166811, 167097, 167316, >>>>>>>>> 167353, >>>>>>>>> ...(83 more)]} 0 35991 >>>>>>>>> Oct 1, 2009 8:34:32 AM org.apache.solr.common.SolrException log >>>>>>>>> SEVERE: java.lang.OutOfMemoryError: GC overhead limit exceeded >>>>>>>>> at java.util.Arrays.copyOfRange(Arrays.java:3209) >>>>>>>>> at java.lang.String.<init>(String.java:215) >>>>>>>>> at >>>>>>>>> com.ctc.wstx.util.TextBuffer.contentsAsString(TextBuffer.java:384) >>>>>>>>> at >>>>>>>>> com.ctc.wstx.sr.BasicStreamReader.getText(BasicStreamReader.java:821) >>>>>>>>> at org.apache.solr.handler.XMLLoader.readDoc(XMLLoader.java:280) >>>>>>>>> at >>>>>>>>> org.apache.solr.handler.XMLLoader.processUpdate(XMLLoader.java:139) >>>>>>>>> at org.apache.solr.handler.XMLLoader.load(XMLLoader.java:69) >>>>>>>>> at >>>>>>>>> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentSt >>>>>>>>> reamHandlerBase.java:54) >>>>>>>>> at >>>>>>>>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase. >>>>>>>>> java:131) >>>>>>>>> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316) >>>>>>>>> at >>>>>>>>> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:3 >>>>>>>>> 38) >>>>>>>>> at >>>>>>>>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java: >>>>>>>>> 241) >>>>>>>>> at >>>>>>>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application >>>>>>>>> FilterChain.java:235) >>>>>>>>> at >>>>>>>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh >>>>>>>>> ain.java:206) >>>>>>>>> at >>>>>>>>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.ja >>>>>>>>> va:233) >>>>>>>>> at >>>>>>>>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.ja >>>>>>>>> va:175) >>>>>>>>> at >>>>>>>>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128 >>>>>>>>> ) >>>>>>>>> at >>>>>>>>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102 >>>>>>>>> ) >>>>>>>>> at >>>>>>>>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java >>>>>>>>> :109) >>>>>>>>> at >>>>>>>>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286) >>>>>>>>> at >>>>>>>>> org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java: >>>>>>>>> 879) >>>>>>>>> at >>>>>>>>> org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(H >>>>>>>>> ttp11NioProtocol.java:719) >>>>>>>>> at >>>>>>>>> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java: >>>>>>>>> 2080) >>>>>>>>> at >>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.ja >>>>>>>>> va:886) >>>>>>>>> at >>>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:9 >>>>>>>>> 08) >>>>>>>>> at java.lang.Thread.run(Thread.java:619) >>>>>>>>> >>>>>>>>> Oct 1, 2009 8:40:06 AM org.apache.solr.core.SolrCore execute >>>>>>>>> INFO: [zeta-main] webapp=/solr path=/update params={} status=500 >>>>>>>>> QTime=5265 >>>>>>>>> Oct 1, 2009 8:40:12 AM org.apache.solr.common.SolrException log >>>>>>>>> SEVERE: java.lang.OutOfMemoryError: GC overhead limit exceeded >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Jeff Newburn >>>>>>>>> Software Engineer, Zappos.com >>>>>>>>> jnewb...@zappos.com - 702-943-7562 >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>> >>>>>> >>>> >>> >>> >> -- >> - Mark >> >> http://www.lucidimagination.com >> >> >> >> >> -- - Mark http://www.lucidimagination.com
