Agreed, commit every second. Assuming I understand what you're saying correctly: There shouldn't be any index readers - as at this point, just writing to the index. Did I understand correctly what you meant?
-Nick -----Original Message----- From: Lance Norskog [mailto:goks...@gmail.com] Sent: 13 January 2010 22:28 To: solr-user@lucene.apache.org Subject: Re: Interesting OutOfMemoryError on a 170M index The time in autocommit is in milliseconds. You are committing every second while indexing. This then causes a build-up of sucessive index readers that absorb each commit, which is probably the out-of-memory. On Wed, Jan 13, 2010 at 10:36 AM, Minutello, Nick <nick.minute...@credit-suisse.com> wrote: > > Hi, > > I have a bit of an interesting OutOfMemoryError that I'm trying to > figure out. > > My client & Solr server are running in the same JVM (for deployment > simplicity). FWIW, I'm using Jetty to host Solr. I'm using the > supplied code for the http-based client interface. Solr 1.3.0. > > My app is adding about 20,000 documents per minute to the index - one > at a time (it is listening to an event stream and for every event, it > adds a new document to the index). > The size of the documents, however, is tiny - the total index growth > is only about 170M (after about 1 hr and the OutOfMemoryError) At this > point, there is zero querying happening - just updates to the index > (only adding documents, no updates or deletes) After about an hour or > so, my JVM runs out of heap space - and if I look at the memory > utilisation over time, it looks like a classic memory leak. It slowly > ramps up until we end up with constant FULL GC's and eventual OOME. > Max heap space is 512M. > > In Solr, I'm using autocommit (to buffer the updates) > <autoCommit> > <maxDocs>10000</maxDocs> > <maxTime>1000</maxTime> > </autoCommit> > > (Aside: Now, I'm not sure if I am meant to call commit or not on the > client SolrServer class if I am using autocommit - but as it turns > out, I get OOME whether I do that or not) > > Any suggestions/advice of quick things to check before I dust off the > profiler? > > Thanks in advance. > > Cheers, > Nick > > ====================================================================== > ========= > Please access the attached hyperlink for an important electronic > communications disclaimer: > http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html > > ====================================================================== > ========= > > -- Lance Norskog goks...@gmail.com =============================================================================== Please access the attached hyperlink for an important electronic communications disclaimer: http://www.credit-suisse.com/legal/en/disclaimer_email_ib.html ===============================================================================
