The latest Solr 1.4 can index 200k records in several minutes, then commit in a few seconds. I don't know but I'm guessing it is due to Lucene improvements. It does not use much memory doing this.
Lance On Tue, Aug 25, 2009 at 2:43 PM, Fuad Efendi <f...@efendi.ca> wrote: > I do commit once a day, millions of small docs... it takes 20 minutes in > average... why OOM? I see only reduced I/O... > > > -----Original Message----- > From: Edward Capriolo [mailto:edlinuxg...@gmail.com] > Sent: August-25-09 5:35 PM > To: solr-user@lucene.apache.org > Subject: Re: frequency of commit when building index from scratch > > On Tue, Aug 25, 2009 at 5:29 PM, Bill Au<bill.w...@gmail.com> wrote: > > Just curious, how often do folks commit when building their Solr/Lucene > > index from scratch for index with millions of documents? Should I just > wait > > and do a single commit at the end after adding all the documents to the > > index? > > > > Bill > > > > Bill in most cases you probably cannot do one large commit as you will > hit OOM. How many documents can be uncommitted is based on the size of > the documents. Committing every document is slow. I have done a commit > every 10,000 mostly. Results may vary. Someone might have a better > answer then me. > > > -- Lance Norskog goks...@gmail.com
