Vaijanath, My suggestion is to:
- turn off autocommit - double-check that mergeFactor is not too high (e.g. higher than 50) - double check your servers open file limit (ulimit -a is the command to run) and if it's low (e.g. 1024) increase it (info about how to do might be on the Wiku, if not Google will help) - don't commit unless you really need searchers to pick up the changes. committing every 1000 docs sounds wrong. Commit at the end, or just optimize (see below) - optimize once at the end of a larger index run. optimize infrequently, as optimization essentially rewrites your whole index, which means a lot of IO work for larger indices Otis -- Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch ----- Original Message ---- > From: Vaijanath N. Rao <[EMAIL PROTECTED]> > To: solr-user@lucene.apache.org > Cc: [EMAIL PROTECTED] > Sent: Monday, May 19, 2008 2:49:01 AM > Subject: Re: Auto commit and optimize settings > > Hi Otis and Solr-users, > > I was under the impression that when one call optimize all the indexes > created so far get's merged. Hence I went about the question on optimize. > > The reason I want optimize is that I have autoCommit feature in the > solrConfig.xml to commit after every 1000 documents. Once I do that I > get too many files open error after some time, while crawling and > indexing a large number of sites. > > Is there a way I can avoid too many files open issue all-together and > yet have index committed after every 1000 docs. > > --Thanks and Regards > Vaijanath > > Otis Gospodnetic wrote: > > Hi, > > > > There is no such option currently and it is not likely that such feature > > will > be added because index optimization is not really a quick and lightweight > operation, so one typically optimized only after the index is fully built and > one knows the index will remain unchanged for a while. If you do need to > optimize periodically for some reason, just send optimize commands to Solr > from > your own application. > > > > > > Otis > > -- > > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch > > > > > > ----- Original Message ---- > > > >> From: Vaijanath N. Rao > >> To: solr-user@lucene.apache.org > >> Cc: [EMAIL PROTECTED] > >> Sent: Monday, May 19, 2008 1:13:03 AM > >> Subject: Auto commit and optimize settings > >> > >> Hi Solr-Users, > >> > >> I have gone through the solrConfig.xml file in the example directory of > >> the solr build (nightly build). I wanted to know is there a way to tell > >> solr to optimize the index after certain number of seconds elapsed or > >> number of records indexed as we do in case of auto-commit. > >> > >> --Thanks and Regards > >> Vaijanath > >> > > > > > >
