Let's take a step back. Why do you need to optimize? You said: "As long as I'm not optimizing, search and indexing times are satisfactory." :)
You don't need to optimize just because you are continuously adding and deleting documents. On the contrary! Otis -- Sematext is hiring -- http://sematext.com/about/jobs.html?mls Lucene, Solr, Nutch, Katta, Hadoop, HBase, UIMA, NLP, NER, IR ----- Original Message ---- > From: Jerome L Quinn <jlqu...@us.ibm.com> > To: solr-user@lucene.apache.org > Sent: Thu, November 12, 2009 6:30:42 PM > Subject: Solr 1.3 query and index perf tank during optimize > > > Hi, everyone, this is a problem I've had for quite a while, > and have basically avoided optimizing because of it. However, > eventually we will get to the point where we must delete as > well as add docs continuously. > > I have a Solr 1.3 index with ~4M docs at around 90G. This is a single > instance running inside tomcat 6, so no replication. Merge factor is the > default 10. ramBufferSizeMB is 32. maxWarmingSearchers=4. > autoCommit is set at 3 sec. > > We continually push new data into the index, at somewhere between 1-10 docs > every 10 sec or so. Solr is running on a quad-core 3.0GHz server. > under IBM java 1.6. The index is sitting on a local 15K scsi disk. > There's nothing > else of substance running on the box. > > Optimizing the index takes about 65 min. > > As long as I'm not optimizing, search and indexing times are satisfactory. > > When I start the optimize, I see massive problems with timeouts pushing new > docs > into the index, and search times balloon. A typical search while > optimizing takes > about 1 min instead of a few seconds. > > Can anyone offer me help with fixing the problem? > > Thanks, > Jerry Quinn
