Hmm - came out worse than it looked. Here is a better attempt: MergeFactor: 10
BUF DOCS/S 32 37.40 80 39.91 120 40.74 512 38.25 Mark Miller wrote: > Here is an example using the Lucene benchmark package. Indexing 64,000 > wikipedia docs (sorry for the formatting): > > [java] ------------> Report sum by Prefix (MAddDocs) and Round (4 > about 32 out of 256058) > [java] Operation round mrg flush runCnt > recsPerRun rec/s elapsedSec avgUsedMem avgTotalMem > [java] MAddDocs_8000 0 10 32.00MB 8 > 8000 37.40 1,711.22 124,612,472 182,689,792 > [java] MAddDocs_8000 - 1 10 80.00MB - - 8 - - - 8000 - > - 39.91 - 1,603.76 - 266,716,128 - 469,925,888 > [java] MAddDocs_8000 2 10 120.00MB 8 > 8000 40.74 1,571.02 348,059,488 548,233,216 > [java] MAddDocs_8000 - 3 10 512.00MB - - 8 - - - 8000 - > - 38.25 - 1,673.05 - 746,087,808 - 926,089,216 > > After about 32-40, you don't gain much, and it starts decreasing once > you start getting to high. 8GB is a terrible recommendation. > > Also, from the javadoc in IndexWriter: > > * <p> <b>NOTE</b>: because IndexWriter uses > * <code>int</code>s when managing its internal storage, > * the absolute maximum value for this setting is somewhat > * less than 2048 MB. The precise limit depends on > * various factors, such as how large your documents are, > * how many fields have norms, etc., so it's best to set > * this value comfortably under 2048.</p> > > Mark Miller wrote: > >> 8 GB is much larger than is well supported. Its diminishing returns over >> 40-100 and mostly a waste of RAM. Too high and things can break. It >> should be well below 2 GB at most, but I'd still recommend 40-100. >> >> Fuad Efendi wrote: >> >> >>> Reason of having big RAM buffer is lowering frequency of IndexWriter flushes >>> and (subsequently) lowering frequency of index merge events, and >>> (subsequently) merging of a few larger files takes less time... especially >>> if RAM Buffer is intelligent enough (and big enough) to deal with 100 >>> concurrent updates of existing document without 100-times flushing to disk >>> of 100 document versions. >>> >>> I posted here thread related; I had 1:5 timing for Update:Merge (5 minutes >>> merge, and 1 minute update) with default SOLR settings (32Mb buffer). I >>> increased buffer to 8Gb on Master, and it triggered significant indexing >>> performance boost... >>> >>> -Fuad >>> http://www.linkedin.com/in/liferay >>> >>> >>> >>> >>> >>>> -----Original Message----- >>>> From: Mark Miller [mailto:markrmil...@gmail.com] >>>> Sent: October-23-09 3:03 PM >>>> To: solr-user@lucene.apache.org >>>> Subject: Re: Too many open files >>>> >>>> I wouldn't use a RAM buffer of a gig - 32-100 is generally a good number. >>>> >>>> Fuad Efendi wrote: >>>> >>>> >>>> >>>>> I was partially wrong; this is what Mike McCandless (Lucene-in-Action, >>>>> >>>>> >>>>> >>> 2nd >>> >>> >>> >>>>> edition) explained at Manning forum: >>>>> >>>>> mergeFactor of 1000 means you will have up to 1000 segments at each >>>>> >>>>> >>>>> >>> level. >>> >>> >>> >>>>> A level 0 segment means it was flushed directly by IndexWriter. >>>>> After you have 1000 such segments, they are merged into a single level 1 >>>>> segment. >>>>> Once you have 1000 level 1 segments, they are merged into a single level >>>>> >>>>> >>>>> >>> 2 >>> >>> >>> >>>>> segment, etc. >>>>> So, depending on how many docs you add to your index, you'll could have >>>>> 1000s of segments w/ mergeFactor=1000. >>>>> >>>>> http://www.manning-sandbox.com/thread.jspa?threadID=33784&tstart=0 >>>>> >>>>> >>>>> So, in case of mergeFactor=100 you may have (theoretically) 1000 >>>>> >>>>> >>>>> >>> segments, >>> >>> >>> >>>>> 10-20 files each (depending on schema)... >>>>> >>>>> >>>>> mergeFactor=10 is default setting... ramBufferSizeMB=1024 means that you >>>>> need at least double Java heap, but you have -Xmx1024m... >>>>> >>>>> >>>>> -Fuad >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>>> I am getting too many open files error. >>>>>> >>>>>> Usually I test on a server that has 4GB RAM and assigned 1GB for >>>>>> tomcat(set JAVA_OPTS=-Xms256m -Xmx1024m), ulimit -n is 256 for this >>>>>> server and has following setting for SolrConfig.xml >>>>>> >>>>>> >>>>>> >>>>>> <useCompoundFile>true</useCompoundFile> >>>>>> >>>>>> <ramBufferSizeMB>1024</ramBufferSizeMB> >>>>>> >>>>>> <mergeFactor>100</mergeFactor> >>>>>> >>>>>> <maxMergeDocs>2147483647</maxMergeDocs> >>>>>> >>>>>> <maxFieldLength>10000</maxFieldLength> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> >>>> -- >>>> - Mark >>>> >>>> http://www.lucidimagination.com >>>> >>>> >>>> >>>> >>>> >>> >>> >>> >> >> > > > -- - Mark http://www.lucidimagination.com
