8 GB is much larger than is well supported. Its diminishing returns over 40-100 and mostly a waste of RAM. Too high and things can break. It should be well below 2 GB at most, but I'd still recommend 40-100.
Fuad Efendi wrote: > Reason of having big RAM buffer is lowering frequency of IndexWriter flushes > and (subsequently) lowering frequency of index merge events, and > (subsequently) merging of a few larger files takes less time... especially > if RAM Buffer is intelligent enough (and big enough) to deal with 100 > concurrent updates of existing document without 100-times flushing to disk > of 100 document versions. > > I posted here thread related; I had 1:5 timing for Update:Merge (5 minutes > merge, and 1 minute update) with default SOLR settings (32Mb buffer). I > increased buffer to 8Gb on Master, and it triggered significant indexing > performance boost... > > -Fuad > http://www.linkedin.com/in/liferay > > > >> -----Original Message----- >> From: Mark Miller [mailto:markrmil...@gmail.com] >> Sent: October-23-09 3:03 PM >> To: solr-user@lucene.apache.org >> Subject: Re: Too many open files >> >> I wouldn't use a RAM buffer of a gig - 32-100 is generally a good number. >> >> Fuad Efendi wrote: >> >>> I was partially wrong; this is what Mike McCandless (Lucene-in-Action, >>> > 2nd > >>> edition) explained at Manning forum: >>> >>> mergeFactor of 1000 means you will have up to 1000 segments at each >>> > level. > >>> A level 0 segment means it was flushed directly by IndexWriter. >>> After you have 1000 such segments, they are merged into a single level 1 >>> segment. >>> Once you have 1000 level 1 segments, they are merged into a single level >>> > 2 > >>> segment, etc. >>> So, depending on how many docs you add to your index, you'll could have >>> 1000s of segments w/ mergeFactor=1000. >>> >>> http://www.manning-sandbox.com/thread.jspa?threadID=33784&tstart=0 >>> >>> >>> So, in case of mergeFactor=100 you may have (theoretically) 1000 >>> > segments, > >>> 10-20 files each (depending on schema)... >>> >>> >>> mergeFactor=10 is default setting... ramBufferSizeMB=1024 means that you >>> need at least double Java heap, but you have -Xmx1024m... >>> >>> >>> -Fuad >>> >>> >>> >>> >>>> I am getting too many open files error. >>>> >>>> Usually I test on a server that has 4GB RAM and assigned 1GB for >>>> tomcat(set JAVA_OPTS=-Xms256m -Xmx1024m), ulimit -n is 256 for this >>>> server and has following setting for SolrConfig.xml >>>> >>>> >>>> >>>> <useCompoundFile>true</useCompoundFile> >>>> >>>> <ramBufferSizeMB>1024</ramBufferSizeMB> >>>> >>>> <mergeFactor>100</mergeFactor> >>>> >>>> <maxMergeDocs>2147483647</maxMergeDocs> >>>> >>>> <maxFieldLength>10000</maxFieldLength> >>>> >>>> >>>> >>> >>> >> -- >> - Mark >> >> http://www.lucidimagination.com >> >> >> > > > > -- - Mark http://www.lucidimagination.com
