ramBufferSizeMB limits the amount of memory used for indexing and when it's exceeded the buffer is flushed to disk into a new segment. This is independent of hard/soft commits.
Soft commits do not _force_ the in-memory structures to be written to a segment and do not update the segments file, hard commits do both of these things. For the purposes of that blog I decided it just leads to confusion to mention segments being created wrt soft commits; they're not "real" until a hard commit. bq: If the indexing rate is high, there's not really much difference between soft commits and hard commits Not necessarily true. Soft commits open a new searcher and to all the autowarming, whereas hard commits with openSearcher=false do not. This can be a _very_ expensive operation. Second difference: the segments file is not updated unless you hard commit. So even though I have a bunch of segments written to disk, if I crash my server they won't be found on restart. Graceful shutdown is a different story. At least that's my understanding. Best, Erick On Mon, Oct 2, 2017 at 2:49 AM, alessandro.benedetti <a.benede...@sease.io> wrote: > Hi Erick, > you said : > ""mentions that for soft commit, "new segments are created that will > be merged"" > > Wait, how did that get in there? Ignore it, I'm taking it out. " > > but I think you were not wrong, based on another mailing list thread message > by Shawn, I read : > [1] > > "If you are using the correct DirectoryFactory type, a soft commit has > the *possibility* of not writing to disk, but the amount of memory > reserved is fairly small. > > Looking into the source code for NRTCachingDirectoryFactory, I see that > maxMergeSizeMB defaults to 4, and maxCachedMB defaults to 48. This is a > little bit different than what the javadoc states for > NRTCachingDirectory (5 and 60): > > http://lucene.apache.org/core/6_6_0/core/org/apache/lucene/store/NRTCachingDirectory.html > > The way I read this, assuming the amount of segment data created is > small, only the first few soft commits will be entirely handled in > memory. After that, older segments must be flushed to disk to make room > for new ones. > > If the indexing rate is high, there's not really much difference between > soft commits and hard commits. This also assumes that you have left the > directory at the default of NRTCachingDirectoryFactory. If this has > been changed, then there is no caching in RAM, and soft commit probably > behaves *exactly* the same as hard commit. > " > > [1] > http://lucene.472066.n3.nabble.com/High-disk-write-usage-td4344356.html#a4344551 > > > > ----- > --------------- > Alessandro Benedetti > Search Consultant, R&D Software Engineer, Director > Sease Ltd. - www.sease.io > -- > Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
