Thanks Emir and Erick! Helps me a lot to understand the commit process. A few more questions:
1. https://lucidworks.com/2013/08/23/understanding- transaction-logs-softcommit-and-commit-in-sorlcloud/ mentions that for soft commit, "new segments are created that will be merged". Does that mean without hard commit, soft commits will create many small segments in memory and that could also slow down query? As I understand merge policy only kicks in with hard commit. 2. Without hard commit configure, will the segments still be fsync to disk when accumulated updates exceeds rambuffersizeMB? Is there any concern to increase rambuffersizeMB to a large value? 3. Can transaction logs be disabled in solr cloud? Will functionalities(replication, peer sync) break without transaction logs? Thanks, Wei On Fri, Sep 29, 2017 at 8:33 AM, Erick Erickson <erickerick...@gmail.com> wrote: > More than you want to know about hard and soft commits here: > https://lucidworks.com/2013/08/23/understanding- > transaction-logs-softcommit-and-commit-in-sorlcloud/ > > You don't need to read it though, Emir did an admirable job of telling > you why turning off hard commits is a terrible idea. > > Best, > Erick > > On Fri, Sep 29, 2017 at 1:07 AM, Emir Arnautović > <emir.arnauto...@sematext.com> wrote: > > Hi Wei, > > Hard commits are about data durability. It will roll over transaction > logs and create index new index segment. If configured with > openSearcher=false, they do not affect query performance much (other then > taking some resources) since they do not invalidate caches. If you have > transaction logs enabled, without hard commits it would grow infinitely and > can result in full disk. In case of heavy indexing, even rare hard commits > can result in large transaction logs causing Solr restart after crash to > take a while because transaction logs are replayed. > > Soft commits are the one that are affecting query performance and should > be as rare as your requirements allow. They invalidate caches causing cold > searches or if you have warming set up, take resources to do the warming. > > > > I would recommend to keep hard commits, set to every 20-60 seconds > (depending on indexing volume) and make sure openSearcher is set to false. > > > > HTH, > > Emir > > > >> On 29 Sep 2017, at 06:55, Wei <weiwan...@gmail.com> wrote: > >> > >> Hello All, > >> > >> What are the impacts if solr cloud is configured to have only soft > commits > >> but no hard commits? In this way if a non-leader node crashes, will it > >> still be able to recover from the leader? Basically we are wondering > in a > >> read heavy & write heavy scenario, whether taking hard commit out could > >> help to improve query performance and what are the consequences. > >> > >> Thanks, > >> Wei > > >
