If hard commit max time is 300 sec then commit happens every 300 sec on tlog leader. And new segments pop up on the leader every 300 sec, during indexing. Polling interval on other replicas 150 sec, but not every poll attempt they fetch new segment from the leader, afaiu. Erick, do you mean that on all other tlog replicas(not leaders) commit occurs every poll? воскресенье, 09 декабря 2018г., 19:21 +03:00 от Erick Erickson erickerick...@gmail.com :
>Not quite, 600000. The polling interval is half the commit interval.... > >This has always bothered me a little bit, I wonder at the utility of a >config param. We already have old-style replication with a >configurable polling interval. Under very heavy indexing loads, it >seems to me that either the tlogs will grow quite large or we'll be >pulling a lot of unnecessary segments across the wire, segments >that'll soon be merged away and the merged segment re-pulled. > >Apparently, though, nobody's seen this "in the wild", so it's >theoretical at this point. >On Sun, Dec 9, 2018 at 1:48 AM Vadim Ivanov < vadim.iva...@spb.ntk-intourist.ru> wrote: > > Thanks, Edward, for clues. > What bothers me is newSearcher start, warming, cache clear... all that CPU > consuming stuff in my heavy-indexing scenario. > With NRT I had autoSoftCommit: 300000 . > So I had new Searcher no more than every 5 min on every replica. > To have more or less the same effect with TLOG - PULL collection, > I suppose, I have to have : 300000 > (yes, I understand that newSearchers start asynchronously on leader and > replicas) > Am I right? > -- > Vadim > > >> -----Original Message----- >> From: Edward Ribeiro [mailto:edward.ribe...@gmail.com] >> Sent: Sunday, December 09, 2018 12:42 AM >> To: solr-user@lucene.apache.org >> Subject: Re: Soft commit and new replica types >> >> Some insights in the new replica types below: >> >> On Sat, December 8, 2018 08:42, Vadim Ivanov < >> vadim.iva...@spb.ntk-intourist.ru wrote: >> >>> >>> From Ref guide we have: >>> " NRT is the only type of replica that supports soft-commits..." >>> "If TLOG replica does become a leader, it will behave the same as if it >>> was a NRT type of replica." >>> Does it mean, that if we do not have NRT replicas in the cluster then >>> autoSoftCommit section in solconfig.xml Ignored completely (even on TLOG >>> leader)? >>> >> >> No, not completely. Both TLOG and PULL nodes will periodically poll the >> leader for changes in index segments' files and download those segments >> from the leader. If hard commit max time is defined in solrconfig.xml the >> polling interval of each replica will be half that value. Or else if the >> soft commit max time is defined then the replicas will use half the soft >> commit max time as the interval. If neither are defined then the poll >> interval will be 3 seconds (hard coded). See here: >> https://github.com/apache/lucene- >> solr/blob/75b183196798232aa6f2dcaaaab117f309119053/solr/core/src/java/o >> rg/apache/solr/cloud/ReplicateFromLeader.java#L68-L77 >> >> If the TLOG is the leader it will index locally and append the doc to >> transaction log as a NRT node would do as well as it will synchronously >> replicate the data to other TLOG replicas' transaction logs (PULL nodes >> don't have transaction logs). But TLOG/PULL replicas doesn't support soft >> commits nor real time gets, afaik. >> >>> >> >>> >>> 60000 >>> >>> >>> Should we say that in autoCommit section openSearcher is always true in >>> that case? >> >> >> >> 10000 >> 30000 >> 512m >> false >> >> >> Does it mean that new Searcher always starts on all replicas when hard >> commit happens on leader? >> >> >> Nope. Or at least, the searcher is not synchronously created. Each non >> leader replica will periodically fetch the index changes from the leader >> and open a new searcher to reflect those changes as seen here: >> https://github.com/apache/lucene- >> solr/blob/75b183196798232aa6f2dcaaaab117f309119053/solr/core/src/java/o >> rg/apache/solr/handler/IndexFetcher.java#L653 >> But it's important to note that the potential delay between the leader's >> hard commit and the other replicas fetching those changes from the leader >> and opening a new searcher to reflect latest changes. >> >> PS: I am still digging these new replica types so I can have misunderstood >> or missed some aspect of it. >> >> Regards, >> Edward >
