bq: I am pretty sure that anytime a core starts for *any* reason, all the transaction logs that are present will get replayed.
This isn't quite true. If Solr is shut down gracefully, or a hard commit happened before shutdown (with no new docs added) then the tlog will _not_ be replayed on startup. It's only when Solr is killed without a commit and thus the tlog is not truncated (and segments not closed) by hard commit that the tlog will be replayed on startup. Which is why I strongly recommend that people stop Solr with the script rather than "kill -9". Best, Erick On Thu, Jul 20, 2017 at 5:39 AM, Shawn Heisey <apa...@elyograg.org> wrote: > On 7/18/2017 11:53 AM, suresh pendap wrote: >> After looking at the source code I see that the default values for >> numRecordsToKeep is 100 and maxNumLogsToKeep is 10. >> >> So it seems by default the replica can only have 1000 document updates lag >> before the replica goes for a Full recovery from the leader. > > I don't think that's quite right. In many situations, the number of > documents in the transaction log will likely be less than 1000. > > Enough logs will be kept that *at least* 100 documents are there, if > that can be accomplished with ten logfiles or less. Based on a quick > reading of the code, if the newest ten logs have less than 100 > documents, then there will be less than 100 docs available. This would > not end up being a problem for data integrity, because small infrequent > updates would be the only way to end up with less than 100 docs, and in > that situation, the small number of documents in the transaction log, > when replayed at core startup, will be enough to ensure integrity. > > I think the reasons the default numbers are so small is an attempt to > keep startup time low. I am pretty sure that anytime a core starts for > *any* reason, all the transaction logs that are present will get > replayed. I know for sure that this happens on Solr restart; I think it > also happens on core reload. By keeping the required minimum documents > at a low value like 100, there's a better chance that the transaction > logs will be small, and therefore core startup will be fast. > > On a system where there are no hard commits, all updates end up going > into a single super-large transaction log. This meets the default > configuration numbers, because there are less than ten logs present, and > what is present contains at least 100 documents. Unfortunately, this > means that when the core starts, it will replay that HUGE transaction > log, a process that could take hours. This situation is prevented by > enabling autoCommit with a relatively short maxTime value. Setting > openSearcher to false in the autoCommit ensures that document visibility > behavior is not altered by autoCommit. > > Thanks, > Shawn >
