I infer that this statement: "takes a while to recover before cloud becomes green" indicates that the node is in recovery or something while indexing. If you're still indexing, the new documents will be written to the followers tlog while the follower is recovering, leading to it growing. I expect that after followers all recover, the tlog shrinks after a few commits have gone by.
If that's all true, the question is why the follower goes into recovery in the first place. Prior to 5.2, there was a situation in which very heavy indexing could cause a follower to go into Leader Initiated Recovery (LIR) (look for this in both the leader and follower logs). Here's the blog Tim Potter wrote on this subject: https://lucidworks.com/blog/2015/06/10/indexing-performance-solr-5-2-now-twice-fast/ The smoking gun here is 1> heavy indexing is required 2> the _leader_ stays up 3> the _follower_ goes into recovery for no readily apparent reason 4> the nail in the coffin for this particular issue is seeing that the follower went into LIR. 5> You'll also see a very large number of threads on the leader waiting on sending the updates to the follower. If this is a problem, prior to 5.2 there are really only two solutions 1> throttle indexing 2> take all of the followers offline during indexing. When indexing is completed, bring the followers back up and let them replicate the full index down from the leader. Best, Erick On Fri, Oct 30, 2015 at 8:28 AM, Rallavagu <rallav...@gmail.com> wrote: > 4.10.4 solr cloud, 3 zk quorum, jdk 8 > > autocommit: 15 sec, softcommit: 2 min > > Under heavy indexing load with above settings, i have seen tlog growing > (into GB). After the updates stopped coming in, it settles down and takes a > while to recover before cloud becomes "green". > > With 15 second autocommit setting, what could potentially cause tlog to > grow? What to look for?
