Thanks Erick for you attention!
My comments below, but supposing that the problem resides in zookeeper
I'll collect more information from zk logs and solr logs and be back soon.
> bq. I've noticed that some replicas stop receiving updates from the
> leader without any visible signs from the cluster status.
>
> Hmm, yes, this isn't expected at all. What are you seeing that causes
> you to say this? You'd have to be monitoring the log for update
> messages to the replicas that aren't leaders or the like. If anyone is
> going to have a prayer of reproducing we'll need more info on exactly
> what you're seeing and how you're measuring this.
Meanwhile, I have log level WARN... I'l decrease it to INFO and see. Tnx
>
> Have you changed any configurations in your replicas at all? We'd need
> the exact steps you performed if so.
Command to create replicas was like this (implicit sharding and custom CoreName
) :
mysolr07:8983/solr/admin/collections?action=ADDREPLICA
&collection=rpk94
&shard=rpk94_1_0
&property.name=rpk94_1_0_07
&type=tlog
&node=mysolr07:8983_solr
>
> On a quick test I didn't see this, but if it were that easy to
> reproduce I'd expect it to have shown up before.
Yesterday I've tried to reproduce... trying to change leader with
REBALANCELEADERS command.
It ended up with no leader at all for the shard and I could not set leader at
all for a long time.
There was a problem trying to register as the
leader:org.apache.solr.common.SolrException: Could not register as the leader
because creating the ephemeral registration node in ZooKeeper failed
...
Deleting duplicate registration:
/collections/rpk94/leader_elect/rpk94_1_117/election/2983181187899523085-core_node73-n_0000000022
...
Index fetch failed :org.apache.solr.common.SolrException: No registered
leader was found after waiting for 4000ms , collection: rpk94 slice: rpk94_1_117
...
Even to delete all replicas for the shard and recreate Replica to the same node
with the same name did not help - no leader for that shard.
I had to delete collection, wait till morning and then it recreated
successfully.
Suppose some weird znodes were deleted from zk by morning.
>
> NOTE: just looking at the cloud graph and having a node be active is
> not _necessarily_ sufficient for the node to be up to date. It
> _should_ be sufficient if (and only if) the node was shut down
> gracefully, but a "kill -9" or similar doesn't give the replicas on
> the node the opportunity to change the state. The "live_nodes" znode
> in ZooKeeper must also contain the node the replica resides on.
Node was live, cluster was healthy
>
> If you see this state again, you could try pinging the node directly,
> does it respond? Your URL should look something like:
> http://host:port/solr/colection_shard1_replica_t1/query?q=*:*&distrib=false
Yes, sure I did. Ill replica responded and number of documents differs with the
leader
>
> The "distrib=false" is important as it won't forward the query to any
> other replica. If what you're reporting is really happening, that node
> should respond with a document count different from other nodes.
>
> NOTE: there's a delay between the time the leader indexes a doc and
> it's visible on the follower. Are you sure you're waiting for
> leader_commit_interval+polling_interval+autowarm_time before
> concluding that there's a problem? I'm a bit suspicious that checking
> the versions is concluding that your indexes are out of sync when
> really they're just catching up normally. If it's at all possible to
> turn off indexing for a few minutes when this happens and everything
> just gets better then it's not really a problem.
Sure, the problem was on many shards but not on all shards
and for the long time.
>
> If we prove out that this is really happening as you think, then a
> JIRA (with steps to reproduce) is _definitely_ in order.
>
> Best,
> Erick
> On Wed, Oct 24, 2018 at 2:07 AM Vadim Ivanov
> <vadim.iva...@spb.ntk-intourist.ru> wrote:
> >
> > Hi All !
> >
> > I'm testing Solr 7.5 with TLOG replicas on SolrCloud with 5 nodes.
> >
> > My collection has shards and every shard has 3 TLOG replicas on different
> > nodes.
> >
> > I've noticed that some replicas stop receiving updates from the leader
> > without any visible signs from the cluster status.
> >
> > (all replicas active and green in Admin UI CLOUD graph). But indexversion of
> > 'ill' replica not increasing with the leader.
> >
> > It seems to be dangerous, because that 'ill' replica could become a leader
> > after restart of the nodes and I already experienced data loss.
> >
> > I didn't notice any meaningfull records in solr log, except that probably
> > problem occurs when leader changes.
> >
> > Meanwhile, I monitor indexversion of all replicas in a cluster by mbeans and
> > recreate ill replicas when difference with the leader indexversion more
> > than one
> >
> > Any suggestions?
> >
> > --
> >
> > Best regards, Vadim
> >
> >
> >
