[
https://issues.apache.org/jira/browse/SOLR-14123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17000250#comment-17000250
]
David Hunt commented on SOLR-14123:
-----------------------------------
I noticed in
[https://github.com/apache/lucene-solr/blob/fa27e476f74bc4ba83e3fcdc39b421bc53a45d16/solr/core/src/java/org/apache/solr/cloud/ZkController.java]
registerLiveNodesListener that it only looks at the top 3 nodes lost. I don't
entirely understand what the code is trying to do but it looks suspicious,
especially when dealing with larger clusters.
> autoAddReplicas is not reliable when multiple nodes go down.
> ------------------------------------------------------------
>
> Key: SOLR-14123
> URL: https://issues.apache.org/jira/browse/SOLR-14123
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: AutoScaling
> Affects Versions: 8.3
> Reporter: David Hunt
> Priority: Major
> Labels: autoscale
>
> I started noticing problems in our production environment with indexing being
> blocked due to a minimum replication factor not being met. We have
> autoAddReplicas triggers in place to add replicas when nodes our lost but it
> doesn't seem to correctly add all replicas that have been lost when nodes are
> lost. I’ve been able to reproduce this behavior consistently in a development
> environment.
> Repro:
> # Setup a 10 node SolrCloud cluster.
> # Create autoAddReplicas to trigger on nodeLost with waitFor set to 10
> minutes.
> # Create 15 collections with 2 shards and 4 replicas.
> # Kill 3 Solr nodes.
> # 15 minutes later kill 1 more Solr node.
> Results:
> Monitor your shards/replicas. You’ll see some replicas added to make up for
> the lost replicas but not all. An hour later many shards are still missing
> replicas.
> Expected:
> All lost replicas should be added on the 6 remaining healthy nodes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
