Maybe the problem here is some confusion/ambuguity about the meaning of "down" ?
TL;DR: think of "onlyIfDown" as "onlyIfShutDownCleanly" IIUC, the purpose of the 'onlyIfDown' is a safety valve so (by default) the cluster will prevent you from removing a replica that wasn't shutdown *cleanly* and is officially in a "down" state -- as recorded in the ClusterState for the collection (either the collections state.json or the global clusterstate.json if you have an older solr instance) when you kill -9 a solr node, the replicas that were hosted on that node will typically still be listed in the cluster state as "active" -- but it will *not* be in live_nodes, which is how solr knows that replica can't currently be used (and leader recovery happens as needed, etc...). If, however, you shut the node down cleanly (or if -- for whatever reason -- the node is up, but the replica's SolrCore is not active) then the cluster state will record that replica as "down" Where things unfortunately get confusing, is that the CLUSTERSTATUS api call -- aparently in an attempt to try and implify things -- changes the recorded status of any replica to "down" if that replica is hosted on a node which is not in live_nodes. I suspect that since hte UI uses the CLUSTERSTATUS api to get it's state information, it doesn't display much diff between a replica shut down cleanly and a replica that is hosted on a node which died abruptly. I suspect that's where your confusion is coming from? Ultimately, what onlyIfDown is trying to do is help ensure that you don't accidently delete a replica that you didn't mean to. the opertaing assumption is that the only replicas you will (typically) delete are replicas that you shut down cleanly ... if a replica is down because of a hard crash, then that is an exceptional situation and presumibly you will either: a) try to bring the replica back up; b) delete the replica using onlyIfDown=false to indicate that you know the replica you are deleting isn't 'down' intentionally, but you want do delete it anyway. On Wed, 20 Jul 2016, Erick Erickson wrote: : Date: Wed, 20 Jul 2016 08:26:32 -0700 : From: Erick Erickson <erickerick...@gmail.com> : Reply-To: solr-user@lucene.apache.org : To: solr-user <solr-user@lucene.apache.org> : Subject: Re: Send kill -9 to a node and can not delete down replicas with : onlyIfDown. : : Yes, it's the intended behavior. The whole point of the : onlyIfDown flag was as a safety valve for those : who wanted to be cautious and guard against typos : and the like. : : If you specify onlyIfDown=false and the node still : isn't removed from ZK, it's not right. : : Best, : Erick : : On Tue, Jul 19, 2016 at 10:41 PM, Jerome Yang <jey...@pivotal.io> wrote: : > What I'm doing is to simulate host crashed situation. : > : > Consider this, a host is not connected to the cluster. : > : > So, if a host crashed, I can not delete the down replicas by using : > onlyIfDown='true'. : > But in solr admin ui, it shows down for these replicas. : > And whiteout "onlyIfDown", it still show a failure: : > Delete replica failed: Attempted to remove replica : : > demo.public.tbl/shard0/core_node4 with onlyIfDown='true', but state is : > 'active'. : > : > Is this the right behavior? If a hosts gone, I can not delete replicas in : > this host? : > : > Regards, : > Jerome : > : > On Wed, Jul 20, 2016 at 1:58 AM, Justin Lee <lee.justi...@gmail.com> wrote: : > : >> Thanks for taking the time for the detailed response. I completely get what : >> you are saying. Makes sense. : >> On Tue, Jul 19, 2016 at 10:56 AM Erick Erickson <erickerick...@gmail.com> : >> wrote: : >> : >> > Justin: : >> > : >> > Well, "kill -9" just makes it harder. The original question : >> > was whether a replica being "active" was a bug, and it's : >> > not when you kill -9; the Solr node has no chance to : >> > tell Zookeeper it's going away. ZK does modify : >> > the live_nodes by itself, thus there are checks as : >> > necessary when a replica's state is referenced : >> > whether the node is also in live_nodes. And an : >> > overwhelming amount of the time this is OK, Solr : >> > recovers just fine. : >> > : >> > As far as the write locks are concerned, those are : >> > a Lucene level issue so if you kill Solr at just the : >> > wrong time it's possible that that'll be left over. The : >> > write locks are held for as short a period as possible : >> > by Lucene, but occasionally they can linger if you kill : >> > -9. : >> > : >> > When a replica comes up, if there is a write lock already, it : >> > doesn't just take over; it fails to load instead. : >> > : >> > A kill -9 won't bring the cluster down by itself except : >> > if there are several coincidences. Just don't make : >> > it a habit. For instance, consider if you kill -9 on : >> > two Solrs that happen to contain all of the replicas : >> > for a shard1 for collection1. And you _happen_ to : >> > kill them both at just the wrong time and they both : >> > leave Lucene write locks for those replicas. Now : >> > no replica will come up for shard1 and the collection : >> > is unusable. : >> > : >> > So the shorter form is that using "kill -9" is a poor practice : >> > that exposes you to some risk. The hard-core Solr : >> > guys work extremely had to compensate for this kind : >> > of thing, but kill -9 is a harsh, last-resort option and : >> > shouldn't be part of your regular process. And you should : >> > expect some "interesting" states when you do. And : >> > you should use the bin/solr script to stop Solr : >> > gracefully. : >> > : >> > Best, : >> > Erick : >> > : >> > : >> > On Tue, Jul 19, 2016 at 9:29 AM, Justin Lee <lee.justi...@gmail.com> : >> > wrote: : >> > > Pardon me for hijacking the thread, but I'm curious about something you : >> > > said, Erick. I always thought that the point (in part) of going : >> through : >> > > the pain of using zookeeper and creating replicas was so that the : >> system : >> > > could seamlessly recover from catastrophic failures. Wouldn't an OOM : >> > > condition have a similar effect (or maybe java is better at cleanup on : >> > that : >> > > kind of error)? The reason I ask is that I'm trying to set up a solr : >> > > system that is highly available and I'm a little bit surprised that a : >> > kill : >> > > -9 on one process on one machine could put the entire system in a bad : >> > > state. Is it common to have to address problems like this with manual : >> > > intervention in production systems? Ideally, I'd hope to be able to : >> set : >> > up : >> > > a system where a single node dying a horrible death would never require : >> > > intervention. : >> > > : >> > > On Tue, Jul 19, 2016 at 8:54 AM Erick Erickson < : >> erickerick...@gmail.com> : >> > > wrote: : >> > > : >> > >> First of all, killing with -9 is A Very Bad Idea. You can : >> > >> leave write lock files laying around. You can leave : >> > >> the state in an "interesting" place. You haven't given : >> > >> Solr a chance to tell Zookeeper that it's going away. : >> > >> (which would set the state to "down"). In short : >> > >> when you do this you have to deal with the consequences : >> > >> yourself, one of which is this mismatch between : >> > >> cluster state and live_nodes. : >> > >> : >> > >> Now, that rant done the bin/solr script tries to stop Solr : >> > >> gracefully but issues a kill if solr doesn't stop nicely. Personally : >> > >> I think that timeout should be longer, but that's another story. : >> > >> : >> > >> The onlyIfDown='true' option is there specifically as a : >> > >> safety valve. It was provided for those who want to guard against : >> > >> typos and the like, so just don't specify it and you should be fine. : >> > >> : >> > >> Best, : >> > >> Erick : >> > >> : >> > >> On Mon, Jul 18, 2016 at 11:51 PM, Jerome Yang <jey...@pivotal.io> : >> > wrote: : >> > >> > Hi all, : >> > >> > : >> > >> > Here's the situation. : >> > >> > I'm using solr5.3 in cloud mode. : >> > >> > : >> > >> > I have 4 nodes. : >> > >> > : >> > >> > After use "kill -9 pid-solr-node" to kill 2 nodes. : >> > >> > These replicas in the two nodes still are "ACTIVE" in zookeeper's : >> > >> > state.json. : >> > >> > : >> > >> > The problem is, when I try to delete these down replicas with : >> > >> > parameter onlyIfDown='true'. : >> > >> > It says, : >> > >> > "Delete replica failed: Attempted to remove replica : : >> > >> > demo.public.tbl/shard0/core_node4 with onlyIfDown='true', but state : >> is : >> > >> > 'active'." : >> > >> > : >> > >> > From this link: : >> > >> > < : >> > >> : >> > : >> http://www.solr-start.com/javadoc/solr-lucene/org/apache/solr/common/cloud/Replica.State.html#ACTIVE : >> > >> > : >> > >> > < : >> > >> : >> > : >> http://www.solr-start.com/javadoc/solr-lucene/org/apache/solr/common/cloud/Replica.State.html#ACTIVE : >> > >> > : >> > >> > < : >> > >> : >> > : >> http://www.solr-start.com/javadoc/solr-lucene/org/apache/solr/common/cloud/Replica.State.html#ACTIVE : >> > >> > : >> > >> > < : >> > >> : >> > : >> http://www.solr-start.com/javadoc/solr-lucene/org/apache/solr/common/cloud/Replica.State.html#ACTIVE : >> > >> > : >> > >> > : >> > >> : >> > : >> http://www.solr-start.com/javadoc/solr-lucene/org/apache/solr/common/cloud/Replica.State.html#ACTIVE : >> > >> > : >> > >> > It says: : >> > >> > *NOTE*: when the node the replica is hosted on crashes, the : >> replica's : >> > >> state : >> > >> > may remain ACTIVE in ZK. To determine if the replica is truly : >> active, : >> > you : >> > >> > must also verify that its node : >> > >> > < : >> > >> : >> > : >> http://www.solr-start.com/javadoc/solr-lucene/org/apache/solr/common/cloud/Replica.html#getNodeName-- : >> > >> > : >> > >> > is : >> > >> > under /live_nodes in ZK (or use : >> ClusterState.liveNodesContain(String) : >> > >> > < : >> > >> : >> > : >> http://www.solr-start.com/javadoc/solr-lucene/org/apache/solr/common/cloud/ClusterState.html#liveNodesContain-java.lang.String- : >> > >> > : >> > >> > ). : >> > >> > : >> > >> > So, is this a bug? : >> > >> > : >> > >> > Regards, : >> > >> > Jerome : >> > >> : >> > : >> : -Hoss http://www.lucidworks.com/
