Well we're always operating on consensus, just sometimes it's lazy consensus. If the sentiment in the community is unclear, we (should) clarify with a vote before commiting... Ideally it wouldn't get to the point of a veto. At least that's my understanding.
If Pierre comes up with a patch to fix a threading issue we should consider it. If there's a competing patch that should be considered too. If there's no alternate proposal developed enough to create a patch and it looks technically sound, it should go in. May the best patch win. On Tue, Dec 19, 2023 at 11:46 AM David Smiley <dsmi...@apache.org> wrote: > My reply might be a little surprising; maybe I hit "send" too quickly. Of > course one should work to invest in getting more consensus; maybe the idea > isn't fully understood; maybe the concerns aren't fully understood. But > consensus isn't so much a state that is achieved or not; it's shades of > gray. Many people can be silent or not follow-up with a response of any > kind. In the end, no technical change is voted on, there is just the > potential for a veto. Announcing concluding intentions (I'm about to go do > XYZ) is an opportunity for a veto to be expressed. > > On Tue, Dec 19, 2023 at 11:19 AM David Smiley <dsmi...@apache.org> wrote: > > > You may be surprised at what can be accomplished without "consensus" :-). > > Vetoes are the blocker. If you/anyone are convinced enough and put > forth a > > proposal of what you are going to do, get feedback, and say you are going > > to do it (in spite of concerns but obviously try to address them!), go > for > > it. > > > > On Tue, Dec 19, 2023 at 10:45 AM Ilan Ginzburg <ilans...@gmail.com> > wrote: > > > >> The message by Pierre is regarding fixing existing code. > >> > >> The leader on demand doesn't seem to be a short term solution in any > case, > >> and there wasn't really a consensus around the proposal. > >> > >> Ilan > >> > >> On Tue, Dec 19, 2023 at 4:16 PM David Smiley <dsmi...@apache.org> > wrote: > >> > >> > I would be more in favor of going back to the drawing board on leader > >> > election than incremental improvements. Go back to first principles. > >> The > >> > clarity just isn't there to be maintained. I don't trust it. > >> > > >> > Coincidentally I sent a message to the Apache Curator users list > >> yesterday > >> > to inquire about leader prioritization: > >> > https://lists.apache.org/thread/lmm30qpm17cjf4b93jxv0rt3bq99c0sb > >> > I suspect the "users" list is too low activity to be useful for the > >> Curator > >> > project; I'm going to try elsewhere. > >> > > >> > For shards, there doesn't even need to be a "leader election" recipe > >> > because there are no shard leader threads that always need to be > >> > thinking/doing stuff, unlike the Overseer. It could be more > >> demand-driven > >> > (assign leader on-demand if needs to be re-assigned), and thus be more > >> > scalable as well for many shards. > >> > Some of my ideas on this: > >> > https://lists.apache.org/thread/kowcp2ftc132pq0y38g9736m0slchjg7 > >> > > >> > On Mon, Dec 18, 2023 at 11:33 AM Pierre Salagnac < > >> > pierre.salag...@gmail.com> > >> > wrote: > >> > > >> > > We recently had a couple of issues with production clusters because > of > >> > race > >> > > conditions in shard leader election. By race condition here, in mean > >> for > >> > a > >> > > single node. I'm not discussing how leader election is distributed > >> > > across multiple Solr nodes, but how multiple threads in a single > Solr > >> > node > >> > > conflict with each other. > >> > > > >> > > On the overall, when two threads (on the same server) concurrently > >> join > >> > > leader election for the same replica, the outcome is unpredictable. > it > >> > may > >> > > end in two nodes thinking they are the leader or not having any > >> leader at > >> > > all. > >> > > I identified two scenarios, but maybe there are more: > >> > > > >> > > 1. Zookeeper session expires while an election is already in > progress. > >> > > When we re-create the Zookeeper session, we re-register all the > cores, > >> > and > >> > > join elections for all of them. If an election is already > in-progress > >> or > >> > is > >> > > triggered for any reason, we can have two threads on the same Solr > >> server > >> > > node running leader election for the same core. > >> > > > >> > > 2. Command REJOINLEADERELECTION is received twice concurrently for > the > >> > same > >> > > core. > >> > > This scenario is much easier to reproduce with an external client. > It > >> > > occurs for us since we have customizations using this command. > >> > > > >> > > > >> > > The code for leader election hasn't changed much for a while, and I > >> don't > >> > > understand the full history behind it. I wonder whether > multithreading > >> > was > >> > > already discussed and/or taken into account. The code has a "TODO: > >> can we > >> > > even get into this state?" that makes me think this issue was > already > >> > > reproduced but noy fully solved/understood. > >> > > Since this code has many calls to Zookeeper, I don't think we can > just > >> > > "synchronize" it with mutual exclusions, as these calls that involve > >> the > >> > > network can be incredibly slow when something bad happens. We don't > >> want > >> > > any thread to be blocked by another waiting for a remote call to > >> > complete. > >> > > > >> > > I would like to get some opinions about making this code more robust > >> to > >> > > concurrency. Unless the main opinion is "no, this code should > >> actually be > >> > > mono threaded !", I can give it a try. > >> > > > >> > > Thanks > >> > > > >> > > >> > > > -- http://www.needhamsoftware.com (work) https://a.co/d/b2sZLD9 (my fantasy fiction book)
