Re: First class support for node roles

2021-12-08 Thread Noble Paul
Jan, We are actually saying the same things, but differently. Today we have only 2 roles - data - overseer We have set default values for them . A user who starts a 9.0 Solr will get the default behavior and it is the same exact behavior as we get in 8x. Imagine you introduce a new role "

Re: First class support for node roles

2021-12-08 Thread Jan Høydahl
> And in some future version it may read an additional property > solr.overseer.priority=3 to give some priority to it, or it could read some > solr.overseer.strict=true to refuse to place overseer strictly on nodes with > the role This sysProp example was just an example, and maybe a bad examp

Re: First class support for node roles

2021-12-08 Thread Jan Høydahl
> > > My proposal is that ALL roles are always ALLOW if not specified explicitly. > > As explained several times before, this is a problem for new roles introduced > in future. > Those roles will get turned on on all nodes after an upgrade, whether a user > wants or not. But you're not hearing

Re: First class support for node roles

2021-12-07 Thread Ishan Chattopadhyaya
> My proposal is that ALL roles are always ALLOW if not specified explicitly. As explained several times before, this is a problem for new roles introduced in future. Those roles will get turned on on all nodes after an upgrade, whether a user wants or not. A user explicitly mentions which roles h

Re: First class support for node roles

2021-12-07 Thread Noble Paul
There is no complexity here. You just think something is complex because you haven't spent any time to understand this. A role is a tag for a node. Period. How complex is that? We just added standard mechanism to declare and discover the nodes. There is no extra cost or complexity On Wed, De

Re: First class support for node roles

2021-12-07 Thread Jan Høydahl
> > Another user with a 100 node cluster who today have three overseer nodes that > they have shielded from having data by specifying createNodeSet manually or > by other means, can choose to adopt rhe role system, and define tree > dedicated nodes with the overseer role but without the data ro

Re: First class support for node roles

2021-12-06 Thread Ishan Chattopadhyaya
Ilan, I missed addressing one aspect in your mail. On Mon, Dec 6, 2021 at 8:37 PM Ilan Ginzburg wrote: > Ishan, > > > > Using a string separate from the role definitions (Ishan) makes it too > easy to have roles for which the default configuration is unknown. > > > > Ilan, can you please elabora

Re: First class support for node roles

2021-12-06 Thread Ishan Chattopadhyaya
Hi all, I think we've captured all recent feedback in the updated proposal. Thanks to Mike and others for helping us streamline the proposal to better suit the current overseer role. Thanks to Ilan and others for the concept of defaultIfAbsent concept. I think the proposal is not only complete, but

Re: First class support for node roles

2021-12-06 Thread Ishan Chattopadhyaya
On Mon, Dec 6, 2021 at 9:18 PM Ishan Chattopadhyaya < ichattopadhy...@gmail.com> wrote: > I've updated the SIP document with the recent changes. Also, added a new > section to provide guidance for adding new roles. > > On Mon, Dec 6, 2021 at 8:37 PM Ilan Ginzburg wrote: > >> Ishan, >> >> > > Usin

Re: First class support for node roles

2021-12-06 Thread Ishan Chattopadhyaya
I've updated the SIP document with the recent changes. Also, added a new section to provide guidance for adding new roles. On Mon, Dec 6, 2021 at 8:37 PM Ilan Ginzburg wrote: > Ishan, > > > > Using a string separate from the role definitions (Ishan) makes it too > easy to have roles for which th

Re: First class support for node roles

2021-12-06 Thread Ilan Ginzburg
Ishan, > > Using a string separate from the role definitions (Ishan) makes it too easy > > to have roles for which the default configuration is unknown. > > Ilan, can you please elaborate (perhaps with an example) as to what you mean > here? If the default string for all roles for nodes with no

Re: First class support for node roles

2021-12-06 Thread Ishan Chattopadhyaya
On Mon, Dec 6, 2021 at 6:38 PM Jan Høydahl wrote: > > > Also, we should not put so much emhasis on "nodes without roles > defined" as if that should be a common way of starting nodes in a huge > cluster. > > > > Jan, the need to tackle "nodes without roles defined" separately is to > cater to tho

Re: First class support for node roles

2021-12-06 Thread Jan Høydahl
> > Also, we should not put so much emhasis on "nodes without roles defined" as > > if that should be a common way of starting nodes in a huge cluster. > > Jan, the need to tackle "nodes without roles defined" separately is to cater > to those users who do not use the roles functionality; we nee

Re: First class support for node roles

2021-12-06 Thread Ishan Chattopadhyaya
> Using a string separate from the role definitions (Ishan) makes it too easy to have roles for which the default configuration is unknown. Ilan, can you please elaborate (perhaps with an example) as to what you mean here? As per my proposal, a node that was started with explicit roles, but withou

Re: First class support for node roles

2021-12-06 Thread Jan Høydahl
Are we making a non-issue into a configuration mess? The overseer's job is diminishing by every version, and we should not fool ourself into believing that a stray overseer will kill an upgrade, and therefore complicate the whole role system. Also, we should not put so much emhasis on "nodes wi

Re: First class support for node roles

2021-12-06 Thread Ilan Ginzburg
Noble got my intention correctly. I think role specific code should only have to deal with the various configuration options for the role. When configuration was binary (role defined or not), then the default is one of the two values, but even then we saw that for data we wanted default true and f

Re: First class support for node roles

2021-12-05 Thread Ishan Chattopadhyaya
Role specific configurations can go into /node_roles/${rolename} znode, and that is outside the scope of this SIP. The concept of role specific modes (eg allowed, preferred for overseer) is a welcome addition to original proposal to model the overseer functionality properly without any confusion to

Re: First class support for node roles

2021-12-05 Thread Mark Miller
> I'm very strongly in favor of not letting users design a system in which the > cluster can be "live" without an overseer. > I understand that the overseer can be taxing to the cluster, That is really just an implementation choice. Bluntly, It doesn't do anything smartwatch could not handle on

Re: First class support for node roles

2021-12-05 Thread Gus Heck
So I think we're loosing sight of the original concept of "default" and conflating it with role configuration. When we started talking about "default roles" the idea was "default" was a flag that indicated if the role was active on a Solr Node where no roles had been specified. Plain and simple. F

Re: First class support for node roles

2021-12-05 Thread Ishan Chattopadhyaya
I think I understand Ilan's motivation for two defaults. Here's a summary of what I understand Ilan's proposal, and a follow up proposal that achieves the similar effect with less perceived complexity to user. *Ilan's proposal (as I understand it):* 1. Every role to have two defaults. Example: da

Re: First class support for node roles

2021-12-05 Thread Noble Paul
Ilan was asking how what should be the overseer role in the following situations a) role=overseer,data:on b) role=overseer: preferred,data:on c) role=data:on I'm saying a shouldn't be valid. Only b & c are valid On Mon, Dec 6, 2021, 12:44 PM Mike Drob wrote: > Ilan, > > Can you provide a

Re: First class support for node roles

2021-12-05 Thread Mike Drob
Ilan, Can you provide a more detailed concrete example? I’m having a lot of trouble understanding what you are proposing, beyond that it is somehow contraindicated with what Ishan/Noble suggest. Apologies for my failure to understand. Thanks, Mike On Sun, Dec 5, 2021 at 5:21 PM Ilan Ginzburg w

Re: First class support for node roles

2021-12-05 Thread Noble Paul
On Mon, Dec 6, 2021, 10:21 AM Ilan Ginzburg wrote: > If we go with optional role params, we need two defaults: > 1. the param value to use when the role is specified without a parameter, > and > 2. the param value to use for the role on a node for which the role is > not specified at all. > > I d

Re: First class support for node roles

2021-12-05 Thread Ilan Ginzburg
If we go with optional role params, we need two defaults: 1. the param value to use when the role is specified without a parameter, and 2. the param value to use for the role on a node for which the role is not specified at all. I don't know how to sensibly name these defaults, but the actual valu

Re: First class support for node roles

2021-12-04 Thread Ishan Chattopadhyaya
I'm +1 on this. It "looks" complicated at first, but simplifies all headaches going forward. On Sun, Dec 5, 2021 at 11:46 AM Noble Paul wrote: > I shall update the SIP proposal if we have a consensus on this > configuration > > On Sun, Dec 5, 2021 at 4:58 PM Noble Paul wrote: > >> >> >> On Sun,

Re: First class support for node roles

2021-12-04 Thread Noble Paul
I shall update the SIP proposal if we have a consensus on this configuration On Sun, Dec 5, 2021 at 4:58 PM Noble Paul wrote: > > > On Sun, Dec 5, 2021 at 4:47 PM Gus Heck wrote: > >> I like this in that it's an example of how the overseer might be extended >> without creating a new role :) >>

Re: First class support for node roles

2021-12-04 Thread Noble Paul
On Sun, Dec 5, 2021 at 4:47 PM Gus Heck wrote: > I like this in that it's an example of how the overseer might be extended > without creating a new role :) > > Not entirely sure if I'm for or against an enum implementation here, but > it makes me a bit nervous. Enums with complexity can quickly g

Re: First class support for node roles

2021-12-04 Thread Gus Heck
I like this in that it's an example of how the overseer might be extended without creating a new role :) Not entirely sure if I'm for or against an enum implementation here, but it makes me a bit nervous. Enums with complexity can quickly get into difficulty for unit tests (especially if one wante

Re: First class support for node roles

2021-12-04 Thread Noble Paul
typo - On Sun, Dec 5, 2021 at 2:37 PM Noble Paul wrote: > I recommend the following format for the role spec > > roles=: > > each role will have an enum of allowed values and a default value > > >- role name: *data* > - values: [*on*, *off]* > - default: *allowed* > > -

Re: First class support for node roles

2021-12-04 Thread Gus Heck
On Sat, Dec 4, 2021 at 7:01 PM Ilan Ginzburg wrote: > If we go with no negative node roles and overseer node role is not strict > (i.e. it’s a "preferred overseer"), then one would need to define a second > node role "no_overseer" to explicitly exclude a node from ever becoming > overseer (which

Re: First class support for node roles

2021-12-04 Thread Noble Paul
I recommend the following format for the role spec roles=: each role will have an enum of allowed values and a default value - role name: *data* - values: [*on*, *off]* - default: *allowed* - role name: *overseer* - values: [*allowed*, *disallowed*, *preferred]* -

Re: First class support for node roles

2021-12-04 Thread Ilan Ginzburg
If we go with no negative node roles and overseer node role is not strict (i.e. it’s a "preferred overseer"), then one would need to define a second node role "no_overseer" to explicitly exclude a node from ever becoming overseer (which I think is a useful feature until we switch the cluster defaul

Re: First class support for node roles

2021-12-03 Thread Gus Heck
I really don't think we should have types of roles. Not negative/positive and not strict/non-strict. You have a role or you don't. What that means is up to the code implementing the role. Roles should be free to configure a preference order (binary, or n-ary or whatever, strict or loose), prohibit

Re: First class support for node roles

2021-12-02 Thread Noble Paul
Negative roles have a place Example is overseer There are 3 possible choices for that role a) preferred: always be in front of the election queue b) on: not preferred, but can be an overseer if no preferred overseer nodes are available c) off: never become an overseer Today we only have options

Re: First class support for node roles

2021-12-02 Thread Mike Drob
Negative roles add a lot of complexity, I would really want to stay away from them. That’s why I want strict roles up front. It’s maybe ok to push this decision out, but it also seems like the sort of thing we should consider at the start. On Thu, Dec 2, 2021 at 5:52 PM Noble Paul wrote: > Yes.

Re: First class support for node roles

2021-12-02 Thread Noble Paul
Yes. Negative roles is not a bad idea. If I start a node for machine learning purposes, I wouldn't want that node to ever participate in overseer election On Fri, Dec 3, 2021, 6:50 AM Ilan Ginzburg wrote: > If we have non strict roles (like overseer), then it does make sense > to have negative r

Re: First class support for node roles

2021-12-02 Thread Ilan Ginzburg
If we have non strict roles (like overseer), then it does make sense to have negative roles. That way I can define which are the two nodes that I'd prefer the overseer to run on, and a few other nodes on which it should definitely never run for various reasons. And in case these "!overseer" are the

Re: First class support for node roles

2021-12-02 Thread Houston Putman
> > With the Strict/Loose option and sensible defaults, users cannot trip >> themselves up by default, but the option is there for people to tinker and >> have an iron grip over their cluster. >> > > +1 to sensible defaults so users don't trip themselves. The option to > tinker for tighter grip can

Re: First class support for node roles

2021-12-01 Thread Ishan Chattopadhyaya
On Thu, Dec 2, 2021 at 1:31 AM Gus Heck wrote: > I think the key is to let the roles have full control of the implications > of having/not having that role. No need for even a strict/loose > designation. The question of do you have the role is yes/no with no logic > to guess if the role is impli

Re: First class support for node roles

2021-12-01 Thread Ishan Chattopadhyaya
On Thu, Dec 2, 2021 at 1:20 AM Houston Putman wrote: > This doesn't really address my concern around what happens if all of our >> existing OVERSEER candidates are down. When at least one of them is up, the >> overseer will go there, and that is good and expected. But what happens if >> all of th

Re: First class support for node roles

2021-12-01 Thread Ishan Chattopadhyaya
On Thu, Dec 2, 2021 at 12:54 AM Mike Drob wrote: > Noble wrote: > > We are not modifying the way the "overseer role" works today. We are > just changing the definition and standardizing the configuration & > discoverability > Ishan wrote: > > As of this SIP, we're not planning to modify the OVERS

Re: First class support for node roles

2021-12-01 Thread Gus Heck
argh bad edit... The question of will it come up with the role is "have_explicit ? use_explicit : use_defaults. On Wed, Dec 1, 2021 at 3:01 PM Gus Heck wrote: > I think the key is to let the roles have full control of the implications > of having/not having that role. No need for even a strict/

Re: First class support for node roles

2021-12-01 Thread Gus Heck
I think the key is to let the roles have full control of the implications of having/not having that role. No need for even a strict/loose designation. The question of do you have the role is yes/no with no logic to guess if the role is implied or not, The question of will it come up with the role

Re: First class support for node roles

2021-12-01 Thread Houston Putman
> > This doesn't really address my concern around what happens if all of our > existing OVERSEER candidates are down. When at least one of them is up, the > overseer will go there, and that is good and expected. But what happens if > all of the overseer eligible nodes are down. Your comment, and th

Re: First class support for node roles

2021-12-01 Thread Mike Drob
Noble wrote: > We are not modifying the way the "overseer role" works today. We are just changing the definition and standardizing the configuration & discoverability Ishan wrote: > As of this SIP, we're not planning to modify the OVERSEER role (which currently stands for preferred overseer). We ca

Re: First class support for node roles

2021-11-30 Thread Ishan Chattopadhyaya
On Tue, Nov 30, 2021 at 12:53 AM Mike Drob wrote: > Replying to the top post in this thread because there has been a lot of > discussion and I don't want to look like I'm continuing any of those > particular threads. > > I finally had time to sit down and think about this with the attention it >

Re: First class support for node roles

2021-11-30 Thread Ishan Chattopadhyaya
On Tue, Nov 30, 2021 at 12:53 AM Mike Drob wrote: > Replying to the top post in this thread because there has been a lot of > discussion and I don't want to look like I'm continuing any of those > particular threads. > > I finally had time to sit down and think about this with the attention it >

Re: First class support for node roles

2021-11-30 Thread Ishan Chattopadhyaya
Hi Mike, Thanks for your thoughtful responses and the structured manner of your composing this mail (with clear labels as to your intent) is a refreshing change. I think we all should learn from this in future communication. On Tue, Nov 30, 2021 at 12:53 AM Mike Drob wrote: > Replying to the top

Re: First class support for node roles

2021-11-30 Thread Noble Paul
On Tue, Nov 30, 2021 at 6:23 AM Mike Drob wrote: > Replying to the top post in this thread because there has been a lot of > discussion and I don't want to look like I'm continuing any of those > particular threads. > > I finally had time to sit down and think about this with the attention it > d

Re: First class support for node roles

2021-11-29 Thread Gus Heck
> > This brings up another point, I’d like to see if we can somehow collapse > overseer election with overseer role advertising I would be strongly in > favor of that. > It would seem natural to move it into this structure, I agree, though that might be done in a follow on ticket. > > Or Zk node

Re: First class support for node roles

2021-11-29 Thread Mike Drob
On Mon, Nov 29, 2021 at 2:40 PM Gus Heck wrote: > > >> CLARIFICATION: I do not like that we are storing node liveness in two >> different places now. We have the live nodes and we have the node roles >> stored in two different places in zookeeper and it feels like this would >> lead to race condi

Re: First class support for node roles

2021-11-29 Thread Gus Heck
> > CLARIFICATION: I do not like that we are storing node liveness in two > different places now. We have the live nodes and we have the node roles > stored in two different places in zookeeper and it feels like this would > lead to race conditions or split brain or other hard to diagnose bugs when

Re: First class support for node roles

2021-11-29 Thread Mike Drob
Replying to the top post in this thread because there has been a lot of discussion and I don't want to look like I'm continuing any of those particular threads. I finally had time to sit down and think about this with the attention it deserves and am generally happy with how the conversation has s

Re: First class support for node roles

2021-11-25 Thread Noble Paul
+1 On Thu, Nov 25, 2021, 6:02 PM Ishan Chattopadhyaya < ichattopadhy...@gmail.com> wrote: > I've started a formal vote thread for the proposal here. Thanks all. > > On Thu, Nov 25, 2021 at 12:10 PM Ishan Chattopadhyaya < > ichattopadhy...@gmail.com> wrote: > >> We have closed the Google Doc for

Re: First class support for node roles

2021-11-24 Thread Ishan Chattopadhyaya
I've started a formal vote thread for the proposal here. Thanks all. On Thu, Nov 25, 2021 at 12:10 PM Ishan Chattopadhyaya < ichattopadhy...@gmail.com> wrote: > We have closed the Google Doc for further comments, and all changes have > been incorporated into the confluence document: > https://cwi

Re: First class support for node roles

2021-11-24 Thread Ishan Chattopadhyaya
We have closed the Google Doc for further comments, and all changes have been incorporated into the confluence document: https://cwiki.apache.org/confluence/display/SOLR/SIP-15+Node+roles Thanks for all the comments and feedback. On Thu, Nov 25, 2021 at 12:09 PM Ishan Chattopadhyaya < ichattopadhy

Re: First class support for node roles

2021-11-24 Thread Ishan Chattopadhyaya
> Maybe one way to provide for the future is to place some requirements on roles, and designate that role-specific coordination information > be placed within the subtree for that role under /node-roles// To facilitate this we could > 1. move the currently designed set of ephemeral nodes down one l

Re: First class support for node roles

2021-11-24 Thread Ishan Chattopadhyaya
On Thu, Nov 25, 2021 at 4:35 AM Gus Heck wrote: > Things I think still need to be clarified to define "Role" explicitly. > Primarily to guide future implementations. Things I can think of include: > >- What the bar for something to be a role vs being more appropriate as >a property/tag ty

Re: First class support for node roles

2021-11-24 Thread Noble Paul
On Thu, Nov 25, 2021 at 10:05 AM Gus Heck wrote: > Things I think still need to be clarified to define "Role" explicitly. > Primarily to guide future implementations. Things I can think of include: > >- What the bar for something to be a role vs being more appropriate as >a property/tag t

Re: First class support for node roles

2021-11-24 Thread Ishan Chattopadhyaya
On Thu, Nov 25, 2021 at 5:25 AM Jan Høydahl wrote: > Very good points to clarify Gus. > > Also, as mentioned in Google Doc, I think the role framework should > clarify the expectations for a node having a role. > Does it mean that the node is eligible for - "MAY" - a certain > feature/task, or do

Re: First class support for node roles

2021-11-24 Thread Jan Høydahl
Very good points to clarify Gus. Also, as mentioned in Google Doc, I think the role framework should clarify the expectations for a node having a role. Does it mean that the node is eligible for - "MAY" - a certain feature/task, or does it mean that the node "MUST" execute that task. I think "MA

Re: First class support for node roles

2021-11-24 Thread Gus Heck
Things I think still need to be clarified to define "Role" explicitly. Primarily to guide future implementations. Things I can think of include: - What the bar for something to be a role vs being more appropriate as a property/tag type concept. Essentially what is and is not expected (at thi

Re: First class support for node roles

2021-11-24 Thread Ishan Chattopadhyaya
> Moreover I think it's a very role specific notion, that is not a good > fit in the roles framework, so maybe should never be part of this SIP. Regarding "capable" vs "currently providing", I agree with Ilan here ^ On Wed, Nov 24, 2021 at 10:38 PM Gus Heck wrote: > You do make a good point t

Re: First class support for node roles

2021-11-24 Thread Gus Heck
You do make a good point that the details can be very role specific. Maybe one way to provide for the future is to place some requirements on roles, and designate that role-specific coordination information be placed within the subtree for that role under /node-roles// To facilitate this we could

Re: First class support for node roles

2021-11-24 Thread Ilan Ginzburg
I agree with Ishan that "currently providing" shouldn't be a current concern for the SIP. Moreover I think it's a very role specific notion, that is not a good fit in the roles framework, so maybe should never be part of this SIP. Even on the Overseer example. Suppose we changed the current design

Re: First class support for node roles

2021-11-24 Thread Gus Heck
So we are potentially talking past each other. Let me re-clarify since you've interpreted "configuration" differently (I think). What I'm referring to in ZK is the "reflection of configuration" not the actual configuration. This is why I'm asking the question of how much info should be retained abo

Re: First class support for node roles

2021-11-24 Thread Ishan Chattopadhyaya
On Wed, Nov 24, 2021 at 9:05 PM Ishan Chattopadhyaya < ichattopadhy...@gmail.com> wrote: > > > On Wed, Nov 24, 2021 at 8:19 PM Gus Heck wrote: > >> Left some additional comments in the google doc, extracting the key point >> for others here so they can know if they want to read the google doc >>

Re: First class support for node roles

2021-11-24 Thread Ishan Chattopadhyaya
On Wed, Nov 24, 2021 at 8:19 PM Gus Heck wrote: > Left some additional comments in the google doc, extracting the key point > for others here so they can know if they want to read the google doc > commentary: > I'm addressing them here, inline. > There is presently disagreement with my proposa

Re: First class support for node roles

2021-11-24 Thread Gus Heck
Left some additional comments in the google doc, extracting the key point for others here so they can know if they want to read the google doc commentary: There is presently disagreement with my proposal that the sip specify both how we track configuration (which I have been referring to as "capab

Re: First class support for node roles

2021-11-23 Thread Ishan Chattopadhyaya
I've synced back all changes to confluence. Both represent the same document as of now. Please review and suggest if there are any outstanding concerns. Thanks for all the feedback. I wish to proceed with the implementation based on lazy consensus again. On Tue, Nov 23, 2021 at 9:38 PM Mike Drob

Re: First class support for node roles

2021-11-23 Thread Mike Drob
I should clarify, that my -1 was specifically about reaching lazy consensus and not about that proposal itself. On Tue, Nov 23, 2021 at 9:42 AM Ishan Chattopadhyaya < ichattopadhy...@gmail.com> wrote: > > -1, I would like to see a proposal on list where it will be permanently > available > > It w

Re: First class support for node roles

2021-11-23 Thread Ishan Chattopadhyaya
> -1, I would like to see a proposal on list where it will be permanently available It will be available in the confluence document (as it currently is). The google docs is just for collecting temporary feedback, thanks to easy inline commenting capability. We promised to consolidate feedback in t

Re: First class support for node roles

2021-11-23 Thread Ishan Chattopadhyaya
> -1, I would like to see a proposal on list where it will be permanently available > in the archives instead of being directed to a Google doc which can be edited at any point in time. > This has been an incredibly difficult proposal to keep track of. This entire SIP process has been incredibly d

Re: First class support for node roles

2021-11-23 Thread Ishan Chattopadhyaya
Okay, sure, I'll rescind my assertion about having reached a lazy consensus. I'll work through all the recent comments and feedback. I think at the last moment, there was an edit by Noble to the Google Doc that hasn't made it into the SIP document, and I'll review that change, the recent comments a

Re: First class support for node roles

2021-11-23 Thread Gus Heck
Also, I don't know that it's defined anywhere, but I feel that declaring lazy consensus on a SIP in less than a week is rushing things. Some folks won't have the flexibility to address things daily and I've been uncomfortably pulled away from paid work trying to keep up with this as is. Lazy consen

Re: First class support for node roles

2021-11-23 Thread Gus Heck
IMHO two things must happen before lazy consensus wait period can begin: 1) The wiki must be updated to reflect the results of discussion in the google doc. 2) The fact that the wiki has been updated, and that you think the discussion is at an end must be posted here, preferably with a summary

Re: First class support for node roles

2021-11-23 Thread Mike Drob
-1, I would like to see a proposal on list where it will be permanently available in the archives instead of being directed to a Google doc which can be edited at any point in time. This has been an incredibly difficult proposal to keep track of. Mike On Tue, Nov 23, 2021 at 6:09 AM Ishan Chatto

Re: First class support for node roles

2021-11-23 Thread Jan Høydahl
There is still discussion and disagreement in the Google doc. The Google doc proposes that zk role structure is 1:1 for all roles -- except the "data" role. I assume the reason for that choice, even if not spelled out in the doc, is to optimize for large clusters with 1000 nodes, and thus avoid

Re: First class support for node roles

2021-11-23 Thread Ishan Chattopadhyaya
This proposal has passed with lazy consensus. We can proceed to the implementation phase. Thanks to everyone for feedback, esp. Gus for the patience. On Sun, Nov 21, 2021 at 2:24 PM Gus Heck wrote: > Thanks for adding that :) and Thanks to Nobel for translating my ravings > :). Sorry about the a

Re: First class support for node roles

2021-11-21 Thread Gus Heck
Thanks for adding that :) and Thanks to Nobel for translating my ravings :). Sorry about the acronym. Amusingly I had to look up gish gallop to understand what you meant :) I can assure you that I will never gish gallop you or anyone on the list, and I find such techniques repugnant. If I'm not mak

Re: First class support for node roles

2021-11-20 Thread Ishan Chattopadhyaya
Added both your suggestions to Google Docs for inline commenting and the SIP document at confluence. Thanks for the feedback. On Sun, Nov 21, 2021 at 10:28 AM Ishan Chattopadhyaya < ichattopadhy...@gmail.com> wrote: > I'll add the lifecycle details to the document as well. > > On Sun, 21 Nov, 202

Re: First class support for node roles

2021-11-20 Thread Ishan Chattopadhyaya
I'll add the lifecycle details to the document as well. On Sun, 21 Nov, 2021, 9:08 am Ishan Chattopadhyaya, < ichattopadhy...@gmail.com> wrote: > Hi Gus, > Thanks for bringing it up again. Initially, I wasn't clear what you meant > and mistook them for gish gallop. > > These were the things I am/

Re: First class support for node roles

2021-11-20 Thread Ishan Chattopadhyaya
Hi Gus, Thanks for bringing it up again. Initially, I wasn't clear what you meant and mistook them for gish gallop. These were the things I am/was confused about (Noble explained it to me what you mean): - adherence to DRY via zk - runtime config info should come from zk - add dynamic features - s

Re: First class support for node roles

2021-11-20 Thread Gus Heck
It is difficult to track everything in this thread for sure. Specifically, I don't feel my email of Wed, Nov 17, 7:43 PM (EST) has been addressed or responded to except for one difference of opinion with Jan on whether or not we should prohibit the notion of any role ever changing at runtime. Note

Re: First class support for node roles

2021-11-20 Thread Ishan Chattopadhyaya
On Fri, 19 Nov, 2021, 7:03 pm Ilan Ginzburg, wrote: > Is the request here for everybody to express again the concerns > already expressed in this email thread and not addressed? > I think the expectation now (after we've expressed our intention to reach a lazy consensus) is that if someone wants

Re: First class support for node roles

2021-11-19 Thread Ilan Ginzburg
Is the request here for everybody to express again the concerns already expressed in this email thread and not addressed? I suggest instead the authors review the thread, match expressed concerns with how the concern was addressed (or not addressed) and provide an exhaustive list. This proposal i

Re: First class support for node roles

2021-11-18 Thread Noble Paul
After so many back and forth mails, I just can't say who has an outstanding concern and if they are already addressed or not. I think the Google doc would help us get clarity on that. Please take a moment to give your inputs On Fri, Nov 19, 2021, 9:18 AM Ishan Chattopadhyaya < ichattopadhy...@gmai

Re: First class support for node roles

2021-11-18 Thread Ishan Chattopadhyaya
Apologies, the vote hasn't passed formally and I was under some confusion on the process. I'd like to proceed with a lazy consensus and proceed to the implementation phase now. However, I would appreciate it if someone wants to bring out any outstanding concerns about the SIP document. To facili

Re: First class support for node roles

2021-11-18 Thread Ishan Chattopadhyaya
The SIP passed the voting phase. Thanks for all for the feedback and insights. Looking forward to your collaboration and reviews as we implement this. On Thu, Nov 18, 2021 at 9:42 PM Ishan Chattopadhyaya < ichattopadhy...@gmail.com> wrote: > > It's fine if we don't provide any ability for runtime

Re: First class support for node roles

2021-11-18 Thread Ishan Chattopadhyaya
> It's fine if we don't provide any ability for runtime modification of roles at this time but I'm leery of precluding it in the future. In future, the necessity for such a facility can dictate our course of action. We cannot lay down rules cast in stone for functionality that we can't foresee yet

Re: First class support for node roles

2021-11-18 Thread Ishan Chattopadhyaya
Thanks Jan, I added both those points to the SIP document in the Notes section. On Thu, Nov 18, 2021 at 7:18 PM Jan Høydahl wrote: > 18. nov. 2021 kl. 01:43 skrev Gus Heck : > > > 2) Roles will not be checked by loading config from disk or caching disk > config in memory. (zk ONLY source of trut

Re: First class support for node roles

2021-11-18 Thread Gus Heck
Code may want to know the role of other nodes (i.e. Someone just sent me a request to create a collection, where can I put it?). DRY... one source of truth is all I'm after. Someone changes the file on disk before restarting, we shouldn't see the change locally but not see it globally, etc. It's f

Re: First class support for node roles

2021-11-18 Thread Jan Høydahl
18. nov. 2021 kl. 01:43 skrev Gus Heck : > > 2) Roles will not be checked by loading config from disk or caching disk > config in memory. (zk ONLY source of truth) > It sounds a bit backward for a local node to first parse solr.node.roles, determine its local set of roles, then publish them to

Re: First class support for node roles

2021-11-17 Thread Noble Paul
On Thu, Nov 18, 2021 at 6:15 AM Ilan Ginzburg wrote: > Current Overseer role defaults to all nodes can, and that changes only > when some nodes get the Overseer role. > The current "overseer" role means "preferred-oveerseer". It was implemented as such because if preferred overseers are not avail

Re: First class support for node roles

2021-11-17 Thread Gus Heck
ack meant to write "default +zookeep4r -overseer" any config should be explicit. On Wed, Nov 17, 2021 at 7:43 PM Gus Heck wrote: > I think we should establish a known set of default roles. Other roles are > non-default roles. New features can be added as default or non-default as > required. D

Re: First class support for node roles

2021-11-17 Thread Gus Heck
I think we should establish a known set of default roles. Other roles are non-default roles. New features can be added as default or non-default as required. Default roles are explicitly loaded at startup if no other information is provided. I'd even be fine with a config that said +zookeeper -over

Re: First class support for node roles

2021-11-17 Thread Ilan Ginzburg
Current Overseer role defaults to all nodes can, and that changes only when some nodes get the Overseer role. This is a default value per node based on the presence of the role on any node of the cluster. But the Overseer role is "weak" in that nodes not having the role can become overseer even if

Re: First class support for node roles

2021-11-17 Thread Ishan Chattopadhyaya
On Wed, Nov 17, 2021 at 8:12 PM Jan Høydahl wrote: > I think your VOTE is premature as several design decisions are obviously > not landed. That may be the reason there are no votes yet, and I'm not > going to vote either. > > >1. If "-Dsolr.node.roles" parameter is not passed, it is implici

Re: First class support for node roles

2021-11-17 Thread Jan Høydahl
I think your VOTE is premature as several design decisions are obviously not landed. That may be the reason there are no votes yet, and I'm not going to vote either. > If "-Dsolr.node.roles" parameter is not passed, it is implicitly assumed to > be "-Dsolr.nodes.role=data" (due to backcompat re

Re: First class support for node roles

2021-11-15 Thread Ishan Chattopadhyaya
Thanks Jan, I've updated the SIP document with all the applicable changes with a link to this thread (which contains the summary at the end). I'll initiate the vote thread now. Thanks to everyone for contributing. On Mon, Nov 15, 2021 at 6:53 PM Jan Høydahl wrote: > Thanks for trying to summariz

  1   2   >