Re: Welcome Matthew Biscocho as Solr Committer

2025-04-30 Thread Ilan Ginzburg
Welcome Matt, and congrats! On Tue, Apr 29, 2025 at 7:35 PM Jason Gerlowski wrote: > > The Project Management Committee (PMC) for Apache Solr has invited > Matthew Biscocho to become a committer and we are pleased to announce > that Matthew > has accepted! > > Matt, the tradition is that new comm

Re: Welcome Pierre Salagnac as Solr Committer!

2024-12-06 Thread Ilan Ginzburg
Welcome and congratulations Pierre! Having worked with you for the last what? 15 years? I'm really happy to see you become a committer. Ilan On Fri, Dec 6, 2024 at 10:31 AM David Smiley wrote: > > The Project Management Committee (PMC) for Apache Solr has invited Pierre > Salagnac to become a co

Re: Moving away from Zookeeper in SolrJ

2024-10-25 Thread Ilan Ginzburg
The current implementation removes non live nodes from the set of nodes to connect to. Getting the live nodes requires connecting to a specific node in the cluster that is therefore live when that happens. Worst case, if there is a single node up in the cluster, the client ends with a single node i

Re: Welcome Christos Malliaridis as Solr Committer

2024-10-23 Thread Ilan Ginzburg
Congratulations and welcome, Christos! On Wed, Oct 23, 2024 at 11:08 PM Tomás Fernández Löbbe < tomasflo...@gmail.com> wrote: > Welcome Christos! > > On Wed, Oct 23, 2024 at 12:35 PM Jason Gerlowski > wrote: > > > Congrats and welcome Christos! > > > > On Sat, Oct 19, 2024 at 8:33 PM Jan Høydahl

Re: transient cores in Solr Cloud

2024-09-12 Thread Ilan Ginzburg
I think the fundamental underlying question is how SolrCloud is run. I’m under the impression that most deployments of SolrCloud tend to use all collections/shards all the time, in which case unloading cores is not overly useful. The use case for transient cores is when different collections or di

Re: Berlin Buzzwords attendance

2024-05-31 Thread Ilan Ginzburg
I'll be there! I'll be talking about a project that uses Solr Learning > to > > > Rank :) > > > > > > https://2024.berlinbuzzwords.de/sessions/?id=WKMGWH > > > > > > On Fri, May 31, 2024 at 10:18 AM Ilan Ginzburg > > wrote: > &

Berlin Buzzwords attendance

2024-05-31 Thread Ilan Ginzburg
Hi Solr devs! I will be attending Berlin Buzzwords this year (June 9-11) and I know a few others here will be present as well. Would be nice to meet in person! 🍻 Ilan

Re: SolrCloud architecture overview doc?

2024-05-31 Thread Ilan Ginzburg
Mark, can you please share the rest of the internal presentation or its general idea? Curious how much (or if) your take on Overseer differs from the "distributed mode" option. Granted, "distributed mode" does not rely on optimistic concurrency (which I get from the lyrics of the song) but on dist

Re: SolrCloud architecture overview doc?

2024-05-30 Thread Ilan Ginzburg
Mike Drob wrote: > > I think Mark has given this as a presentation a few times, maybe > https://www.youtube.com/watch?v=eVK0wLkLw9w > > On Thu, May 30, 2024 at 3:03 AM Ilan Ginzburg wrote: > > > Hi, > > > > I'm looking for a blog post or documentation giv

SolrCloud architecture overview doc?

2024-05-30 Thread Ilan Ginzburg
Hi, I'm looking for a blog post or documentation giving a good overview of the architecture of SolrCloud (not the user experience and use of Solr). The audience would be engineers knowledgeable about distributed systems that know nothing about SolrCloud (side note: I looked for something similar f

Re: Welcome Sanjay Dutt as Solr committer!

2024-05-20 Thread Ilan Ginzburg
Welcome Sanjay and congrats! On Mon, May 20, 2024 at 8:46 PM Jason Gerlowski wrote: > Welcome and congratulations! > > On Mon, May 20, 2024 at 2:04 PM Ishan Chattopadhyaya > wrote: > > > > Welcome Sanjay. Many congratulations! > > > > On Mon, 20 May, 2024, 9:53 pm David Smiley, wrote: > > > >

Re: PRS, important changes needed

2024-05-02 Thread Ilan Ginzburg
There is no plan to remove the non PRS way to manage replica state before making PRS the default way to manage replica state (in addition to the current state.json option) then letting PRS bake for a while with all new deployments (for example a whole release), right? Ilan On Thu, May 2, 2024 a

Re: The design of ClusterStateProvider & ClusterState

2024-04-05 Thread Ilan Ginzburg
I would suggest doing any such change in two independent steps: - Moving classes around without any functional change ("pure" refactoring) - A change to what a class exposes, its behavior etc. Otherwise it is very hard to track what has simply moved and what has changed. Is the principal motivatio

Re: DISCUSS: Optionality of JIRA issues

2024-03-24 Thread Ilan Ginzburg
I do like having a place where a discussion can be had on a code change. Years later it helps. Also, some Jiras get comments or questions long after the code has been merged. If people find that it really slows them down to create a jira, we can create a catch-all jira per released Solr version, r

Re: [DISCUSS] Community Virtual Meetup, February 2024

2024-02-13 Thread Ilan Ginzburg
Thanks for organizing this. I vote *not* 2/20 and *not* 2/22. On Tue 13 Feb 2024 at 22:53, Bruno Roustant wrote: > I vote for 2/23 > > Thanks Jason >

Re: Tracking contributors uniquely

2024-02-13 Thread Ilan Ginzburg
Keep in mind the regulations in some part of the world if you plan to maintain a file containing "full name, primary email, and email aliases" For example https://europa.eu/youreurope/citizens/consumers/internet-telecoms/data-protection-online-privacy/index_en.htm Ilan On Tue, Feb 13, 2024 at 8:1

Re: Overseer, expiring queued messages

2024-02-02 Thread Ilan Ginzburg
for simpler requests. To my knowledge (unless things changed recently and I've missed it) there's no way to cancel an async (or sync, for that matter) Collection API request. Ilan On Fri, Feb 2, 2024 at 7:40 AM David Smiley wrote: > On Thu, Feb 1, 2024 at 1:53 PM Ilan Gi

Re: Overseer, expiring queued messages

2024-02-01 Thread Ilan Ginzburg
I'd be in favor of the Overseer dropping synchronous requests for which the requestor is no longer waiting (ephemeral ZK node is gone). For sync or async requests, we could let the caller set a timeout after which the processing should not start if it hasn't already, or for async messages allow a c

Re: BackupRepository changes

2024-01-31 Thread Ilan Ginzburg
Having used the BackupRepository for other things than backups in the ZERO branch, I suspect there might be two separate concerns that are mixed into the same class/abstraction. One is verifying the data to be persisted is not corrupted (this heavily depends on the type of data being persisted obvi

Re: [DISCUSS] SIP-20: Separation of Compute and Storage in SolrCloud

2024-01-22 Thread Ilan Ginzburg
e sessions for code walkthrough or discussions. Ilan On Tue, Jan 23, 2024 at 1:49 AM Ilan Ginzburg wrote: > To clarify - since you mentioned durability, I'm assuming the commit that >> happens on each update batch is a "hard commit"? >> > Yes. > > If the us

Re: [DISCUSS] SIP-20: Separation of Compute and Storage in SolrCloud

2024-01-22 Thread Ilan Ginzburg
> of that check is a ZooKeeper node read. > > And that ZK read happens on each query? Or does the current strategy have > it check "every X seconds" like how PULL/TLOG replicas check for updates? > > Is that ZK check looking at the "shard term" stuff, or wo

Re: [DISCUSS] SIP-20: Separation of Compute and Storage in SolrCloud

2024-01-17 Thread Ilan Ginzburg
only on the > leader (like TLOG/PULL) or on all replicas (like NRT), or some other > arrangement altogether? > > On the querying side: what situations cause index data to be pulled from > the remote store? > > (These last questions might be a bit lengthy to get into via email,

[DISCUSS] SIP-20: Separation of Compute and Storage in SolrCloud

2024-01-13 Thread Ilan Ginzburg
I have created SIP-20 https://cwiki.apache.org/confluence/display/SOLR/SIP-20%3A+Separation+of+Compute+and+Storage+in+SolrCloud In the next few days I will create a Jira + a branch that implements the SIP proposal and that includes documentation on how to approach that branch and what's in it. Th

Re: multithreading in leader election

2023-12-19 Thread Ilan Ginzburg
The message by Pierre is regarding fixing existing code. The leader on demand doesn't seem to be a short term solution in any case, and there wasn't really a consensus around the proposal. Ilan On Tue, Dec 19, 2023 at 4:16 PM David Smiley wrote: > I would be more in favor of going back to the

Re: multithreading in leader election

2023-12-19 Thread Ilan Ginzburg
I think it's a worthy problem to address given we (we work at the same company) ran into a production incident due to it. Who's familiar and interested enough in leader election code to help review such changes? Thanks, Ilan On Mon, Dec 18, 2023 at 5:33 PM Pierre Salagnac wrote: > We recently h

Re: LBSolrClient and "zombie" check at core level vs node level

2023-11-20 Thread Ilan Ginzburg
I believe state in ZK should be used to decide that an entity is down/unreachable, but given updates to entity states is slow requiring session expiration first, then the time for the watches to notify, it cannot IMO be the only parameter. The proof of the API is in the call (pudding analogy). I

Re: Change active replica counting in tests

2023-11-04 Thread Ilan Ginzburg
ov 4, 2023 at 11:28 PM Ilan Ginzburg wrote: > Created SOLR-17067 <https://issues.apache.org/jira/browse/SOLR-17067> and PR > 1965 <https://github.com/apache/solr/pull/1965> > > On Sat, Nov 4, 2023 at 8:28 PM Ilan Ginzburg wrote: > >> Thanks Hoss for your

Re: Change active replica counting in tests

2023-11-04 Thread Ilan Ginzburg
Created SOLR-17067 <https://issues.apache.org/jira/browse/SOLR-17067> and PR 1965 <https://github.com/apache/solr/pull/1965> On Sat, Nov 4, 2023 at 8:28 PM Ilan Ginzburg wrote: > Thanks Hoss for your input. I was focused on that specific method (to > contribute back a test cha

Re: Change active replica counting in tests

2023-11-04 Thread Ilan Ginzburg
llection, and then fix > activeClusterShape() to pass in collectionState.getActiveSlices() while > leaving clusterShape() alone. > > > : Date: Fri, 3 Nov 2023 18:25:52 +0100 > : From: Ilan Ginzburg > : Reply-To: dev@solr.apache.org > : To: dev@solr.apache.org > : Subject: Chan

Change active replica counting in tests

2023-11-03 Thread Ilan Ginzburg
Hi, I plan to change the method compareActiveReplicaCountsForShards() in SolrCloudTestCase (code link ) that counts the total number of active replicas for a collection to only

Re: Change suggestion: more efficient replica state tracking

2023-10-03 Thread Ilan Ginzburg
> > I wonder if ZK session expiration and re establishment works nicely for > others? The code handling this is in ZkController.onReconnect(). Answering my own question: the issue was specific to our fork so I assume ZK session expiration and re establishment does work nicely in general. On Mon,

Re: Change suggestion: more efficient replica state tracking

2023-10-02 Thread Ilan Ginzburg
Not sure I totally follow what you mean Mark. We thought making actual replica state = published replica state AND node state, which would set practical replica states to down when an ephemeral Zookeeper node for a SolrCloud node disappears. This works nicely for the going down part, but still requ

Re: Change suggestion: more efficient replica state tracking

2023-09-27 Thread Ilan Ginzburg
t; impact. We are running a cluster with many collections/shards and PRS > has been a huge improvement for us in processing nodes going down/up. > > The idea of ephemeral nodes seems interesting, but maybe some added > risk around Zookeeper session expiration and re-establishing replica >

Change suggestion: more efficient replica state tracking

2023-09-26 Thread Ilan Ginzburg
*TL;DR; a way to track replica state using EPHEMERAL nodes that disappear automatically when a node goes down.* Hi, When running a cluster with many collections and replicas per node, processing of DOWNNODE messages takes more time. In a public cloud setup, the node that went down can come back q

Re: Discuss SIP-14 Embedded Zookeeper

2023-09-11 Thread Ilan Ginzburg
Getting rid of standalone Solr mode is not part of SIP-14, I suggest we discuss it separately from the effort to make SolrCloud easier to deploy or the default. On Sat, Sep 9, 2023 at 4:11 PM Ishan Chattopadhyaya < ichattopadhy...@gmail.com> wrote: > +1 fully support this, also support the move f

Re: updateLog tlog directory config silently ignored (SOLR-16962)

2023-09-08 Thread Ilan Ginzburg
Re backcompat issue: might be an option to fix that config but use a different name for it then, so that any currently existing (and ignored) config continues to be ignored. Ilan On Thu, Sep 7, 2023, 6:14 PM Michael Gibney wrote: > I'd like to call some extra attention to SOLR-16962 [1]. I'm fa

Re: Welcome Alex Deparvu as Solr committer

2023-08-02 Thread Ilan Ginzburg
Welcome Alex! On Wed, Aug 2, 2023, 5:52 PM Michael Gibney wrote: > Welcome, Alex, and congrats! > Michael > > On Wed, Aug 2, 2023 at 11:08 AM Alex Deparvu wrote: > > > > Thank you so much Anshum! > > > > I am honored to be invited and very excited to join the community as a > > committer and ho

Re: Java Thread priority tuning for Quality-of-Service

2023-07-23 Thread Ilan Ginzburg
I also think we should gate at the entry point (decide to process a request, delay its processing or reject it) then do it as fast as possible (and potentially decide to reject and abandon during processing if the realization that it should have been rejected happens too late). Slowing down process

Re: SolrCloud separating compute from storage

2023-07-12 Thread Ilan Ginzburg
23-07-19+Meeting+notes> (July 19th). Ilan On Wed, Jul 12, 2023 at 10:54 AM Mikhail Khludnev wrote: > Hello Ilan, > Late comment, though. > > On Fri, Apr 28, 2023 at 8:33 PM Ilan Ginzburg wrote: > > > ... > > We're considering improving this approach by making the

Collection API LIST implementation

2023-05-31 Thread Ilan Ginzburg
In CollectionsHandler LIST_OP, the implementation of LIST loads the state.json of all collections in the cluster in order to return the names of all collections. With a large cluster and a large number of collections, it seems more efficient to directly get all children of /collections from Zookee

Re: SolrCloud separating compute from storage

2023-04-29 Thread Ilan Ginzburg
orders of magnitude more space than in memory, more or less. And we run smaller shard sizes indeed! Thanks everybody for the feedback so far. Ilan On Sat 29 Apr 2023 at 07:08, Shawn Heisey wrote: > On 4/28/23 11:33, Ilan Ginzburg wrote: > > Salesforce has been working for a while on s

SolrCloud separating compute from storage

2023-04-28 Thread Ilan Ginzburg
Hi, This is a long message, apologies. If responses are positive, there will likely be plenty of other opportunities to discuss the topics mentioned here. I'm trying to see if the community would be interested in a contribution allowing SolrCloud nodes to be totally stateless with persistent stor

Re: Welcome Andy Webb as Solr committer

2023-03-07 Thread Ilan Ginzburg
Welcome Andy and congrats! On Tue, Mar 7, 2023 at 7:52 AM Mikhail Khludnev wrote: > Welcome, Andy! > > On Mon, Mar 6, 2023 at 10:54 PM Andy Webb wrote: > > > hi all, thank you for the invitation and welcome messages - this has been > > an unexpected honour! > > > > I'm currently Technical Archi

Re: Shard (slice) leader replica and state.json

2023-02-17 Thread Ilan Ginzburg
nt-guide/collection-management.html#modifycollection-parameters > > > I hope this helps. > > Thanks. > Hitesh. > > On Thu, Feb 16, 2023 at 8:52 AM Ilan Ginzburg wrote: > > > Hi, > > > > We're testing SolrCloud under high scale and high load (

Re: Shard (slice) leader replica and state.json

2023-02-16 Thread Ilan Ginzburg
t; > - > > perReplicaState > > Optional > > Default: false > > If true the states of individual replicas will be maintained as > individual > > child of the state.json. > > --- > > There is also an option to modify this prope

Re: Welcome Justin Sweeney as Solr committer

2023-02-16 Thread Ilan Ginzburg
Welcome Justin! On Tue, Feb 14, 2023 at 2:53 PM Kevin Risden wrote: > Welcome Justin! > > Kevin Risden > > > On Mon, Feb 13, 2023 at 11:58 AM Houston Putman > wrote: > > > Welcome Justin! > > > > - Houston > > > > On Sun, Feb 12, 2023 at 5:56 AM Alessandro Benedetti < > a.benede...@sease.io > >

Re: Welcome Colvin Cowie as Solr committer

2023-02-16 Thread Ilan Ginzburg
Congrats and welcome Colvin! On Wed, Feb 15, 2023 at 1:21 PM Jason Gerlowski wrote: > Welcome Colvin! > > On Tue, Feb 14, 2023 at 2:34 PM Anshum Gupta > wrote: > > > > Congratulations and welcome, Colvin! > > > > On Tue, Feb 14, 2023 at 1:57 AM Colvin Cowie > > > wrote: > > > > > Thanks everyo

Shard (slice) leader replica and state.json

2023-02-16 Thread Ilan Ginzburg
Hi, We're testing SolrCloud under high scale and high load (many replicas per node, multiple collection creations, nodes up and down, backed up Overseer queues) and are *running into shard leader election issues* when state.json and the Zookeeper leader registration node for the shard disagree (le

Re: Focusing on single implementations of core logic

2022-09-23 Thread Ilan Ginzburg
I see two orthogonal aspects discussed here: - How/where state is tracked (e.g. state.json, PRS directory structure) - How/from where state is updated (Overseer vs distributed, PRS works with either mode) I think a desirable future has a more efficient state tracking adapted to the actual state ac

Re: Quarterly Committer Meetings

2022-01-14 Thread Ilan Ginzburg
Can somebody please add me to the confluence page? I plan to attend the committer meeting. (in 1+ year INFRA didn't manage to solve my login issue , and I gave up) Thanks, Ilan On Fri, Jan 14, 2022 at 4:19 AM David Smiley wrote: > I created a w

Re: Custom props to replica state in zk

2021-12-30 Thread Ilan Ginzburg
[Replying to Solr rather than Lucene dev list] If SOLR-15472 takes care of not using the leaders for queries, what would the replica state property be used for? Ilan On Thu, Dec 30, 2021 at 5:40 PM Alex Bulygin wrote: > Hi all! > > Anyone who

Re: First class support for node roles

2021-12-06 Thread Ilan Ginzburg
Ishan, > > Using a string separate from the role definitions (Ishan) makes it too easy > > to have roles for which the default configuration is unknown. > > Ilan, can you please elaborate (perhaps with an example) as to what you mean > here? If the default string for all roles for nodes with no

Re: First class support for node roles

2021-12-06 Thread Ilan Ginzburg
:on". Now, there are no surprises of implicit >>> default. Users who don't use roles at all will get this functionality >>> turned on, just as the developer wanted. Users who use roles will have to >>> explicitly append "ui:on" to their roles string

Re: First class support for node roles

2021-12-05 Thread Ilan Ginzburg
>> >>>>> values: [on, off] >>>>> default: allowed >>>>> >>>>> role name: overseer >>>>> >>>>> values: [allowed, disallowed, preferred] >>>>> default : allowed >>>>> >>

Re: First class support for node roles

2021-12-04 Thread Ilan Ginzburg
>> from them. That’s why I want strict roles up front. It’s maybe ok to push >>> this decision out, but it also seems like the sort of thing we should >>> consider at the start. >>> >>> On Thu, Dec 2, 2021 at 5:52 PM Noble Paul wrote: >>> >&g

Re: First class support for node roles

2021-12-02 Thread Ilan Ginzburg
If we have non strict roles (like overseer), then it does make sense to have negative roles. That way I can define which are the two nodes that I'd prefer the overseer to run on, and a few other nodes on which it should definitely never run for various reasons. And in case these "!overseer" are the

Re: First class support for node roles

2021-11-24 Thread Ilan Ginzburg
>>>>>>>>>>>>>> much), or I am simply mistaken about something. Always feel free >>>>>>>>>>>>>> to ask me to explain! >>>>>>>>>>>>>> >>>>>>>>>>&

Re: First class support for node roles

2021-11-19 Thread Ilan Ginzburg
Is the request here for everybody to express again the concerns already expressed in this email thread and not addressed? I suggest instead the authors review the thread, match expressed concerns with how the concern was addressed (or not addressed) and provide an exhaustive list. This proposal i

Re: First class support for node roles

2021-11-17 Thread Ilan Ginzburg
t;>> Thanks, >>> Ishan >>> >>> On Fri, Nov 12, 2021 at 11:38 PM Gus Heck wrote: >>>> >>>> Yeah we should only be looking for and only be reporting (if we choose to >>>> report to the user) a specific set of env variables. Anythin

Node roles PR question w.r.t. SIP process

2021-11-17 Thread Ilan Ginzburg
I'm not familiar with the SIP process and the meaning of the "[VOTE] SIP-15 Node roles" thread Ishan has started (43 hours ago, so still ~30 hours to go before the stated 72 hours deadline). Does the vote need to end with a "yes" for the node roles code to be added to the codebase, or is that inde

Re: First class support for node roles

2021-11-11 Thread Ilan Ginzburg
Houston made a very valid comment back then on the placement plugin support of environment variables (dropped as a consequence). https://issues.apache.org/jira/browse/SOLR-15019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17286680#comment-17286680 It cou

Re: First class support for node roles

2021-11-04 Thread Ilan Ginzburg
for the user when it comes to roles. This discussion is > about the UX rather than the impl. > > Most of our discussions are about how we should implement it > > > > On Thu, Nov 4, 2021, 9:27 PM Ilan Ginzburg wrote: > >> A lot of the value of this SIP relies on the pseu

Re: First class support for node roles

2021-11-04 Thread Ilan Ginzburg
s should in principle be accessible from ZK whether or not >>>> there's a core for a given collection on a given node). >>>> >>>> Considering the above, and especially given Ishan that you say "The >>>> coordinator role is the biggest motiv

Re: First class support for node roles

2021-11-03 Thread Ilan Ginzburg
;>> If you read more closely, my way can provide full >> back compatibility. To say or imply it doesn't isn't helping. Perhaps you >> need to re-read? >> >>> >> >> >> >>>>>> >> >>> >> >> >>

Future of Overseer

2021-11-02 Thread Ilan Ginzburg
A note about Overseer, since the fact that we want to eventually remove it got mentioned twice in the node roles SIP email thread: in Solr 9 the Overseer is there by default, unchanged, but two config parameters allow to not use it for cluster state updates and collection API command executions (i.

Re: First class support for node roles

2021-11-01 Thread Ilan Ginzburg
On Mon, Nov 1, 2021 at 12:53 PM Ishan Chattopadhyaya < ichattopadhy...@gmail.com> wrote: > I've removed the concept of "!data" from the SIP proposal. A node that > doesn't have -Dnode.roles parameter will be assumed to have > -Dnode.roles=data. If a node is started with a node.roles param, it must

Re: First class support for node roles

2021-10-29 Thread Ilan Ginzburg
o, for the proposal, lets say "data" is a special role which is >>>>> assumed by default, and is enabled on all nodes unless there's a !data. It >>>>> is presumed that data nodes can also serve queries directly, so adding a >>>>> "query"

Re: Leader election in Kube-land

2021-10-28 Thread Ilan Ginzburg
The idea is tempting... Limiting to one tlog replica per shard might not be sufficient though. What if a node has too many shard leaders and we want to rebalance these across the cluster to other nodes? What if a node has some intrinsic issues (runs out of memory each time or unable to start due to

Re: First class support for node roles

2021-10-27 Thread Ilan Ginzburg
In other words, roles are all "positive", but their consequences are only negative (rejecting when the matching positive role is not present). We can also consider no role defined = all roles allowed. Will make things simpler. On Wed, Oct 27, 2021 at 6:14 PM Ilan Ginzburg wrote: &

Re: First class support for node roles

2021-10-27 Thread Ilan Ginzburg
How do we expect the roles to be used? One way I see is a node refusing to do anything related to a role it doesn't have. For example if a node does not have role "data", any attempt to create a core on it would fail. A node not having the role "query", will refuse to have anything to do with handl

Re: Welcome Michael Gibney as Solr committer

2021-10-06 Thread Ilan Ginzburg
Welcome Michael! On Wed, Oct 6, 2021 at 7:53 PM Dawid Weiss wrote: > Hello everyone! > > Please welcome Michael Gibney as the latest Solr committer. Michael > - it's a tradition for you to introduce yourself. I realize the > Lucene/ Solr invitations are in parallel, but you should do it on both

Re: ZkCmdExecutor

2021-10-05 Thread Ilan Ginzburg
release any waiting replicas. > > That core sorter should be looking at LIR to start the leader capable > replicas first. > > Mark > > On Tue, Oct 5, 2021 at 12:10 AM Mark Miller wrote: > >> >> >> On Mon, Oct 4, 2021 at 5:24 AM Ilan Ginzburg wrote:

Re: legacyCloud removal?

2021-10-04 Thread Ilan Ginzburg
Work was done in https://issues.apache.org/jira/browse/SOLR-12823 Ilan On Mon, Oct 4, 2021 at 2:42 PM David Smiley wrote: > For 9, shall we remove logic associated with legacyCloud=true? > https://issues.apache.org/jira/browse/SOLR-8256 this was set to true for > 7.0. Since that is before our

Re: ZkCmdExecutor

2021-10-04 Thread Ilan Ginzburg
Thanks Mark for your write ups! This is an area of SolrCloud I'm currently actively exploring at work (might publish my notes as well at some point). I think terms value (fitness to become leader) should participate in the election node ordering, as well as a terms goal (based on highest known ter

Re: ZkCmdExecutor

2021-09-30 Thread Ilan Ginzburg
Independent of how interactions with ZK are implemented (direct or via Curator), we should first clean up what these interactions do or expect. Take shard leader elector. First a replica is elected, then we check if it is fit for the job, run another election if not, look at other replicas (hopefu

Re: ZkCmdExecutor

2021-09-28 Thread Ilan Ginzburg
that often a smooth event - > this just describes a way I have been able to make it smooth without having > to completely rewrite everything. > > Mark > > On Tue, Sep 28, 2021 at 2:06 AM Ilan Ginzburg wrote: > >> Should ZK disconnect be handled at the individual call

Re: ZkCmdExecutor

2021-09-28 Thread Ilan Ginzburg
Should ZK disconnect be handled at the individual call level to begin with? Aren’t we implementing “recipes” (equivalent to “transactions” in a DB world) that combine multiple actions and that implicitly assume ZK continuity over the course of execution? It seems these should rather fail and retry

Re: PerReplicaStatesIntegrationTest

2021-09-27 Thread Ilan Ginzburg
3fb94c2226dd67aa7/solr/solrj/src/java/org/apache/solr/common/cloud/ZkStateReader.java#L2071 >>>> And I also tried to highlight this... seems maybe not the dev list (I can't >>>> find it now) but at least in JIRA somewhere. >>>> So maybe all ClusterStateProviders

Re: Welcome Ilan Ginzburg to Solr PMC

2021-09-23 Thread Ilan Ginzburg
, 2021 at 3:26 AM Dawid Weiss wrote: > >> >> Congratulations and welcome, Ilan. >> >> On Wed, Sep 22, 2021 at 3:32 PM Ishan Chattopadhyaya < >> ichattopadhy...@gmail.com> wrote: >> >>> I am pleased to announce that Ilan Ginzburg has accepte

Re: PerReplicaStatesIntegrationTest

2021-09-22 Thread Ilan Ginzburg
Not sure Gus I would blame the create collection code. To the best of my recollection, when the create collection call returns the collection IS fully created. This doesn't mean though (and that's the problem IMO) that the cluster state on the node that issued the collection creation call is aware

Re: Welcome András Salamon as Solr committer

2021-09-02 Thread Ilan Ginzburg
Welcome András! On Thu, Sep 2, 2021 at 12:01 AM Jan Høydahl wrote: > Hi all, > > Please join me in welcoming András Salamon as the latest Solr committer. > András, it's tradition for you to introduce yourself with a brief bio. > > Congratulations and Welcome! > Jan >

Re: SolrCloud Alone: Deprecate Standalone Mode

2021-08-12 Thread Ilan Ginzburg
Where I work we currently run most of production on Solr standalone (cluster managed by custom code) on a slightly modified Solr fork. This cluster scales orders of magnitude more than SolrCloud (try having multiple hundreds thousand collections on a 16 nodes SolrCloud cluster, even if you allow yo

Re: luceneMatchVersion and Version classes in Solr 9.0 and beyond

2021-05-07 Thread Ilan Ginzburg
Do we risk not being able to support the Solr N-1 indexes in Solr N (i.e. no easy migration path) if between Solr N-1 and Solr N Lucene has advanced from P to P+2 and P+2 does not support indexes in P format? On Fri, May 7, 2021 at 12:23 AM David Smiley wrote: > Thanks for raising this. I'm not

Re: Proposal to use AffinityPlacementFactory as default in 9.0

2021-04-29 Thread Ilan Ginzburg
: > IIRC it wasn't the nodes calculated, but rather the number of collections > already in the cluster that caused the issue. See > https://issues.apache.org/jira/browse/SOLR-14665 > > On Thu, Apr 29, 2021 at 1:09 PM Ilan Ginzburg wrote: > >> Yes Gus, this was veri

Re: Proposal to use AffinityPlacementFactory as default in 9.0

2021-04-29 Thread Ilan Ginzburg
esult that lead to deprecation of autoscaling? > > On Thu, Apr 29, 2021 at 11:30 AM Ilan Ginzburg wrote: > >> Expliciting (I think) your suggestion from the Slack thread Jan: >> >>- Add support for a new solr.xml config called something like >>forceDefaultLeg

Re: Proposal to use AffinityPlacementFactory as default in 9.0

2021-04-29 Thread Ilan Ginzburg
Expliciting (I think) your suggestion from the Slack thread Jan: - Add support for a new solr.xml config called something like forceDefaultLegacyPlacementStrategy - Do not add anything in solr.xml At runtime: - If a placement plugin is explicitly configured (existing plugin config

Re: Replicas stuck in DOWN state

2021-04-28 Thread Ilan Ginzburg
node has a more recent entry in live_nodes than > the DOWNODE msg then drop it? Not sure if this is at all possible? > > Jan > > 28. apr. 2021 kl. 10:18 skrev Ilan Ginzburg : > > When a SolrCloud node goes down and back up in relatively rapid sequence > (not unusual in Pub

Replicas stuck in DOWN state

2021-04-28 Thread Ilan Ginzburg
When a SolrCloud node goes down and back up in relatively rapid sequence (not unusual in Public Cloud environments), it appears possible that the DOWNNODE cluster state change message gets processed (or completes processing) after the node has restarted. This delayed execution will then mark repl

Re: Welcome Eric Pugh to the Apache Solr PMC

2021-04-24 Thread Ilan Ginzburg
Congrats Eric! Ilan Le sam. 24 avr. 2021 à 17:48, Alexandre Rafalovitch a écrit : > Congratulations Eric, > > Regards, >Alex. > > On Sat, 24 Apr 2021 at 07:14, Eric Pugh > wrote: > > > > Thank you all! I’m excited about the future of Solr, and it’s > potential to solve new use cases in s

Re: Solr2

2021-04-11 Thread Ilan Ginzburg
Do you consider performance/scale of SolrCloud or corectness (shards with no leader, replicas not recovering, non functional Overseer etc) as the most important areas for improvement if we were to move to a v2? Ilan Le sam. 10 avr. 2021 à 22:37, Mark Miller a écrit : > Now a I was saying ... So

Re: Welcome Bruno to the Apache Solr PMC

2021-03-12 Thread Ilan Ginzburg
Congratulations Bruno! On Thu, Mar 11, 2021 at 2:00 AM Mike Drob wrote: > I am pleased to announce that Bruno has accepted an invitation to join the > Solr PMC! > > Congratulations, and welcome aboard! >