Thank you Mark, or should I say MarkGPT :) this was a very informative read!
I have probably inadvertently found myself in the "migrate to Apache
Curator" discussion. while this was an excellent behind the scenes view, I
am hoping that upgrading to a modern zk client will magically fix all of
the a
Oh and the upside to actually getting this right beyond some bug prevention
and minor better behavior, is that the system can weather high load
situations dramatically better. For instance, it’s not uncommon if you are
hammering the system with indexing to start running into exceptions around
losin
And lastly, while you can give up on session loss and not just on session
expiration and in most cases that won’t be detrimental, other cases and zk
recipes can rely on the fact that only session loss and not connection loss
are a showstopper.
A little key piece that’s not super clear in there: this type of
alternative approach allows you to ensure requests against an old session
will get a session expired exception and never succeed.
Apologies is this not exactly clear, I spoke, some AI automatically turned
into text that it found to be clearer, and I pasted…
The concept behind these retries with Zookeeper is to allow for recovery of
lost connections if they happen before the session times out. It is
recommended to only fail a
Hi,
I would like to take a look at the Zk timeout configs, a lot of it is
currently hardcoded.
My plan is to have it based on a system property and go for some
consistency across the board.
There is one confusing bit of code I ran into: the ZkCmdExecutor [0]. this
needs the zk timeout only to
ctly. But it's also the opposite of what ZK
tells you is the right idea, and they are correct. Back off on connection
issues - chill out - let it come back - then continue. That is, among other
reasons, why retrying the way we do with ZkCmdExecutor is also not a good
idea. If you do this, intermi
I think the problem has been a mix of cases. For many, search is not the
sytem of record, so data loss is not great, but not the end of the world.
For others, the number of issues you could run into caused them to take
different approaches. If you can keep the ZooKeeper connection stable
enough, yo
Do typical setups (users) of SolrCloud not care about no data loss in the
cluster, or are clusters maintained stable enough that these issues do not
happen? I feel moving to a cloud infra drives more instability and suspect
these issues becoming more prevalent/problematic then.
Indeed Terms (LIR)
Well, the replicas are still waiting for the leader, so not no wait, you
just don’t have leaders waiting for full shards that lessens the problem. A
leader should have no wait and release any waiting replicas.
That core sorter should be looking at LIR to start the leader capable
replicas first.
M
On Mon, Oct 4, 2021 at 5:24 AM Ilan Ginzburg wrote:
> Thanks Mark for your write ups! This is an area of SolrCloud I'm currently
> actively exploring at work (might publish my notes as well at some point).
>
> I think terms value (fitness to become leader) should participate in the
> election nod
Thanks Mark for your write ups! This is an area of SolrCloud I'm currently
actively exploring at work (might publish my notes as well at some point).
I think terms value (fitness to become leader) should participate in the
election node ordering, as well as a terms goal (based on highest known
ter
Okay, I added some basic suggestions to that leader election Jira.
Between everything I’ve dropped in this thread, I don’t see why anyone
could not fix leader election and leader sync up or come up with good
replacements or make good improvements, so I’ll just leave it at that.
Finally, if anyone
Ilan:
So I have never had any disagreements with your analysis of what does not
work. I have never had any competing designs or approaches. I am not
against different designs.
When I assert this design works and scales, it's mainly to point out that
design is never the problem I've seen here. I'v
I filed
https://issues.apache.org/jira/browse/SOLR-15672 Leader Election is flawed
- for future reference if anyone looks at tackling leader election issues.
I’ll drop a couple notes and random suggestions there
Mark
On Sat, Oct 2, 2021 at 12:47 PM Mark Miller wrote:
> At some point digging thr
At some point digging through some of this stuff, I often start to think, I
wonder how good our tests are at catching certain categories of problems.
Various groups of code branches and behaviors.
I do notice that as I get the test flying, they do start to pick up a lot
more issues. A lot more bug
Tim was referring to code that addresses those issues from the ref branch.
I’ve been trying to remember the items that Ilan has brought up, for reason
I thought this was third one, but I can only come up with shard leader loss
and overseer loss other than leader sync.
I also recalled I have slide
I just want to say that I appreciate the insights you shared over the last
couple days. By "copy paste", was Tim referring to copying your insights
and pasting them into the code? This is what I was thinking. Or at least
some way to make these insights more durable / findable. Could be a link
f
Tim hit me with the obvious question here.
“I’m assuming there are reasons, but what about a little copy past on some
of these issues that you mentioned.”
I say the obvious question because I kind of flippantly jump through some
lines of code and then say and then you just do a, b and c and that’
That covers a lot of current silliness you will see, pretty simply as most
of it comes down remove silly stuff, but you can find some related wildness
in ZkController#register.
// check replica's existence in clusterstate first
zkStateReader.waitForState(collection, 100, TimeUnit.MILLISECONDS,
Ignoring lots of polling, inefficiencies, early defensive raw sleeps,
various races and bugs and a laundry list of items involved in making
leader processes good enough to enter a collection creation contest, here
is a more practical small set of notes off the top of my head on a quick
inspection a
On Thu, Sep 30, 2021 at 3:31 AM Ilan Ginzburg wrote:
> Independent of how interactions with ZK are implemented (direct or via
> Curator), we should first clean up what these interactions do or expect.
>
> Take shard leader elector. First a replica is elected, then we check if it
> is fit for the
Independent of how interactions with ZK are implemented (direct or via
Curator), we should first clean up what these interactions do or expect.
Take shard leader elector. First a replica is elected, then we check if it
is fit for the job, run another election if not, look at other replicas
(hopefu
You actually capture most of the history of cloud there AB.
ZK is the heart of the system. It’s a rare chance you get the time or
financing to lay that out on something that will be used.
I didn’t get it done, changed jobs, and that mostly closed the window on
that.
Then you have a poor heart th
I very much agree. That code is the root of a very surprising amount of
evil and has been for a surprisingly long time.
There is a long list of reasons that I won’t iterate of why I don’t see
that as likely happening though - just starting with Ive brought it up to
various people over a couple yea
+1 to start working towards using Curator, this is long overdue and sooner or
later we need to eat this frog - as you dig deeper and deeper it turns out that
many issues in Solr can be attributed to our home-grown ZK code, there are
maybe 2 people on the Solr team who understand what’s going on
P.S. this is not actually the zookeeper design I would submit to any
competition :)
I’ve gone different routes in addressing the zookeeper short fall. This one
is relatively easy, impactful and isolated for the right developer.
Personally, with fewer scale and isolation limits, by the far the bes
les in mind of where this is problematic
>>> in existing code (or it would already be a bug) but the existing single
>>> call level retry approach feels fragile.
>>>
>>> Ilan
>>>
>>> On Mon 27 Sep 2021 at 19:04, Mark Miller wrote:
>>>
&g
ndividual actions?
>>
>> I don’t have any existing examples in mind of where this is problematic
>> in existing code (or it would already be a bug) but the existing single
>> call level retry approach feels fragile.
>>
>> Ilan
>>
>> On Mon 27 Sep 2021 at
>>
>> The easiest short term change is to simply modify what handles most zk
>> retries - the ZkCmdExecutor - already plugged into SolrZkClient where it
>> retries. It tries to guess when a session times out and does fall back
>> retries up to that point.
>>
>>
e a variety of ways you could do it.
>
> The easiest short term change is to simply modify what handles most zk
> retries - the ZkCmdExecutor - already plugged into SolrZkClient where it
> retries. It tries to guess when a session times out and does fall back
> retries up to that
There are a variety of ways you could do it.
The easiest short term change is to simply modify what handles most zk
retries - the ZkCmdExecutor - already plugged into SolrZkClient where it
retries. It tries to guess when a session times out and does fall back
retries up to that point.
Because
On Wed, Sep 22, 2021 at 9:06 PM Mark Miller wrote:
...
> Zk alerts us when it losses a connection via callback. When the connection
> is back, another callback. An unlimited number of locations trying to work
> this out on there own is terrible zk. In an ideal world, everything enters
> a zk quie
I’m checking that I’m not in some old branch somehow … I’d have sweared
someone got rid of ZkCmdExecutor.
I can’t touch this overseer, I’m dying to see it go, so forgetting about
the fact that it’s insane that it goes to zk like this to deal with
leadership or that it’s half impervious to
34 matches
Mail list logo
