I see two orthogonal aspects discussed here: - How/where state is tracked (e.g. state.json, PRS directory structure) - How/from where state is updated (Overseer vs distributed, PRS works with either mode)
I think a desirable future has a more efficient state tracking adapted to the actual state access patterns, and not relying on a single/central point of execution (i.e. Overseer) to manipulate the data (given ZK is already such a central point). state.json as an evolution of clusterstate.json was a first step in the right direction, separating per collection state. PRS is another step into separating state so that independent changes can happen... independently. In my opinion, we should explore managing state by splitting a collection's state.json into per shard "sub-state" files where replicas are tracked (shards and replica states), and the main file tracking collection shard info (collection topology). This would allow for example a given replica to only watch its shard sub-state file, and not have that watcher be called for events unrelated to the shard. It could be seen as an alternate implementation of PRS. I also think that (orthogonal to the above), we should reduce our reliance on "synchronous asynchronous state updates" (which really is what happens with all the watchers, they always get notified, and always too late), make the code more robust to handling stale state (detect the staleness, update the local copy). This would reduce the need for constantly watching ZK and reduce the load on ZK. The implementation of DSU was seen (by me at least) as the first step to more fundamentally changing how state is managed, because the distributed code is a lot simpler than the Overseer based update code and makes it easier to reason about staleness and updates (the Overseer code wasn't removed, so no code simplification happened, but if we're clear on the end goal, we should indeed remove all the code that isn't part of it, or doing any change will be too painful/impossible). Once we decide which direction we want to go, we have to remove the code that isn't part of that direction, or the overall complexity makes ambitious refactoring almost impossible. Two hard problems therefore: 1. deciding what we want (likely the hardest of the two), 2. implementing it. Ilan On Fri, Sep 23, 2022 at 7:44 AM Mark Miller <markrmil...@gmail.com> wrote: > It’s hard to see how they are not related. You can look at both as trying > to solve largely the same problem - that the Overseer is mind boggling > inefficient. And it’s hard to see how pursuing one makes sense with the > other. If you were to eliminate the Overseer and essentially distribute > most of its duties to be doable by any individual node, the idea of > efficient state updates is kind of irrelevant in its current form. It’s > somewhat opposing directions given one is, at its core, improving the > Overseer and the other is removing it. And both are basically in their > infancy in terms of what is left on both approaches. It feels to me like > Houston has framed the problem set pretty well. > > On Wed, Sep 21, 2022 at 12:15 PM David Smiley <dsmi...@apache.org> wrote: > > > I think we should have separate threads for these two topics. I'll > start a > > new thread for Distributed State Updates vs Overseer. If they become > > related for certain points then they can be raised here? > > > > > > On Wed, Sep 21, 2022 at 11:44 AM Houston Putman <hous...@apache.org> > > wrote: > > > > > Hey everyone, > > > > > > We've seen some interesting developments over the last 2 years in the > way > > > that Solr state and distributed logic is handled. Notably we've seen > the > > > introduction of PerReplicaStates (PRS) and the Distributed State > Updates > > > (no overseer). > > > > > > I think for the health of our code and future maintainability, we > should > > > really look to decide on what implementations we want to use for State > > > management and Distributed operations. Basically do we want to adopt or > > > abandon PRS/Distributed State Updates. Note that these are separate > > > concepts, so the decision on each will be separate. > > > > > > I bring this up because I see PRS a lot through the code and it feels > > like > > > the code is too separate from the original way of managing state. There > > is > > > a lot of "if (prsEnabled)" logic throughout the core, and its very hard > > to > > > understand how PRS actually works with this logic spread all over the > > > place. If we want to move forward with PRS, then we hopefully would be > > able > > > to consolidate the logic. > > > > > > I don't see the Distributed State Update logic nearly as much, but I > > > imagine our code can only get cleaner with one implementation versus > two. > > > > > > This is just my opinion, let me know what y'all think about making > > > decisions or going forward with the status quo. > > > > > > - Houston > > > > > > -- > - MRM >
