[
https://issues.apache.org/jira/browse/SOLR-14788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17229071#comment-17229071
]
Shawn Heisey commented on SOLR-14788:
-------------------------------------
Mark, I have something of extremely narrow focus that I would like to bring up.
I mentioned it briefly on SOLR-7191 ... now that we are seeing real work
towards a major overhaul, I think it's worth mentioning again. I don't know if
anyone is going to agree with me. :)
One of the problems noted with the overseer is that it is very easy for its
work queue to accrue so many items that the zookeeper max packet length
(default about one megabyte) is exceeded.
Increasing the folder depth of messages in the overseer work queue by one level
will allow the default ZK packet size to work successfully with over one
billion items in the queue. It would be accomplished by creating folders
inside the queue and limiting the number of items in each folder to 32768.
32768 folders that each hold 32768 messages is over one billion total messages,
and reading it should fit inside the default ZK packet length.
Naturally such a new method of overseer operation should require a config
option to enable, or there would be backward-compatibility issues with existing
releases. Or maybe there's a way to detect the Solr version of all nodes in
the cloud and automatically enable it if they are all new enough.
It is arguable that situations which accumulate so many overseer work items are
themselves something that should be prevented. Maybe your rewrite of the
overseer takes care of that ... I haven't looked. If a work queue is still
part of the new overseer design, then I think it should be modified as I
described above. Or we could hope that ZOOKEEPER-1162 is taken seriously and
fixed.
I wish I had a clue about the code that makes SolrCloud go. I don't know where
to begin looking.
> Solr: The Next Big Thing
> ------------------------
>
> Key: SOLR-14788
> URL: https://issues.apache.org/jira/browse/SOLR-14788
> Project: Solr
> Issue Type: Task
> Reporter: Mark Robert Miller
> Assignee: Mark Robert Miller
> Priority: Critical
>
> h3.
> [!https://www.unicode.org/consortium/aacimg/1F46E.png!|https://www.unicode.org/consortium/adopted-characters.html#b1F46E]{color:#00875a}*The
> Policeman is on duty!*{color}
> {quote}_{color:#de350b}*When The Policeman is on duty, sit back, relax, and
> have some fun. Try to make some progress. Don't stress too much about the
> impact of your changes or maintaining stability and performance and
> correctness so much. Until the end of phase 1, I've got your back. I have a
> variety of tools and contraptions I have been building over the years and I
> will continue training them on this branch. I will review your changes and
> peer out across the land and course correct where needed. As Mike D will be
> thinking, "Sounds like a bottleneck Mark." And indeed it will be to some
> extent. Which is why once stage one is completed, I will flip The Policeman
> to off duty. When off duty, I'm always* {color:#de350b}*occasionally*{color}
> *down for some vigilante justice, but I won't be walking the beat, all that
> stuff about sit back and relax goes out the window.*{color}_
> {quote}
>
> I have stolen this title from Ishan or Noble and Ishan.
> This issue is meant to capture the work of a small team that is forming to
> push Solr and SolrCloud to the next phase.
> I have kicked off the work with an effort to create a very fast and solid
> base. That work is not 100% done, but it's ready to join the fight.
> Tim Potter has started giving me a tremendous hand in finishing up. Ishan and
> Noble have already contributed support and testing and have plans for
> additional work to shore up some of our current shortcomings.
> Others have expressed an interest in helping and hopefully they will pop up
> here as well.
> Let's organize and discuss our efforts here and in various sub issues.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
