When SolrCloud is in a steady state (eg the number of nodes in the
cluster is not changing and config is not changing), Solr does not
really talk to ZooKeeper other than really light stuff like a heartbeat
and maintaining a connection. So performance is not likely a large
concern here.
Mostly it's just a hassle because ZooKeeper does not currently support
dynamically changing the nodes in an ensemble without doing a rolling
restart. There are JIRA issues that are being worked on that will help
with this though.
Until then, it's just kind of a pain that some nodes have to be special
or you have to do rolling restarts to make additional nodes part of the
zk quorum.
It's really up to you though - having the services separate just seems
"nicer" to me. Easier to maintain. Often, once you start running
ZooKeeper for one thing, you may end up running other things that use
ZooKeeper as well - many people like to colocate this stuff on a single
dedicated ZooKeeper ensemble.
Embedded will run just fine - we simply recommend the other way to save
headaches. If you know what you are getting into, it's certainly a valid
choice.
- Mark
On 11/11/2012 05:11 PM, Anirudha Jadhav wrote:
let me see if i get this correctly,
greater the no.of zookeeper nodes , more the time it takes to come to a
consensus.
During an indexing operation, how many times does a solr client needs to
contact zookeeper for consensus ?
- per docs ? per commit ? ?
thanks,
Ani
On Sun, Nov 11, 2012 at 11:17 AM, Nick Chase <nch...@earthlink.net> wrote:
Thanks, Jack, this is a great explanation! And since a greater number of
ZK nodes tends to degrade write performance, that would be a factor in
making every Solr node a ZK node as well. Much obliged!
---- Nick
On 11/11/2012 10:45 AM, Jack Krupansky wrote:
"Production" typically implies "high availability" and in a distributed
system the goal is that the overall cluster integrity and performance
should not be compromised just because a few "worker" nodes go down.
Solr nodes do a lot of complex operations and are quite prone to running
into "issues" that compromise their integrity and require that they be
taken down, restarted, etc. In fact, taking down a "bunch" of Solr
"worker" nodes should not be a big deal (unless they are all of the
nodes/replicas from a single shard/slice), while taking down a "bunch"
of zookeepers could be catastrophic to maintaining the integrity of the
zookeeper ensemble. (OTOH, if every Solr node is also a zookeeper node,
a "bunch" of Solr nodes would generally be less than a quorum, so maybe
that is not an absolute issue per se.) Zookeeper nodes are categorically
distinct in terms of their importance to maintaining the integrity and
availability of the overall cluster. They are special in that sense. And
they are special because they are maintaining the integrity of the
cluster's configuration information. Even for large clusters their
number will be relatively "few" compared to the "many" of "worker" nodes
(replicas), so zookeeper nodes need to be "protected" from the vagaries
that can disrupt and take Solr nodes down, not the least of which is
incoming traffic.
I'm not sure what the implications would be if you had a large cluster
and because Zookeeper was embedded you had a large number of zookeepers.
Any of the inter-zookeeper operations would take longer and could be
compromised by even a single busy/overloaded/dead Solr node. OTOH, the
Zookeeper ensemble design is supposed to be able to handle a far number
of missing zookeeper nodes.
OTOH, if high availability is not a requirement for a production cluster
(use case?), then non-embedded zookeepers are certainly an annoyance.
Maybe you could think of embedded zookeeper like every employee having
their manager sitting right next to them all the time. How could that be
anything but a bad idea in terms of maximizing worker output - and
distracting/preventing managers from focusing on their own "work"?
-- Jack Krupansky
-----Original Message----- From: Nick Chase
Sent: Sunday, November 11, 2012 7:12 AM
To: solr-user@lucene.apache.org
Subject: Internal Vs. External ZooKeeper
OK, I can't find a definitive answer on this. The wiki says not to use
the embedded ZooKeeper servers for production. But my question is: why
not? Basically, what are the reasons and circumstances that make you
better off using an external ZooKeeper ensemble?
Thanks...
---- Nick
