Re: number of zookeeper & aws instances

2016-04-13 Thread Daniel Collins
Yeah, sorry, my maths was clearly flawed today, thanks for correcting me Shawn. What I meant was in a 3 ZK setup, if you lose one machine, you are okay, but you are also "at risk", since losing anything else would lose quorum. So in our NRT-style scenario, we would have to get that dead machine ba

Re: number of zookeeper & aws instances

2016-04-13 Thread Shawn Heisey
On 4/13/2016 9:34 AM, Daniel Collins wrote: > Just to chip in, more ZKs are probably only necessary if you are doing NRT > indexing. > > Loss of a single ZK (in a 3 machine setup) will block indexing for the time > it takes to get that machine/instance back up That would NOT block indexing. If yo

Re: number of zookeeper & aws instances

2016-04-13 Thread Daniel Collins
Just to chip in, more ZKs are probably only necessary if you are doing NRT indexing. Loss of a single ZK (in a 3 machine setup) will block indexing for the time it takes to get that machine/instance back up, however it will have less impact on search, since the search side can use the existing sta

RE: number of zookeeper & aws instances

2016-04-13 Thread Garth Grimm
org Subject: Re: number of zookeeper & aws instances Just to chip in, more ZKs are probably only necessary if you are doing NRT indexing. Loss of a single ZK (in a 3 machine setup) will block indexing for the time it takes to get that machine/instance back up, however it will have less impac

Re: number of zookeeper & aws instances

2016-04-13 Thread Erick Erickson
bq: or is it dependent on query load and performance sla's Exactly. The critical bit is that every single replica meets your SLA. By that I mean let's claim that your SLA is 500ms. If you can serve 10 qps at that SLA with one replica/shard (i.e. leader only) you can server 50 QPS by adding 4 more

Re: number of zookeeper & aws instances

2016-04-13 Thread Jay Potharaju
Thanks for the feedback Eric. I am assuming the number of replicas help in load balancing and reliability. That being said are there any recommendation for that, or is it dependent on query load and performance sla's. Any suggestions on aws setup? Thanks > On Apr 13, 2016, at 7:12 AM, Erick Er

Re: number of zookeeper & aws instances

2016-04-13 Thread Erick Erickson
For collections with this few nodes, 3 zookeepers are plenty. From what I've seen people don't go to 5 zookeepers until they have hundreds and hundreds of nodes. 100M docs can fit on 2 shards, I've actually seen many more. That said, if the docs are very large and/or the searchers are complex perf

number of zookeeper & aws instances

2016-04-13 Thread Jay Potharaju
Hi, In my current setup I have about 30 million docs which will grow to 100 million by the end of the year. In order to accommodate scaling and query load, i am planning to have atleast 2 shards and 2/3 replicas to begin with. With the above solrcloud setup I plan to have 3 zookeepers in the quoru