Re: solr 6 at scale

2017-05-25 Thread Toke Eskildsen
On Thu, 2017-05-25 at 15:56 -0700, Nawab Zada Asad Iqbal wrote: > I have 31 machine cluster with 3 shards on each (93 shards). Each > machine has 250~GB ram and 3TB SSD for search index (there is another > drive for OS and stuff). One solr process runs for each shard with > 48G heap. So we have 3 l

Re: solr 6 at scale

2017-05-25 Thread Nawab Zada Asad Iqbal
Hi Toke, I don't have any blog, but here is a high level idea: I have 31 machine cluster with 3 shards on each (93 shards). Each machine has 250~GB ram and 3TB SSD for search index (there is another drive for OS and stuff). One solr process runs for each shard with 48G heap. So we have 3 large fi

Re: solr 6 at scale

2017-05-25 Thread Bram Van Dam
>>> It is relatively easy to downgrade to an earlier release within the >>> same major version. We have not switched to 6.5.1 simply because we >>> have no pressing need for it - Solr 6.3 works well for us. > >> That strikes me as a little bit dangerous, unless your indexes are very >> static. Th

Re: solr 6 at scale

2017-05-24 Thread Toke Eskildsen
Nawab Zada Asad Iqbal wrote: > @Toke, I stumbled upon your page last week but it seems that your huge > index doesn't receive a lot of query traffic. It switches between two kinds of usage: Everyday use is very low traffic by researchers using it interactively: 1-2 simultaneous queries, with fa

Re: solr 6 at scale

2017-05-24 Thread Walter Underwood
I remembered why we waited for 6.5.1. It is the object leak in the Zookeeper client code. A very slow leak, but worth getting a fix. I tested our cluster at 6000 requests/minute. It is 18 million documents, four shards by four replicas on big AWS instances (c4.8xlarge). We have very long free t

Re: solr 6 at scale

2017-05-24 Thread Nawab Zada Asad Iqbal
Thanks everyone for the responses, I will go with the latest bits for now; and will share how it goes. @Toke, I stumbled upon your page last week but it seems that your huge index doesn't receive a lot of query traffic. Mine is around 60TB and receives around 120 queries per second; ~90 shards on

Re: solr 6 at scale

2017-05-24 Thread Toke Eskildsen
Shawn Heisey wrote: > On 5/24/2017 3:44 AM, Toke Eskildsen wrote: >> It is relatively easy to downgrade to an earlier release within the >> same major version. We have not switched to 6.5.1 simply because we >> have no pressing need for it - Solr 6.3 works well for us. > That strikes me as a litt

Re: solr 6 at scale

2017-05-24 Thread Shawn Heisey
On 5/24/2017 3:44 AM, Toke Eskildsen wrote: > It is relatively easy to downgrade to an earlier release within the > same major version. We have not switched to 6.5.1 simply because we > have no pressing need for it - Solr 6.3 works well for us. That strikes me as a little bit dangerous, unless yo

Re: solr 6 at scale

2017-05-24 Thread Toke Eskildsen
On Tue, 2017-05-23 at 17:27 -0700, Nawab Zada Asad Iqbal wrote: > Anyone using solr.6.x for multi-terabytes index size: how did you > decide which version to upgrade to? We are still stuck with 4.10 for our 70TB+ (split in 83 shards) index, due to some custom hacks that has not yet been ported. If

Re: solr 6 at scale

2017-05-23 Thread Erick Erickson
I'll quibble a little with Walter and say that 6.4.2 fixes the perf problem in 6.4.0 and 6.4.1. Which doesn't change his recommendation at all, I'd go with 6.5.1. Best, Erick On Tue, May 23, 2017 at 5:49 PM, Walter Underwood wrote: > We are running 6.5.1 in a 16 node cluster, four shards and fou

Re: solr 6 at scale

2017-05-23 Thread Walter Underwood
We are running 6.5.1 in a 16 node cluster, four shards and four replicas. It is performing brilliantly. Our index is 18 million documents, but we have very heavy queries. Students are searching for homework help, so they paste in the entire problem. We truncate queries at 40 terms to limit the

solr 6 at scale

2017-05-23 Thread Nawab Zada Asad Iqbal
Hi all, I am planning to upgrade my solr.4.x installation to a recent stable version. Should I get the latest 6.5.1 bits or will a little older release be better in terms of stability? I am curious if there is way to see solr.6.x adoption in large companies. I have talked to few people and they ar