Re: Tuning solr for large index with rapid writes

Thanks for the good suggestions on read traffic. I have been simulating reads through parsing our elb logs and replaying them from a fleet of test servers acting as frontends using Siege . We are hoping to tune mostly based on exact use case, and so this seems th

Re: Tuning solr for large index with rapid writes

Bram: That works. I try to monitor the number of 0-hit queries when I generate a test set on the theory that those are _usually_ groups of random terms I've selected that aren't a good model. So it's often a sequence like "generate my list, see which ones give 0 results and remove them". Rinse, re

Re: Tuning solr for large index with rapid writes

> If I'm reading this right, you have 420M docs on a single shard? > Yep, you were reading it right. Is Erick mentioned, it's hard to give concrete sizing advice, but we've found 120M to be the magic number. When a shard contains more than 120M documents, performance goes down rapidly & GC pauses

Re: Tuning solr for large index with rapid writes

On 29/04/16 16:33, Erick Erickson wrote: > You have one huge advantage when doing prototyping, you can > mine your current logs for real user queries. It's actually > surprisingly difficult to generate, say, 10,000 "realistic" queries. And > IMO you need something approaching that number to insure

Re: Tuning solr for large index with rapid writes

Good luck! You have one huge advantage when doing prototyping, you can mine your current logs for real user queries. It's actually surprisingly difficult to generate, say, 10,000 "realistic" queries. And IMO you need something approaching that number to insure that you're queries don't hit the cac

Re: Tuning solr for large index with rapid writes

​> If I'm reading this right, you have 420M docs on a single shard? Yep, you were reading it right. Thanks for your guidance. We will do various prototyping following "the sizing exercise". Best, Stephen On Tue, Apr 26, 2016 at 6:17 PM, Erick Erickson wrote: > ​​ > If I'm reading this right, yo

Re: Tuning solr for large index with rapid writes

If I'm reading this right, you have 420M docs on a single shard? If that's true you are pushing the envelope of what I've seen work and be performant. Your OOM errors are the proverbial 'smoking gun' that you're putting too many docs on too few nodes. You say that the document count is "growing qu

Tuning solr for large index with rapid writes

Hello, I'm looking for some guidance on the best steps for tuning a solr cloud cluster which is heavy on writes. We are currently running a solr cloud fleet composed of one core, one shard, and three nodes. The cloud is hosted in AWS, and each solr node is on its own linux r3.2xl instance with 8 c