Re: SolrCloud extended warmup support

2014-07-27 Thread Erick Erickson
H, well _I_ don't know what to say then This is puzzling. How much of a latency difference are you seeing? It'd be interesting to see what happens if you experiment with only going to a single shard (add &distrib=false to the query). Each cache is local to the shard, so it's vaguely possi

Re: SolrCloud extended warmup support

2014-07-25 Thread Jeff Wartes
It¹s a command like this just prior to jetty startup: find -L -type f -exec cat {} > /dev/null \; On 7/24/14, 2:11 PM, "Toke Eskildsen" wrote: >Jeff Wartes [jwar...@whitepages.com] wrote: >> Well, I¹m not sure what to say. I¹ve been observing a noticeable latency >> decrease over the first fe

RE: SolrCloud extended warmup support

2014-07-24 Thread Toke Eskildsen
Jeff Wartes [jwar...@whitepages.com] wrote: > Well, I’m not sure what to say. I’ve been observing a noticeable latency > decrease over the first few thousand queries. How exactly do you get the index files fully cached? The cp-command will (at least for some systems) happily skip copying if the d

Re: SolrCloud extended warmup support

2014-07-24 Thread Jeff Wartes
Well, I’m not sure what to say. I’ve been observing a noticeable latency decrease over the first few thousand queries. I’m not doing anything too tricky either. Same exact query pattern, only one fq, always on the same field, no faceting. The only potential suspects that occur to me could be that

Re: SolrCloud extended warmup support

2014-07-21 Thread Erick Erickson
I've never seen it necessary to run "thousands of queries" to warm Solr. Usually less than a dozen will work fine. My challenge would be for you to measure performance differences on queries after running, say, 12 well-chosen queries as opposed to hundreds/thousands. I bet that if 1> you search acr

Re: SolrCloud extended warmup support

2014-07-21 Thread Jeff Wartes
On 7/21/14, 4:50 PM, "Shawn Heisey" wrote: >On 7/21/2014 5:37 PM, Jeff Wartes wrote: >> I¹d like to ensure an extended warmup is done on each SolrCloud node >>prior to that node serving traffic. >> I can do certain things prior to starting Solr, such as pump the index >>dir through /dev/null to

Re: SolrCloud extended warmup support

2014-07-21 Thread Shawn Heisey
On 7/21/2014 5:37 PM, Jeff Wartes wrote: > I’d like to ensure an extended warmup is done on each SolrCloud node prior to > that node serving traffic. > I can do certain things prior to starting Solr, such as pump the index dir > through /dev/null to pre-warm the filesystem cache, and post-start I