Re: enable disable filter query caching based on statistics

2016-01-08 Thread Alessandro Benedetti
I read the client was happy, so I am only curious to know more :) Apart the readibility, shouldn't be more efficient to put the filters directly in the main query if you don't cache ? ( checking into the code when not caching is adding a lucene boolean query, with specifically 0 score, maybe this i

Re: enable disable filter query caching based on statistics

2016-01-05 Thread Erick Erickson
Binoy: bq: In such a case won't applying fqs normally be the same as applying them as post filters Certainly not, at least AFAIK... By definition, regular FQs are calculated over the entire corpus (not, NOT just the docs that satisfy the query). Then that entire bitset is stored in the filterCac

Re: enable disable filter query caching based on statistics

2016-01-05 Thread Matteo Grolla
Hi Erik, the test was done on thousands of queries of that kind and milions of docs I went from <1500 qpm to ~ 6000 qpm on modest virtualized hardware (cpu bound and cpu was scarce) After that customer happy, time finished and didn't go further but definitely cost was something I'd try When I

Re: enable disable filter query caching based on statistics

2016-01-05 Thread Binoy Dalal
@Eric I might be wrong here so please correct me if I am. In the particular case that Matteo has given applying the filters as post won't make any difference since the query is going to return all docs anyways. In such a case won't applying fqs normally be the same as applying them as post filters?

Re: enable disable filter query caching based on statistics

2016-01-05 Thread Erick Erickson
&fq={!cache=false}n_rea:xxx&fq={!cache=false}provincia:,fq={!cache=false}type: You have a comma in front of the last fq clause, typo? Well, the whole point of caching filter queries is so that the _second_ time you use it, very little work has to be done. That comes at a cost of course fo

Re: enable disable filter query caching based on statistics

2016-01-05 Thread Matteo Grolla
Thanks Erik and Binoy, This is a case I stumbled upon: with queries like q=*:*&fq={!cache=false}n_rea:xxx&fq={!cache=false}provincia:,fq={!cache=false}type: where n_rea filter is highly selective I was able to make > 3x performance improvement disabling cache I think it's because th

Re: enable disable filter query caching based on statistics

2016-01-05 Thread Erick Erickson
Matteo: Let's see if I understand your problem. Essentially you want Solr to analyze the filter queries and decide through some algorithm which ones to cache. I have a hard time thinking of any general way to do this, certainly there's not hing in Solr that does this automatically As Binoy mention

Re: enable disable filter query caching based on statistics

2016-01-05 Thread Binoy Dalal
What is your exact requirement then? I ask, because these settings can solve the problems you've mentioned without the need to add any additional functionality. On Tue, Jan 5, 2016 at 9:04 PM Matteo Grolla wrote: > Hi Binoy, > I know these settings but the problem I'm trying to solve is whe

Re: enable disable filter query caching based on statistics

2016-01-05 Thread Matteo Grolla
Hi Binoy, I know these settings but the problem I'm trying to solve is when these settings aren't enough. 2016-01-05 16:30 GMT+01:00 Binoy Dalal : > If I understand your problem correctly, then you don't want the most > frequently used fqs removed and you do not want your filter cache to gr

Re: enable disable filter query caching based on statistics

2016-01-05 Thread Binoy Dalal
If I understand your problem correctly, then you don't want the most frequently used fqs removed and you do not want your filter cache to grow to very large sizes. Well there is already a solution for both of these. In the solrconfig.xml file, you can configure the parameter to suit your needs. a)

enable disable filter query caching based on statistics

2016-01-05 Thread Matteo Grolla
Hi, after looking at the presentation of cloudsearch from lucene revolution 2014 https://www.youtube.com/watch?v=RI1x0d-yO8A&list=PLU6n9Voqu_1FM8nmVwiWWDRtsEjlPqhgP&index=49 min 17:08 I recognized I'd love to be able to remove the burden of disabling filter query caching from developers the p