Re: MatchAllDocsQuery is much slower in solr5.3.1 compare to solr4.7

2015-11-06 Thread wei
Thanks Yonik. A JIRA bug is opened: https://issues.apache.org/jira/browse/SOLR-8251 Wei On Fri, Nov 6, 2015 at 7:10 PM, Yonik Seeley wrote: > On Fri, Nov 6, 2015 at 9:56 PM, wei wrote: > > Good point! I tried that, on solr5 the query time is around 100-110ms, > and > > on solr4 it is around 6

Re: MatchAllDocsQuery is much slower in solr5.3.1 compare to solr4.7

2015-11-06 Thread Yonik Seeley
On Fri, Nov 6, 2015 at 9:56 PM, wei wrote: > Good point! I tried that, on solr5 the query time is around 100-110ms, and > on solr4 it is around 60-63ms(very consistent). Solr5 is slower. When it's something easy, there comes a point when it makes sense to stop asking more questions and just try i

Re: MatchAllDocsQuery is much slower in solr5.3.1 compare to solr4.7

2015-11-06 Thread wei
Good point! I tried that, on solr5 the query time is around 100-110ms, and on solr4 it is around 60-63ms(very consistent). Solr5 is slower. Thanks, Wei On Fri, Nov 6, 2015 at 6:46 PM, Yonik Seeley wrote: > On Fri, Nov 6, 2015 at 9:30 PM, wei wrote: > > in solr 5.3.1, there is actually a boost,

Re: MatchAllDocsQuery is much slower in solr5.3.1 compare to solr4.7

2015-11-06 Thread Yonik Seeley
On Fri, Nov 6, 2015 at 9:30 PM, wei wrote: > in solr 5.3.1, there is actually a boost, and the score is product of boost > & queryNorm. Hmmm, well, it's worth putting on the list of stuff to investigate. Boosting was also changed in lucene. What happens if you try this multiple times in a row?

Re: MatchAllDocsQuery is much slower in solr5.3.1 compare to solr4.7

2015-11-06 Thread wei
Hi Shawn, I took care of the warm up problem during the test. I setup jmeter project, get query log from our production(>10 queries), and run the same query log through jmeter to hit the solr instances with the same qps(about 40). I removed warmup queries in both the solr setup, and also set t

Re: MatchAllDocsQuery is much slower in solr5.3.1 compare to solr4.7

2015-11-06 Thread wei
the explain part are different in solr4.7 and solr 5.3.1. In solr 4.7, there is only one line 1.0 = (MATCH) MatchAllDocsQuery, product of: 1.0 = queryNorm 1.0 = (MATCH) MatchAllDocsQuery, product of: 1.0 = queryNorm in solr 5.3.1, there is actually a boost, and

Re: MatchAllDocsQuery is much slower in solr5.3.1 compare to solr4.7

2015-11-06 Thread wei
Hi Jack, I also run the test with queries that have query terms(with filter too). Solr5 is faster compare to solr4 in the test. I got the queries set from our production log, almost all of our queries have filter. So that suggest to me that it is not the filter query that is slow. I copy the fq q

Re: MatchAllDocsQuery is much slower in solr5.3.1 compare to solr4.7

2015-11-06 Thread Yonik Seeley
On Fri, Nov 6, 2015 at 3:12 PM, Jack Krupansky wrote: > Just to be clear, I was suggesting that the filter query (fq) was slow That's a possibility. Filters were actually removed in Lucene, so it's a very different code path now. In 4.10, filters were first class, and SolrIndexSearcher used met

Re: MatchAllDocsQuery is much slower in solr5.3.1 compare to solr4.7

2015-11-06 Thread Shawn Heisey
On 11/6/2015 1:01 PM, wei wrote: > Thanks Jack and Shawn. I checked these Jira tickets, but I am not sure if > the slowness of MatchAllDocsQuery is also caused by the removal of > fieldcache. Can someone please explain a little bit? I only glanced at your full output in the message at the start of

Re: MatchAllDocsQuery is much slower in solr5.3.1 compare to solr4.7

2015-11-06 Thread Jack Krupansky
Just to be clear, I was suggesting that the filter query (fq) was slow, not the MatchAllDocsQuery, which should be just as speedy as before. You can test for yourself whether the MADQ by itself is any slower. You could also test using the fq as the main query (q) - with no fq parameter, and see if

Re: MatchAllDocsQuery is much slower in solr5.3.1 compare to solr4.7

2015-11-06 Thread wei
Thanks Jack and Shawn. I checked these Jira tickets, but I am not sure if the slowness of MatchAllDocsQuery is also caused by the removal of fieldcache. Can someone please explain a little bit? Thanks, Wei On Fri, Nov 6, 2015 at 7:15 AM, Shawn Heisey wrote: > On 11/5/2015 10:25 PM, Jack Krupans

Re: MatchAllDocsQuery is much slower in solr5.3.1 compare to solr4.7

2015-11-06 Thread Shawn Heisey
On 11/5/2015 10:25 PM, Jack Krupansky wrote: > I vaguely recall some discussion concerning removal of the field cache in > Lucene. The FieldCache wasn't exactly *removed* ... it's more like it was renamed, improved, and sort of hidden in a miscellaneous package. Some things still require this fun

Re: MatchAllDocsQuery is much slower in solr5.3.1 compare to solr4.7

2015-11-05 Thread Jack Krupansky
I vaguely recall some discussion concerning removal of the field cache in Lucene. -- Jack Krupansky On Thu, Nov 5, 2015 at 10:38 PM, wei wrote: > We are running our search on solr4.7 and I am evaluating whether to upgrade > to solr5.3.1. I found MatchAllDocsQuery is much slower in solr5.3.1. An