Hi Shawn, I took care of the warm up problem during the test. I setup jmeter project, get query log from our production(>100000 queries), and run the same query log through jmeter to hit the solr instances with the same qps(about 40). I removed warmup queries in both the solr setup, and also set the autowarmup of cache to 0 in the solrconfig. I run the test for 1 hour. these two instances are not serving other query traffic but they both get update traffic. I disabled softcommit in solr5 and set the hardcommit to 2 minutes. The solr4 instance is a slave node replicating from solr4 master instance, and the master also has 2 minutes commit cycle, and the testing solr4 instance replicate the index every 2 minutes.
The solr5 is slower than solr4. After some investigation I realized that it seems the queries containing q=*:* are causing the problem. I splitted the query log into two log files, one with q=*:* and another without(almost all our queries have filter queries). when I run the test, solr5 is faster when running query with query keyword, but is much slower when run "q=*:*" query log. There is no other query traffic to both the two instance.(there is index traffic). When I get the query debug log in my first email, I make sure there is no filter cache (verified through the solr console. after hard commit, the filtercache is cleaned) Hope my email address your concern about how I do the test. What obvious to me is that solr5 is faster in one test(with query keyword) and is slower in the other test(without query keyword). Thanks, Wei On Fri, Nov 6, 2015 at 1:41 PM, Shawn Heisey <apa...@elyograg.org> wrote: > On 11/6/2015 1:01 PM, wei wrote: > > Thanks Jack and Shawn. I checked these Jira tickets, but I am not sure if > > the slowness of MatchAllDocsQuery is also caused by the removal of > > fieldcache. Can someone please explain a little bit? > > I only glanced at your full output in the message at the start of this > thread. I thought I saw facet output in it, but it turns out that the > only mention of facets was the timing information from the debug, so > that very likely rules out the FieldCache change as a culprit. > > I am suspecting that the 4.7 index is warmed better, and may have the > specific filter query (categoryIdsPath:1001)already sitting in the > filterCache. > > Try running that query a few of times on both versions, then restart > Solr on both versions so they both start clean, and run the query *once* > on each system, and see whether there's still a large discrepancy. > > If one of the systems is receiving queries from active clients and the > other is not, then the comparison will be unfair, and biased towards the > one that is getting additional queries. Query activity, even if it > seems unrelated to the query you are testing, has a tendency to reduce > overall qtime values. > > Thanks, > Shawn > >
