Thanks, Chris. Adding autoWarming to the filter cache made another big
improvement.
Between increasing the soft commit to 60s, fixing the q:* query, and
autowarming the filter caches my 95% latencies are down to a very acceptable
range — almost an order of magnitude improvement. :-)
-Allan
On February 18, 2014 at 5:32:51 PM, Chris Hostetter (hossman_luc...@fucit.org)
wrote:
: Slowing the soft commits to every 100 seconds helped. The main culprit
: was a bad query that was coming through every few seconds. Something
: about the empty fq param and the q=* slowed everything else down.
:
: INFO: [event] webapp=/solr path=/select
: params={start=0&q=*&wt=javabin&fq=&fq=startTime:1392836400003&version=2}
: hits=1894 status=0 QTime=6943
1) if you are using Solr 4.1 or earlier, then q=* is an expensive &
useless query that doesn't mean what you think it does...
https://issues.apache.org/jira/browse/SOLR-2996
2) an empty "fq" doesn't cost anything -- if you use debugQuery=true you
should see that it's not even included in "parsed_filter_queries" because
it's totally ignored.
3) if that "startTime" value changes at some fixed and regular
interval, that could explain some anomoloies if it's normally the
same and cached, but changes once a day/hour/minute or whatever and is a
bit slow to cache.
bottom line: a softCommit is going to re-open a searcher, which is going
to wipe your caches. if you don't have any (auto)warming configured, that
means any "fq"s, or "q"s that you run regularly are going to pay the
price of being "slow" the first time they are run against a new searcher
is opened.
If your priority is low response time, you really want to open new
searchers as infrequently as your SLA for visibility allows, and use
(auto)warming for those common queries.
-Hoss
http://www.lucidworks.com/
