Slowing the soft commits to every 100 seconds helped. The main culprit was a
bad query that was coming through every few seconds. Something about the empty
fq param and the q=* slowed everything else down.
INFO: [event] webapp=/solr path=/select
params={start=0&q=*&wt=javabin&fq=&fq=startTime:1392836400003&version=2}
hits=1894 status=0 QTime=6943
Thanks for all your help.
-Allan
On February 18, 2014 at 12:24:37 PM, Shawn Heisey (s...@elyograg.org) wrote:
On 2/18/2014 11:51 AM, Allan Carroll wrote:
> I was thinking GC too, but it doesn’t feel like it is. Running jstat -gcutil
> only shows a 10-50ms parnew collection every 10 or 15 seconds and almost no
> full CMS collections. Anything other places to look for GC activity I might
> be missing?
>
> I did a little investigation this morning and found that if I run a query
> once a second, every 10th query is slow. Looks suspiciously like the soft
> commits are causing the slow downs. I could make it further in between.
> Anything else I can look at to make those commits less costly?
It does indeed sound like the 10 second soft commit is the problem. The
"opening a new searcher" part of a commit tends to be fairly expensive.
The impact is even greater when combined with flushing data to disk,
which is why soft commits can be faster than hard commits ... but
building a new searcher is not cheap even then.
Do you have autoCommit configured, with openSearcher=false? If not, you
should.
If you are using Solr caches, reducing (or eliminating) the
autowarmCount values on each cache (particularly the filterCache) can
make commits happen quite a lot faster. With a commit potentially
happening every ten seconds, you might want to configure those caches so
they are pretty small. Frequent commits mean that the caches are
frequently invalidated. If commit frequency is high and autowarmCount
values are low, a large cache is just a waste of memory. The cache
config was the main thing I was interested in seeing when I asked for
solrconfig.xml.
You have a lot of GC tuning going on, which is good - untuned GC and
Solr do NOT get along. I'll just show you what I use and let you make
your own decision.
http://wiki.apache.org/solr/ShawnHeisey#GC_Tuning
Thanks,
Shawn
