Oh, what a lovely anti-pattern! Every second, you're throwing away your
filterCache. And then firing up to 4096 autowarm queries at your index
on the filterCache. And this doesn't even consider the other caches! And
this will get worse with time after restarts if my scenario is accurate.
Having extremely fast autocommit times is fundamentally incompatible
with having large caches. More hardware simply won't help unless
you have so much hardware that you're storing a tiny number of docs
on each shard, which is a ridiculous solution. IMO your only choices
are to
1> drastically reduce your autowarm counts (which, btw, means that
you can also drastically reduce their size)
or
2> drastically increase your commit interval and live with 15 minute
warmup times.
I've never seen autowarm counts that high really do much good anyway.
For that matter, the raw size of your caches is also far too large IMO.
If you're really generating that many unique filter queries, I'd argue
you've got something pathological in your setup.
What you are _not_ going to be able to do is have autowarm counts
that high and 1 second commits.
Here's the long form. I don't know how many entries are in your various
caches, but I bet it's a lot. Note that it picks on filterCache, but the
same logic applies to your other caches, i.e. queryResultCache
in particular.
Your new searcher isn't serving any queries until up to 4096 autowarm
queries are executed. Which you're then immediately throwing
away due to your soft commit interval. I rather suspect you'll be seeing
log messages about "too many on deck searchers" or some such due
to this setting in solrconfig.xml:
<maxWarmingSearchers>2</maxWarmingSearchers>
Do not think you can up this number and cure your problem, you'll
hit OOMs I predict if you do.
Now, here's the kicker. Your filterCache is going to accumulate
entries _because_ of autowarming! Imagine that every second you're
adding 10 new fq clauses that are unique.
Now, when the first soft commit happens, your filterCache has 10
entries in it. The autowarm kicks in and puts those same 10 in
your filterCache and the searcher opens for business.
A second later, another 10 fq clauses have come in and been put
in your filterCache. And another searcher is opened. And now there
are 20 entries. All of which are autowarmed and put back in the
filterCache.
A second later there are 30 entries in your filterCache. A second
after that 40. And so on. If this guess is correct, then if you
restart all your servers your interval between when you index a
doc and can search it will start out OK, but then increase.
Eventually you get to the situation you have now, a huge number
of autowarming queries that take minutes to warm before you can
see any of your recently added documents.
Places to look to see what's happening:
admin/plugins-stats/core. you should see "warmupTime" near the bottom.
admin/plugins-stats/cache/filter cache. You should see warmupTime there too.
/document cache etc.
While you're at the filter cache stats page, take a look at your hit ratio,
I suspect it's very small. If so, it's not doing you much good anyway.
BTW, one way of adding completely useless (but unique) fq clauses is
to follow another anti-pattern like:
fq=timestamp:[NOW/DAY to NOW]
http://searchhub.org/2012/02/23/date-math-now-and-filter-queries/
On Thu, Aug 14, 2014 at 7:21 AM, cwhit <cwhi...@solinkcorp.com> wrote:
> Thanks for the explanation. This makes a lot of sense to me... I'm wondering
> if there's a way to get the best of both worlds. Can throwing more hardware
> at the index give real time updates + a large LRU cache? Would we be CPU
> bound at this point?
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Updates-to-index-not-available-immediately-as-index-scales-even-with-autoSoftCommit-at-1-second-tp4152511p4152983.html
> Sent from the Solr - User mailing list archive at Nabble.com.
