On Jun 26, 2006, at 10:38 PM, Chris Hostetter wrote:
: I'm trying to fully understand the LRUCache and the autowarmCount
: parameter. Why does it make sense to auto-warm filters and query
: results? In my case, if a new document is added it may invalidate
: many filters, and it would require knowing the details of the
: documents added/removed to know which caches could be copied.
:
: Can someone shed light on the scenarios where blindly copying over
: any cached filters (or query results) makes sense?
Autowarming of the filterCache and queryResultCache doesn't just
copy the
cached values -- it reexecutes the queries used as the keys for those
caches and generates new DocSet/DocLists using the *new* searcher,
before
that searcher is made available to threads serving queries over HTTP.
Ah, that was the secret sauce I was missing. I'm still making my way
through the codebase understanding how it is put together, and now I
see the regenerators in the SolrIndexSearcher for these built-in caches.
For named User caches, autowarming doesn't work at all unless you've
specified a regenerator -- which can do whatever it wants using the
new
searcher and the information from the old cache.
How do I use LRUCache as a custom user cache to deal with cache
misses and look up data dynamically then? It seems to me that
LRUCache.get() should deal with misses itself and call the
regenerator if the key is not found. But rather SolrIndexSearcher
deals with this. If I define a custom cache as an LRUCache with a
custom regenerator, it looks like I have to add a bit more custom
code around where I use that cache to deal with misses. Does it make
sense that LRUCache would pass through to a regenerator on .get() if
the key is not found?
The reason autowarming is configured using an autowarmCount is so
you can
control just how much effort Solr should put into the autowarming
of the
new cache ... if you've got a limitless supply of RAM, and an index
that
doesn't change very often, you can make your caches so big that no
DocSet/DocList is ever generated dynamically more then once -- but
what
happens when your index does finally change? ... if your autowarmCount
is the same as the size of your index, Solr could spend days
autowarming
every query ever executed against your index, even if it was only
executed
one time 3 weeks ago. the autowarmCount tells Solr to only warm the N
"best" keys in the cache where "best" is defined by the Cache
implimentation (for an LRUCache, the "best" things are the things most
recently used).
"LRU" abbreviation confuses me.... I see "least recently used" when I
see that, but it really means "last recently used" within Solr. :)
Once upon a time Yonik and I hypothisized that it would be cool to
have
autowarmTimelimit and autowarmPercentage (of current size) params
and some
other things like that so you could have other ways of tweaking
just how
much autowarming is done on your behalf ... but they were never built.
No worries there. The caching is quite nice as it is. As need
arises, more bells and whistles can be added, but the current
parameters are sufficient for my needs so far.
Erik
