You can disable warming, and a new searcher will register (almost)
instantly, no matter the size. However, once you run your first search,
you will be "warming" your searcher, and it will block for a long, long
time, giving the end user a "frozen" page.
Warming is just another word for "running a set of queries before the
searcher is pushed to the front end." Naturally if you disable warming,
your searcher will register right away. I wouldn't recommend it
though. If I disable warming on my documents, my new searchers would
register instantly, but my first search on my web page would be stuck
for 50 seconds or so.
As for the cache size, caching does a cache on entry data, not
documents. That's what warming is for.
On 2/19/2010 12:17 PM, Stefan Neumann wrote:
Hey,
I am quite confused with your configuration. It seems to me, that your
caches are extremly small for 30 million documents (128) and during
warmup you only put up to 20 docs in it. Please correct me if I
misunderstand anything.
In my opinion your warm up duration is not that impressiv, since we
currently disabled warmup, the new searcher is registered only in a few
seconds.
Actually, I would not drop these cache numbers. With a cache of 30k
documents we had a hitraion of 60%, decreasing this size the hitratio
decreased as well. With a hitratio of currently 30% it seems to be
better to disable caching anyway. Of course we would love to use caching
;-).
with best regards,
Stefan
Antonio Lobato wrote:
Drop those cache numbers. Way down. I warm up 30 million documents in about 2
minutes with the following configuration:
<documentCache
class="solr.FastLRUCache"
size="128"
initialSize="10"
cleanupThread="true" />
<queryResultCache
class="solr.FastLRUCache"
size="128"
initialSize="10"
autowarmCount="20"
cleanupThread="true" />
<fieldValueCache
class="solr.FastLRUCache"
size="128"
initialSize="10"
autowarmCount="20"
cleanupThread="true" />
<filterCache
class="solr.FastLRUCache"
size="128"
initialSize="10"
autowarmCount="20"
cleanupThread="true" />
Mind you, I also use Solr 1.4. Also, setup a decent warming query or two, as
so:
<lst> <str name="q">date:[NOW-2DAYS TO NOW]</str> <str name="start">0</str> <str
name="rows">1000000</str> <str name="sort">date desc</str></lst>
Don't warm facets that have a large amount of terms or you will kill your warm
up time.
Hope this helps!
On Feb 17, 2010, at 8:55 AM, Stefan Neumann wrote:
Hi all,
we are facing extremly increasing warmup times the last 15 days, which
we are not able to explain, since the number of documents and their size
is stable. Before the increase we can commit our changes in nearly 20
minutes, now it is about 2 hours.
We were able to identify the warmup of the caches (queryresultCache and
filterCache) as the reason. We tried to decrease the number of warmup
elements from 30000 to 10000 without any impact.
What influences the runtime during the warmup? Is there any possibility
to boost the warmup?
I attach some more information and statistics.
Thanks a lot for your help.
Stefan
Solr: 1.3
Documents: 4.000.000
-Xmx 12G
index size/disc 4.7G
config:
<queryResultWindowSize>100</queryResultWindowSize>
<queryResultMaxDocsCached>200</queryResultMaxDocsCached>
No queries configured for warming.
CACHES:
=======
name: queryResultCache
class: org.apache.solr.search.LRUCache
version: 1.0
description: LRU Cache(maxSize=200000,
initialSize=30000,
autowarmCount=10000,
regenerator=org.apache.solr.search.solrindexsearche...@36eb7331)
stats:
lookups: 15958
hits : 9589
hitratio: 0.60
inserts: 16211
evictions: 0
size: 16169
warmupTime : 1960239
cumulative_lookups: 436250
cumulative_hits: 260678
cumulative_hitratio: 0.59
cumulative_inserts: 174066
cumulative_evictions: 0
name: filterCache
class: org.apache.solr.search.LRUCache
version: 1.0
description: LRU Cache(maxSize=200000,
initialSize=30000,
autowarmCount=30000,
regenerator=org.apache.solr.search.solrindexsearche...@9818f80)
stats:
lookups: 6313622
hits: 6304004
hitratio: 0.99
inserts: 42266
evictions: 0
size: 40827
warmupTime: 1268074
cumulative_lookups: 118887830
cumulative_hits: 118605224
cumulative_hitratio: 0.99
cumulative_inserts: 296134
cumulative_evictions: 0
