Hey, I am quite confused with your configuration. It seems to me, that your caches are extremly small for 30 million documents (128) and during warmup you only put up to 20 docs in it. Please correct me if I misunderstand anything.
In my opinion your warm up duration is not that impressiv, since we currently disabled warmup, the new searcher is registered only in a few seconds. Actually, I would not drop these cache numbers. With a cache of 30k documents we had a hitraion of 60%, decreasing this size the hitratio decreased as well. With a hitratio of currently 30% it seems to be better to disable caching anyway. Of course we would love to use caching ;-). with best regards, Stefan Antonio Lobato wrote: > Drop those cache numbers. Way down. I warm up 30 million documents in about > 2 minutes with the following configuration: > > <documentCache > class="solr.FastLRUCache" > size="128" > initialSize="10" > cleanupThread="true" /> > > <queryResultCache > class="solr.FastLRUCache" > size="128" > initialSize="10" > autowarmCount="20" > cleanupThread="true" /> > > <fieldValueCache > class="solr.FastLRUCache" > size="128" > initialSize="10" > autowarmCount="20" > cleanupThread="true" /> > > <filterCache > class="solr.FastLRUCache" > size="128" > initialSize="10" > autowarmCount="20" > cleanupThread="true" /> > > Mind you, I also use Solr 1.4. Also, setup a decent warming query or two, as > so: > <lst> <str name="q">date:[NOW-2DAYS TO NOW]</str> <str name="start">0</str> > <str name="rows">1000000</str> <str name="sort">date desc</str></lst> > > Don't warm facets that have a large amount of terms or you will kill your > warm up time. > > Hope this helps! > > On Feb 17, 2010, at 8:55 AM, Stefan Neumann wrote: > >> Hi all, >> >> we are facing extremly increasing warmup times the last 15 days, which >> we are not able to explain, since the number of documents and their size >> is stable. Before the increase we can commit our changes in nearly 20 >> minutes, now it is about 2 hours. >> >> We were able to identify the warmup of the caches (queryresultCache and >> filterCache) as the reason. We tried to decrease the number of warmup >> elements from 30000 to 10000 without any impact. >> >> What influences the runtime during the warmup? Is there any possibility >> to boost the warmup? >> >> I attach some more information and statistics. >> >> Thanks a lot for your help. >> >> Stefan >> >> >> Solr: 1.3 >> Documents: 4.000.000 >> -Xmx 12G >> index size/disc 4.7G >> >> config: >> >> <queryResultWindowSize>100</queryResultWindowSize> >> <queryResultMaxDocsCached>200</queryResultMaxDocsCached> >> >> No queries configured for warming. >> >> CACHES: >> ======= >> >> name: queryResultCache >> class: org.apache.solr.search.LRUCache >> version: 1.0 >> description: LRU Cache(maxSize=200000, >> initialSize=30000, >> autowarmCount=10000, >> regenerator=org.apache.solr.search.solrindexsearche...@36eb7331) >> stats: >> >> lookups: 15958 >> hits : 9589 >> hitratio: 0.60 >> inserts: 16211 >> evictions: 0 >> size: 16169 >> warmupTime : 1960239 >> cumulative_lookups: 436250 >> cumulative_hits: 260678 >> cumulative_hitratio: 0.59 >> cumulative_inserts: 174066 >> cumulative_evictions: 0 >> >> >> name: filterCache >> class: org.apache.solr.search.LRUCache >> version: 1.0 >> description: LRU Cache(maxSize=200000, >> initialSize=30000, >> autowarmCount=30000, >> regenerator=org.apache.solr.search.solrindexsearche...@9818f80) >> stats: >> lookups: 6313622 >> hits: 6304004 >> hitratio: 0.99 >> inserts: 42266 >> evictions: 0 >> size: 40827 >> warmupTime: 1268074 >> cumulative_lookups: 118887830 >> cumulative_hits: 118605224 >> cumulative_hitratio: 0.99 >> cumulative_inserts: 296134 >> cumulative_evictions: 0 >> >> >> > > -- Stefan Neumann Dipl.-Ing. freiheit.com technologies gmbh Straßenbahnring 22 / 20251 Hamburg, Germany fon +49 (0)40 / 890584-0 fax +49 (0)40 / 890584-20 HRB Hamburg 70814 1CB2 BA3C 168F 0C2B 6005 FC5E 3EBA BCE2 1BF0 21D3 Geschäftsführer: Claudia Dietze, Stefan Richter, Jörg Kirchhof
