Re: Warming searchers/Caching

2010-12-09 Thread Chris Hostetter
: Our machines have around 8gb of ram and our index is 25gb. What are some good : values for those cache settings. Looks like we have the defaults in place... : : size="16384" : initialSize="4096" : autowarmCount="1024" my personal opinion: set all three options to the same value (16384) it will

Re: Warming searchers/Caching

2010-12-09 Thread Mark
Our machines have around 8gb of ram and our index is 25gb. What are some good values for those cache settings. Looks like we have the defaults in place... size="16384" initialSize="4096" autowarmCount="1024" You are correct, I am just removing the health-check file and our loadbalancer preve

Re: Warming searchers/Caching

2010-12-08 Thread Chris Hostetter
: What am I doing that Solr already provides? the one thing i haven't seen mentioned anywhere in this thread is what you have the "autoWarmCount" value set to on all of the various solr internal caches (as seen in your solrconfig.xml) if that's set, you don't need to manually feed solr any spe

Re: Warming searchers/Caching

2010-12-08 Thread Erick Erickson
or >>>> ordinary >>>> commits, if you weren't using replication) happen quicker than warming >>>> takes >>>> to complete, you can get overlapping indexes being warmed up, and run >>>> out of >>>> RAM (causing garbage colle

Re: Warming searchers/Caching

2010-12-08 Thread Mark
ant to support 'near real time search' better. From: Mark [static.void@gmail.com] Sent: Tuesday, December 07, 2010 10:24 PM To: solr-user@lucene.apache.org Subject: Re: Warming searchers/Caching Maybe I should explain my problem a little more in detail. The problem

Re: Warming searchers/Caching

2010-12-08 Thread Erick Erickson
at >> once. >> >> Solr is not very good at providing 'real time indexing' for this reason, >> although I believe there are some features in post-1.4 trunk meant to >> support 'near real time search' better. >> __________

Re: Warming searchers/Caching

2010-12-08 Thread Mark
___ From: Mark [static.void@gmail.com] Sent: Tuesday, December 07, 2010 10:24 PM To: solr-user@lucene.apache.org Subject: Re: Warming searchers/Caching Maybe I should explain my problem a little more in detail. The problem we are experiencing is after a delta-import we notice a e

Re: Warming searchers/Caching

2010-12-08 Thread Mark
I am using 1.4.1. What am I doing that Solr already provides? Thanks for you help On 12/8/10 5:10 AM, Erick Erickson wrote: What version of Solr are you using? Because it seems like you're doing a lot of stuff that Solr already does for you automatically So perhaps a more complete stateme

RE: Warming searchers/Caching

2010-12-08 Thread Jonathan Rochkind
there are some features in post-1.4 trunk meant to support 'near real time search' better. From: Mark [static.void@gmail.com] Sent: Tuesday, December 07, 2010 10:24 PM To: solr-user@lucene.apache.org Subject: Re: Warming searchers/Caching

Re: Warming searchers/Caching

2010-12-08 Thread Erick Erickson
What version of Solr are you using? Because it seems like you're doing a lot of stuff that Solr already does for you automatically So perhaps a more complete statement of your setup is in order, since we seem to be talking past each other. Best Erick On Tue, Dec 7, 2010 at 10:24 PM, Mark wr

Re: Warming searchers/Caching

2010-12-07 Thread Mark
Maybe I should explain my problem a little more in detail. The problem we are experiencing is after a delta-import we notice a extremely high load time on the slave machines that just replicated. It goes away after a min or so production traffic once everything is cached. I already have a bef

Re: Warming searchers/Caching

2010-12-07 Thread Markus Jelsma
XInclude works fine but that's not what your looking for i guess. Having the 100 top queries is overkill anyway and it can take too long for a new searcher to warmup. Depending on the type of requests, i usually tend to limit warming to popular filter queries only as they generate a very high h

Re: Warming searchers/Caching

2010-12-07 Thread Erick Erickson
Warning: I haven't used this personally, but Xinclude looks like what you're after, see: http://wiki.apache.org/solr/SolrConfigXml#XInclude Best Erick On Tue, Dec 7, 2010 at 6:33 PM, Mark wrote: > Is there any plugin or easy way to auto-warm/cache a new searcher with a > bunch of searches rea

Warming searchers/Caching

2010-12-07 Thread Mark
Is there any plugin or easy way to auto-warm/cache a new searcher with a bunch of searches read from a file? I know this can be accomplished using the EventListeners (newSearcher, firstSearcher) but I rather not add 100+ queries to my solrconfig.xml. If there is no hook/listener available, is