I know so we are not really using it for regular warm-ups (in any case index is updated on hourly basis). Just tried few times to compare results. The issue is I am not even sure if warming up is useful for such regular updates.
On Fri, Feb 4, 2011 at 5:16 PM, Otis Gospodnetic <otis_gospodne...@yahoo.com > wrote: > Salman, > > I only skimmed your email, but wanted to say that this part sounds a little > suspicious: > > > Our warm up script currently executes all distinct queries in our logs > > having count > 5. It was run yesterday (with all the indexing update > every > > It sounds like this will make warmup take a looooong time, assuming you > have > more than a handful distinct queries in your logs. > > Otis > ---- > Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch > Lucene ecosystem search :: http://search-lucene.com/ > > > > ----- Original Message ---- > > From: Salman Akram <salman.ak...@northbaysolutions.net> > > To: solr-user@lucene.apache.org; t...@statsbiblioteket.dk > > Sent: Tue, January 25, 2011 6:32:48 AM > > Subject: Re: Performance optimization of Proximity/Wildcard searches > > > > By warmed index you only mean warming the SOLR cache or OS cache? As I > said > > our index is updated every hour so I am not sure how much SOLR cache > would > > be helpful but OS cache should still be helpful, right? > > > > I haven't compared the results with a proper script but from manual > testing > > here are some of the observations. > > > > 'Recent' queries which are in cache of course return immediately (only > if > > they are exactly same - even if they took 3-4 mins first time). I will > need > > to test how many recent queries stay in cache but still this would work > only > > for very common queries. User can run different queries and I want at > least > > them to be at 'acceptable' level (5-10 secs) even if not very fast. > > > > Our warm up script currently executes all distinct queries in our logs > > having count > 5. It was run yesterday (with all the indexing update > every > > hour after that) and today when I executed some of the same queries > again > > their time seemed a little less (around 15-20%), I am not sure if this > means > > anything. However, still their time is not acceptable. > > > > What do you think is the best way to compare results? First run all the > warm > > up queries and then execute same randomly and compare? > > > > We are using Windows server, would it make a big difference if we move > to > > Linux? Our load is not high but some queries are really complex. > > > > Also I was hoping to move to SSD in last after trying out all software > > options. Is that an agreed fact that on large indexes (which don't fit > in > > RAM) proximity/wildcard/phrase queries (on common words) would be slow > and > > it can be only improved by cache warm up and better hardware? Otherwise > with > > an index of around 150GB such queries will take more than a min? > > > > If that's the case I know this question is very subjective but if a > single > > query takes 2 min on SAS 10K RPM what would its approx time be on a good > SSD > > (everything else same)? > > > > Thanks! > > > > > > On Tue, Jan 25, 2011 at 3:44 PM, Toke Eskildsen > <t...@statsbiblioteket.dk>wrote: > > > > > On Tue, 2011-01-25 at 10:20 +0100, Salman Akram wrote: > > > > Cache warming is a good option too but the index get updated every > hour > > > so > > > > not sure how much would that help. > > > > > > What is the time difference between queries with a warmed index and a > > > cold one? If the warmed index performs satisfactory, then one answer > is > > > to upgrade your underlying storage. As always for IO-caused > performance > > > problem in Lucene/Solr-land, SSD is the answer. > > > > > > > > > > > > -- > > Regards, > > > > Salman Akram > > > -- Regards, Salman Akram
