Re: Realtime search and facets with very frequent commits

Yeah, thanks for pointing this out. I'm not using any relevancy functions (yet). The data indexed for my app is basically log events. The most relevant events are the newest ones, so sorting by timestamp is enough. BTW, your book is great ;) -Janne 2010/3/31 Smiley, David W. > Janne, >

Re: Realtime search and facets with very frequent commits

Janne, Have you found your query relevancy to deteriorate with this setup? Something to be aware of with distributed searches is that the relevancy of each Solr core response is based on the local index to that core. So if you're distributed Solr setup does not distribute documents ran

Re: Realtime search and facets with very frequent commits

an Realtime Search has been superseded by Lucene NRT > search. > >> > >> Otis > >> > >> Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch > >> Hadoop ecosystem search :: http://search-hadoop.com/ > >> > >> > >> >

Re: Realtime search and facets with very frequent commits

arch :: http://search-hadoop.com/ >> >> >> >> - Original Message >> > From: Janne Majaranta >> > To: solr-user@lucene.apache.org >> > Sent: Thu, February 18, 2010 2:12:37 AM >> > Subject: Re: Realtime search and facets with very fre

Re: Realtime search and facets with very frequent commits

e > > From: Janne Majaranta > > To: solr-user@lucene.apache.org > > Sent: Thu, February 18, 2010 2:12:37 AM > > Subject: Re: Realtime search and facets with very frequent commits > > > > Hi, > > > > Yes, I did play with mergeFactor. > > I didn

Re: Realtime search and facets with very frequent commits

he.org > Sent: Thu, February 18, 2010 2:12:37 AM > Subject: Re: Realtime search and facets with very frequent commits > > Hi, > > Yes, I did play with mergeFactor. > I didn't play with mergePolicy. > > Wouldn't that affect indexing speed and possibly memo

Re: Realtime search and facets with very frequent commits

Hi, Yes, I did play with mergeFactor. I didn't play with mergePolicy. Wouldn't that affect indexing speed and possibly memory usage ? I don't have any problems with indexing speed ( 1000 - 2000 docs / sec via the standard HTTP API ). My problem is that I need very warm caches to get fast facetin

Re: Realtime search and facets with very frequent commits

Hi, Have you tried playing with mergeFactor or even mergePolicy? -- Jan Høydahl - search architect Cominvent AS - www.cominvent.com On 16. feb. 2010, at 08.26, Janne Majaranta wrote: > Hey Dipti, > > Basically query optimizations + setting cache sizes to a very high level. > Other than that,

Re: Realtime search and facets with very frequent commits

Hey Dipti, Basically query optimizations + setting cache sizes to a very high level. Other than that, the config is about the same as the out-of-the-box config that comes with the Solr download. I haven't found a magic switch to get very fast query responses + facet counts with the frequency of c

Re: Realtime search and facets with very frequent commits

Hey Janne Can you please let me know what other optimizations are you talking about here. Because in our application we are committing in about 5 mins but still the response time is very low and at times there are some connection time outs also. Just wanted to confirm if you have done some major

Re: Realtime search and facets with very frequent commits

Ok, Thanks Yonik and Otis. I already had static warming queries with facets turned on and autowarming at zero. There were a lot of other optimizations after that however, so I'll try with zero autowarming and static warming queries again. If that doesn't work, I'll go with 3 instances on the same

Re: Realtime search and facets with very frequent commits

Janne, The answers to your last 2 questions are both yes. I've seen that done a few times and it works. I don't have the answer to the always-hot cache question. Otis Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch Hadoop ecosystem search :: http://search-hadoop.com/ - O

Re: Realtime search and facets with very frequent commits

On Thu, Feb 11, 2010 at 3:21 PM, Janne Majaranta wrote: > Hey Jason, > > Do you use faceting with frequent commits ? > And by turning off the caches you mean setting autowarmcount to zero ? > > I did try to turn off autowarming with a 36M documents instance but getting > facets over those document

Re: Realtime search and facets with very frequent commits

Hey Jason, Do you use faceting with frequent commits ? And by turning off the caches you mean setting autowarmcount to zero ? I did try to turn off autowarming with a 36M documents instance but getting facets over those documents takes over 10 seconds. With a warm cache it takes 200ms ... -Janne

Re: Realtime search and facets with very frequent commits

Janne, I usually just turn the caches to next to nearly off for frequent commits. Jason On Thu, Feb 11, 2010 at 9:35 AM, Janne Majaranta wrote: > Hello, > > I have a log search like application which requires indexed log events to be > searchable within a minute > and uses facets and the statsc