The problem is I want the newly added documents to be made searchable every 1-2 seconds so I need the commits. I was hoping that the caches could be stored/tied to the IndexSearcher then a MultiSearcher could take advantage of the multiple sub indexes and their respective caches.
I think the best approach now will be to write a top level federator that can merge the large ~static index and the smaller more dynamic index. - will -----Original Message----- From: Charlie Jackson [mailto:[EMAIL PROTECTED] Sent: Thursday, May 10, 2007 10:53 AM To: solr-user@lucene.apache.org Subject: RE: fast update handlers What about issuing separate commits to the index on a regularly scheduled basis? For example, you add documents to the index every 2 seconds, or however often, but these operations don't commit. Instead, you have a cron'd script or something that just issues a commit every 5 or 10 minutes or whatever interval you'd like. I had to do something similar when I was running a re-index of my entire dataset. My program wasn't issuing commits, so I just cron'd a commit for every half hour so it didn't overload the server. Thanks, Charlie -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Yonik Seeley Sent: Thursday, May 10, 2007 9:07 AM To: solr-user@lucene.apache.org Subject: Re: fast update handlers On 5/10/07, Will Johnson <[EMAIL PROTECTED]> wrote: > I guess I was more concerned with doing the frequent commits and how > that would affect the caches. Say I have 2M docs in my main index but I > want to add docs every 2 seconds all while doing queries. if I do > commits every 2 seconds I basically loose any caching advantage and my > faceting performance goes down the tube. If however, I were to add > things to a smaller index and then roll it into the larger one every ~30 > minutes then I only take the hit on computing the larger filters caches > on that interval. Further, if my smaller index were based on a > RAMDirectory instead of a FSDirectory I assume computing the filter sets > for the smaller index should be fast enough even every 2 seconds. There isn't currently any support for incrementally updating filters. -Yonik
