Re: High add/delete rate and index fragmentation

2009-12-04 Thread Otis Gospodnetic
Hello, > You are right that we would need near realtime support. The problem is not > so much about new records becoming available, but guaranteeing that deleted > records will not be returned. For this reason, our plan would be to update > and search a master index, provided that: (1) search whil

Re: High add/delete rate and index fragmentation

2009-12-04 Thread Rodrigo De Castro
On Wed, Dec 2, 2009 at 2:43 PM, Jason Rutherglen wrote: > It sounds like you're asking about near realtime search support, > I'm not sure. So here's few ideas. > > #1 How often do you need to be able to search on the latest > updates (as opposed to updates from lets say, 10 minutes ago)? > You

Re: High add/delete rate and index fragmentation

2009-12-04 Thread Rodrigo De Castro
On Thu, Dec 3, 2009 at 3:59 PM, Lance Norskog wrote: > #2: The standard architecture is with a master that only does indexing > and one or more slaves that only handle queries. The slaves poll the > master for index updates regularly. Java 1.4 has a built-in system for > this. > How do you achi

Re: High add/delete rate and index fragmentation

2009-12-03 Thread Lance Norskog
#1: Yes, compared to relational DBs, Solr/Lucene in general are biased towards slow indexing and fast queries. It automatically merges segments and keeps fragmentation down. The rate of merging can be controlled. #2: The standard architecture is with a master that only does indexing and one or mor

Re: High add/delete rate and index fragmentation

2009-12-02 Thread Jason Rutherglen
Rodrigo, It sounds like you're asking about near realtime search support, I'm not sure. So here's few ideas. #1 How often do you need to be able to search on the latest updates (as opposed to updates from lets say, 10 minutes ago)? To topic #2, Solr provides master slave replication. The optimi