Re: High add/delete rate and index fragmentation

2009-12-04 Thread Otis Gospodnetic
Hello, > You are right that we would need near realtime support. The problem is not > so much about new records becoming available, but guaranteeing that deleted > records will not be returned. For this reason, our plan would be to update > and search a master index, provided that: (1) search whil

Re: High add/delete rate and index fragmentation

2009-12-04 Thread Rodrigo De Castro
On Wed, Dec 2, 2009 at 2:43 PM, Jason Rutherglen wrote: > It sounds like you're asking about near realtime search support, > I'm not sure. So here's few ideas. > > #1 How often do you need to be able to search on the latest > updates (as opposed to updates from lets say, 10 minutes ago)? > You

Re: High add/delete rate and index fragmentation

2009-12-04 Thread Rodrigo De Castro
On Thu, Dec 3, 2009 at 3:59 PM, Lance Norskog wrote: > #2: The standard architecture is with a master that only does indexing > and one or more slaves that only handle queries. The slaves poll the > master for index updates regularly. Java 1.4 has a built-in system for > this. > How do you achi

Re: High add/delete rate and index fragmentation

2009-12-03 Thread Lance Norskog
#1: Yes, compared to relational DBs, Solr/Lucene in general are biased towards slow indexing and fast queries. It automatically merges segments and keeps fragmentation down. The rate of merging can be controlled. #2: The standard architecture is with a master that only does indexing and one or mor

Re: High add/delete rate and index fragmentation

2009-12-02 Thread Jason Rutherglen
Rodrigo, It sounds like you're asking about near realtime search support, I'm not sure. So here's few ideas. #1 How often do you need to be able to search on the latest updates (as opposed to updates from lets say, 10 minutes ago)? To topic #2, Solr provides master slave replication. The optimi

High add/delete rate and index fragmentation

2009-12-02 Thread Rodrigo De Castro
We are considering Solr to store events which will be added and deleted from the index in a very fast rate. Solr will be used, in this case, to find the right event we need to process (since they may have several attributes and we may search the best match based on the query attributes). Our unders