Re: Vertical Partitioning advice

2009-02-10 Thread Mark Kranz
I ended up pursuing the ParallelWriter http://issues.apache.org/jira/browse/LUCENE-600 , so we can map different fields to different indexes. This appears to keep the indexes in sync, although I still need to do more testing. However, some ugly hackery was needed to get it to extend SolrIndexWrit

Re: Vertical Partitioning advice

2009-02-10 Thread Grant Ingersoll
ParallelReader is definitely out there on the Lucene landscape. See http://www.lucidimagination.com/search/page:2?q=ParallelReader for some background discussion, including Doug's original post on it and some others view of the use case. The key is that the small index has to be rebuilt in

Re: Vertical Partitioning advice

2009-02-09 Thread Mark Kranz
Just an update on my own research: I have discovered the 'ParallelReader' class (subclass of IndexReader) in lucene, which is designed for searching across multiple indexes. This appears to suit our needs - and I do not expect will be too difficult to integrate into Solr. -- View this message i