Re: dataimporthandler large dataset

2011-08-12 Thread Shawn Heisey
On 8/12/2011 3:32 PM, Eric Myers wrote: Recently started looking into solr to solve a problem created before my time. We have a dataset consisting of 390,000,000+ records that had a search written for it using a simple query. The problem is that the dataset needs additional indices to keep oper

Re: dataimporthandler large dataset

2011-08-12 Thread Kyle Lee
We have a 200,000,000 record index with 14 fields, and we can re-index the entire data set in about five hours. One thing to note is that the DataImportHandler uses one thread per entity by default. If you have a multcore box, you can drastically speed indexing by specifying a threadcount of n+1, w