Re: Using DIH to import 10 million records

2012-03-04 Thread Mikhail Khludnev
On Mon, Mar 5, 2012 at 5:56 AM, Lance Norskog wrote: > You can run the DIH with multiple threads feeding from the same query. > FWIW, https://issues.apache.org/jira/browse/SOLR-3011 > Depends also on the size of the document: large documents may index > faster if they have their own threads. Th

Re: Using DIH to import 10 million records

2012-03-04 Thread Sonali Sambhus
Thanks for the info, Shawn! On Mon, Mar 5, 2012 at 6:49 AM, Shawn Heisey wrote: > On 3/4/2012 3:31 AM, Sphene Software wrote: > >> Folks, >> >> I am planning to use DIH for an index of size 10 million records. >> >> I would like to know the following; >> - Can DIH scale for this size of an index

Re: Using DIH to import 10 million records

2012-03-04 Thread Lance Norskog
You can run the DIH with multiple threads feeding from the same query. Depends also on the size of the document: large documents may index faster if they have their own threads. This may then interact with the new NRT multi-commit code. On Sun, Mar 4, 2012 at 5:19 PM, Shawn Heisey wrote: > On 3/4

Re: Using DIH to import 10 million records

2012-03-04 Thread Shawn Heisey
On 3/4/2012 3:31 AM, Sphene Software wrote: Folks, I am planning to use DIH for an index of size 10 million records. I would like to know the following; - Can DIH scale for this size of an indexes - If DIH is a bottleneck, what is the specific issue and how it can be addressed My entire index

Using DIH to import 10 million records

2012-03-04 Thread Sphene Software
Folks, I am planning to use DIH for an index of size 10 million records. I would like to know the following; - Can DIH scale for this size of an indexes - If DIH is a bottleneck, what is the specific issue and how it can be addressed I also read about solrnet. Any experience using this and it's