RE: DIH parallel processing

2015-10-15 Thread Davis, Daniel (NIH/NLM) [C]
This is also what I have done, but I agree with the notion of using something external to load the data. -Original Message- From: Dyer, James [mailto:james.d...@ingramcontent.com] Sent: Thursday, October 15, 2015 9:24 AM To: solr-user@lucene.apache.org Subject: RE: DIH parallel

RE: DIH parallel processing

2015-10-15 Thread Dyer, James
Nabil, What we do is have multiple dih request handlers configured in solrconfig.xml. Then in the sql query we put something like "where mod(id, ${partition})=0". Then an external script calls a full import on each request handler at the same time and monitors the response. This isn't the mo

Re: DIH parallel processing

2015-10-15 Thread Charlie Hull
On 15/10/2015 09:57, nabil Kouici wrote: Hi All, I'm using DIH to index more than 15M from Sql Server to Solr. This take more than 2 hours. Big amount of this time is consumed by data fetching from database. I'm thinking about a solution to have parallel (thread) loud in the same DIH. Each thr