That's promising!!! That's how I have been desigining my project. It must be all the joins that are causing the problems for him? Dennis Gearon
Signature Warning ---------------- EARTH has a Right To Life, otherwise we all die. Read 'Hot, Flat, and Crowded' Laugh at http://www.yert.com/film.php --- On Wed, 6/2/10, David Stuart <david.stu...@progressivealliance.co.uk> wrote: > From: David Stuart <david.stu...@progressivealliance.co.uk> > Subject: Re: Importing large datasets > To: "solr-user@lucene.apache.org" <solr-user@lucene.apache.org> > Date: Wednesday, June 2, 2010, 12:00 PM > How long does it take to do a grab of > all the data via SQL? I found by denormalizing the data into > a lookup table meant that I was able to index about 300k > rows of similar data size with dih regex spilting on some > fields in about 8mins I know it's not quite the scale bit > with batching... > > David Stuar > > On 2 Jun 2010, at 17:58, Blargy <zman...@hotmail.com> > wrote: > > > > > > > > >> One thing that might help indexing speed - create > a *single* SQL query > >> to grab all the data you need without using DIH's > sub-entities, at > >> least the non-cached ones. > >> > > > > Not sure how much that would help. As I mentioned that > without the item > > description import the full process takes 4 hours > which is bearable. However > > once I started to import the item description which is > located on a separate > > machine/database the import process exploded to over > 24 hours. > > > > --View this message in context: > > http://lucene.472066.n3.nabble.com/Importing-large-datasets-tp863447p865324.html > > Sent from the Solr - User mailing list archive at > Nabble.com. >
