Re: DIH keeps failing during full-import

2011-02-07 Thread Erick Erickson
You're probably better off in this instance creating your own process based on SolrJ and your jdbc-driver-of-choice. DIH doesn't provide much in the way of fine-grained control over all aspects of the process, and at +30 hours I suspect you want some better control. FWIW, SolrJ is not very hard at

Re: DIH keeps failing during full-import

2011-02-07 Thread Lance Norskog
It is not reasonable to expect a database session to work over 30 hours, let alone an app/database operation. If you can mark a database record as successfully indexed, the incremental feature can be used to only index non-marked records. SOLR-1499 offers a way to check Solr with a sorted query o

Re: DIH keeps failing during full-import

2011-02-07 Thread Mark
Typo in subject On 2/7/11 7:59 AM, Mark wrote: I'm receiving the following exception when trying to perform a full-import (~30 hours). Any idea on ways I could fix this? Is there an easy way to use DIH to break apart a full-import into multiple pieces? IE 3 mini-imports instead of 1 large imp