This might be a hack, but the CSV importer is really fast. Run the query in your favorite command line and export to CSV, then load it.
You can even make batches. Maybe use ranges of the ID, then delete by query for that range. wunder Walter Underwood wun...@wunderwood.org http://observer.wunderwood.org/ (my blog) > On Aug 15, 2017, at 8:46 AM, Stefan Matheis <ste...@mathe.is> wrote: > > Birgit, > > any chance to utilise one of the caching strategies that DIH offers? > > Like building a complete map for one of the subentities? That would mean > reading the whole table at the beginning and then only doing lookups by key. > > Or getting data from subentities with joins in your main entity? > > Heavily depends on the amount of data we're talking about - but might be > worth a thought. > > Best > -Stefan > > On Aug 15, 2017 4:33 PM, "Erick Erickson" <erickerick...@gmail.com> wrote: > > I presume you're using Data Import Handler? An alternative when you > get into complex imports is to use a SolrJ client, here's a sample. > That way you can use whatever tools the particular JDBC connector will > allow and can be much faster. > > https://lucidworks.com/2012/02/14/indexing-with-solrj/ > > Best, > Erick > > On Tue, Aug 15, 2017 at 7:09 AM, Mannott, Birgit <b.mann...@klopotek.de> > wrote: >> Hi, >> >> I'm using solr 6.6.0 and I have to do a complex data import from an > oracle db concerning 3.500.000 data rows. >> For each row I have 15 additional entities. That means that more than 52 > Million selects are send to the database. >> For every select that is done I optimized the oracle execution path by > creating indizes. >> The execution plans are ok. >> But the import still lasts 12 hours. >> >> I think, the main remaining problem is that oracle cursor sharing is not > used and that for every select a hard parse is done. >> Solr does not use binding variables. This would be the easiest way to use > oracle cursor sharing. But I didn't find anything about influencing the way > solr builds select statements. >> I could force oracle cursor sharing without binding variables but I have > to do this configuration for the session. I'm not allowed to change the > configuration of the whole database system. >> >> Is there a way to execute a command like "ALTER SESSION SET > cursor_sharing = FORCE;" after getting the connection for processing an > entity? >> >> Thanks, >> Birgit
