If you dont want to use your own Solj code, why not try many concurrent indexers that index different data sets. So run seven indexers each getting 500,000 rows at the exact same time perhaps. Its a hack, if it works, but if you have the machinery to do it, why not. or use the deltaquery, but i havent used the DIH that much to know if that will help you.
On Tue, Aug 15, 2017 at 10:57 AM, Mannott, Birgit <b.mann...@klopotek.de> wrote: > Yes, I'm using Data Import Handler and I would prefer a solution for this > way of import because it's already tested and the imported data is ok and > everything is fine. > I just have to speed it up a little... > > But thanks for your info. Next time I'll try indexing with solrj. > > Regards, > Birgit > > > -----Original Message----- > From: Erick Erickson [mailto:erickerick...@gmail.com] > Sent: Tuesday, August 15, 2017 4:33 PM > To: solr-user <solr-user@lucene.apache.org> > Subject: [bulk]: Re: Optimizing Dataimport from Oracle; cursor sharing; > changing oracle session parameters > > I presume you're using Data Import Handler? An alternative when you get > into complex imports is to use a SolrJ client, here's a sample. > That way you can use whatever tools the particular JDBC connector will > allow and can be much faster. > > https://lucidworks.com/2012/02/14/indexing-with-solrj/ > > Best, > Erick > > On Tue, Aug 15, 2017 at 7:09 AM, Mannott, Birgit <b.mann...@klopotek.de> > wrote: > > Hi, > > > > I'm using solr 6.6.0 and I have to do a complex data import from an > oracle db concerning 3.500.000 data rows. > > For each row I have 15 additional entities. That means that more than 52 > Million selects are send to the database. > > For every select that is done I optimized the oracle execution path by > creating indizes. > > The execution plans are ok. > > But the import still lasts 12 hours. > > > > I think, the main remaining problem is that oracle cursor sharing is not > used and that for every select a hard parse is done. > > Solr does not use binding variables. This would be the easiest way to > use oracle cursor sharing. But I didn't find anything about influencing the > way solr builds select statements. > > I could force oracle cursor sharing without binding variables but I have > to do this configuration for the session. I'm not allowed to change the > configuration of the whole database system. > > > > Is there a way to execute a command like "ALTER SESSION SET > cursor_sharing = FORCE;" after getting the connection for processing an > entity? > > > > Thanks, > > Birgit >
