So it is not SolrJ, but Solr is your problem? In your first email there was nothing about heap exceptions, only the runtime about loading.
What do you means by "injecting too many rows", what is "too many"? Some numbers while loading from scratch: - single node 412GB index - 92 fields - 123.6 million docs - 1.937 billion terms - loading from file system - indexing time 9 hrs 5 min - using SolJ ConcurrentUpdateSolrClient --- queueSize=10000, threads=12 --- waitFlush=true, waitSearcher=true, softcommit=false And, Solr must be configured to "swallow" all this :-) You say "8GB per node" so it is SolrCloud? Anyhting else than heap exception? How many commits? Regards Bernd Am 15.02.2018 um 10:31 schrieb LOPEZ-CORTES Mariano-ext: > Injecting too many rows into Solr throws Java heap exception (Higher memory? > We have 8GB per node). > > Have DIH support for paging queries? > > Thanks! > > -----Message d'origine----- > De : Bernd Fehling [mailto:bernd.fehl...@uni-bielefeld.de] > Envoyé : jeudi 15 février 2018 10:13 > À : solr-user@lucene.apache.org > Objet : Re: Reading data from Oracle > > And where is the bottleneck? > > Is it reading from Oracle or injecting to Solr? > > Regards > Bernd > > > Am 15.02.2018 um 08:34 schrieb LOPEZ-CORTES Mariano-ext: >> Hello >> >> We have to delete our Solr collection and feed it periodically from an >> Oracle database (up to 40M rows). >> >> We've done the following test: From a java program, we read chunks of data >> from Oracle and inject to Solr (via Solrj). >> >> The problem : It is really really slow (1'5 nights). >> >> Is there one faster method to do that ? >> >> Thanks in advance. >>
