I guess Solr itself is hogging more memory. May be you can try reloading the core before each import.
On Wed, Apr 1, 2009 at 3:19 PM, Marc Sturlese <marc.sturl...@gmail.com> wrote: > > Hey there, > I am doing performance tests with full-import command from > DataImportHandler. I have configured 20 cores with 1 Gig index each (about a > milion docs per index). If I start doing full-imports indexing from a mysql > table SEQUENCIALY with cron jobs as frequently as possible it will work fine > for about 17 or 20 full-import jobs in total. After that it starts taking > much much longer for each full import... (from 30 min at the begining to 4 > hours for the las full-import before crashing) until I get an > OutOfMemoryError java Heap Space. > > Apr 1 10:30:47 indexer-03 solr: 63534480 [Thread-536] ERROR > org.apache.solr.handler.dataimport.DataImporter - Full Import failed > Apr 1 10:34:04 indexer-03 solr: 63581229 [http-8080-Processor92] ERROR > org.apache.solr.servlet.SolrDispatchFilter - java.lang.OutOfMemoryError: > Java heap space > Apr 1 10:34:23 indexer-03 solr: 63562764 [Thread-546] WARN > org.apache.solr.handler.dataimport.DocBuilder - GC overhead limit exceeded > Apr 1 10:36:11 indexer-03 solr: 63739903 [http-8080-Processor99] ERROR > org.apache.solr.servlet.SolrDispatchFilter - java.lang.OutOfMemoryError: > Java heap space > Apr 1 10:36:11 indexer-03 solr: 63876821 [http-8080-Processor95] ERROR > org.apache.solr.servlet.SolrDispatchFilter - java.lang.OutOfMemoryError: > Java heap space > Apr 1 10:36:20 indexer-03 solr: 63787787 [Thread-546] ERROR > org.apache.solr.handler.dataimport.DataImporter - Full Import failed > Apr 1 10:40:02 indexer-03 solr: 64073790 [http-8080-Processor100] ERROR > org.apache.solr.servlet.SolrDispatchFilter - java.lang.OutOfMemoryError: > Java heap space > Apr 1 10:40:45 indexer-03 solr: 63991575 [http-8080-Processor96] ERROR > org.apache.solr.servlet.SolrDispatchFilter - java.lang.OutOfMemoryError: > Java heap space > > Even I do indexations sequencially I have tryed using a the > ConcurrentGarvageCollector: -XX:+UseConcMarkSweepGC but nothing seems to > change (it's quite logic as I don't index concurrently) > I am runing on > Debian 2.6.26-1-amd64 > mysql for database > java version: > java version "1.6.0_12" > Java(TM) SE Runtime Environment (build 1.6.0_12-b04) > Java HotSpot(TM) 64-Bit Server VM (build 11.2-b01, mixed mode) > > Any idea why this could be happening? > > > > > -- > View this message in context: > http://www.nabble.com/performance-tests-with-DataImportHandler-and-full-import-tp22823145p22823145.html > Sent from the Solr - User mailing list archive at Nabble.com. > > -- --Noble Paul
