Hey there, I am doing performance tests with full-import command from DataImportHandler. I have configured 20 cores with 1 Gig index each (about a milion docs per index). If I start doing full-imports indexing from a mysql table SEQUENCIALY with cron jobs as frequently as possible it will work fine for about 17 or 20 full-import jobs in total. After that it starts taking much much longer for each full import... (from 30 min at the begining to 4 hours for the las full-import before crashing) until I get an OutOfMemoryError java Heap Space.
Apr 1 10:30:47 indexer-03 solr: 63534480 [Thread-536] ERROR org.apache.solr.handler.dataimport.DataImporter - Full Import failed Apr 1 10:34:04 indexer-03 solr: 63581229 [http-8080-Processor92] ERROR org.apache.solr.servlet.SolrDispatchFilter - java.lang.OutOfMemoryError: Java heap space Apr 1 10:34:23 indexer-03 solr: 63562764 [Thread-546] WARN org.apache.solr.handler.dataimport.DocBuilder - GC overhead limit exceeded Apr 1 10:36:11 indexer-03 solr: 63739903 [http-8080-Processor99] ERROR org.apache.solr.servlet.SolrDispatchFilter - java.lang.OutOfMemoryError: Java heap space Apr 1 10:36:11 indexer-03 solr: 63876821 [http-8080-Processor95] ERROR org.apache.solr.servlet.SolrDispatchFilter - java.lang.OutOfMemoryError: Java heap space Apr 1 10:36:20 indexer-03 solr: 63787787 [Thread-546] ERROR org.apache.solr.handler.dataimport.DataImporter - Full Import failed Apr 1 10:40:02 indexer-03 solr: 64073790 [http-8080-Processor100] ERROR org.apache.solr.servlet.SolrDispatchFilter - java.lang.OutOfMemoryError: Java heap space Apr 1 10:40:45 indexer-03 solr: 63991575 [http-8080-Processor96] ERROR org.apache.solr.servlet.SolrDispatchFilter - java.lang.OutOfMemoryError: Java heap space Even I do indexations sequencially I have tryed using a the ConcurrentGarvageCollector: -XX:+UseConcMarkSweepGC but nothing seems to change (it's quite logic as I don't index concurrently) I am runing on Debian 2.6.26-1-amd64 mysql for database java version: java version "1.6.0_12" Java(TM) SE Runtime Environment (build 1.6.0_12-b04) Java HotSpot(TM) 64-Bit Server VM (build 11.2-b01, mixed mode) Any idea why this could be happening? -- View this message in context: http://www.nabble.com/performance-tests-with-DataImportHandler-and-full-import-tp22823145p22823145.html Sent from the Solr - User mailing list archive at Nabble.com.
