Also, which JDBC driver is this? There are quirks with various drivers, which should be documented on the DataImportHandler page.
On Mon, May 10, 2010 at 9:47 PM, caman <aboxfortheotherst...@gmail.com> wrote: > > This may help: > > batchSize : The batchsize used in jdbc connection > > > > http://wiki.apache.org/solr/DataImportHandler#Configuring_DataSources > > > > > > > > > > From: Geek Gamer [via Lucene] > [mailto:ml-node+809069-2054572211-124...@n3.nabble.com] > Sent: Monday, May 10, 2010 9:42 PM > To: caman > Subject: DIH full-import memory issue > > > > Hi, > > I am facing issues with DIH fullimport, > > I have a database with 3 million records that will translate into index size > > of 6GB. > > When I am trying to do full import I am getting out of memory error like : > > INFO: Starting Full Import > May 10, 2010 11:44:06 PM org.apache.solr.handler.dataimport.SolrWriter > readIndexerProperties > WARNING: Unable to read: dataimport.properties > May 10, 2010 11:44:06 PM org.apache.solr.update.DirectUpdateHandler2 > deleteAll > INFO: [] REMOVING ALL DOCUMENTS FROM INDEX > May 10, 2010 11:44:06 PM org.apache.solr.core.SolrDeletionPolicy onInit > INFO: SolrDeletionPolicy.onInit: commits:num=1 > commit{dir=/home/search/SOLR/solr/data/index,segFN=segments_1,version=127354 > 9043650,generation=1,filenames=[segments_1] > May 10, 2010 11:44:06 PM org.apache.solr.core.SolrDeletionPolicy > updateCommits > INFO: newest commit = 1273549043650 > May 10, 2010 11:44:06 PM org.apache.solr.handler.dataimport.JdbcDataSource$1 > > call > INFO: Creating a connection for entity offer with URL: > jdbc:mysql://domU-12-31-39-10-59-01.compute-1.internal/jounce1 > May 10, 2010 11:44:07 PM org.apache.solr.handler.dataimport.JdbcDataSource$1 > > call > INFO: Time taken for getConnection(): 301 > > > > Exception in thread "Timer-1" java.lang.OutOfMemoryError: Java heap space > at java.util.HashMap.newValueIterator(HashMap.java:843) > at java.util.HashMap$Values.iterator(HashMap.java:910) > at > org.mortbay.jetty.servlet.HashSessionManager.scavenge(HashSessionManager.jav > a:180) > at > org.mortbay.jetty.servlet.HashSessionManager.access$000(HashSessionManager.j > ava:36) > at > org.mortbay.jetty.servlet.HashSessionManager$1.run(HashSessionManager.java:1 > 44) > at java.util.TimerThread.mainLoop(Timer.java:512) > at java.util.TimerThread.run(Timer.java:462) > May 10, 2010 11:54:54 PM org.apache.solr.handler.dataimport.DataImporter > doFullImport > SEVERE: Full Import failed > org.apache.solr.handler.dataimport.DataImportHandlerException: > java.lang.OutOfMemoryError: Java heap space > at > org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java: > 424) > at > org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:242 > ) > at > org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:180) > at > org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.ja > va:331) > at > org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:389 > ) > at > org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:370) > > Caused by: java.lang.OutOfMemoryError: Java heap space > at com.mysql.jdbc.MysqlIO.nextRowFast(MysqlIO.java:1621) > at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1398) > at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:2816) > at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:467) > at com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:2510) > at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:1746) > at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2135) > at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2536) > at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2465) > at com.mysql.jdbc.StatementImpl.execute(StatementImpl.java:734) > at > org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.<init>(J > dbcDataSource.java:246) > at > org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.jav > a:210) > at > org.apache.solr.handler.dataimport.JdbcDataSource.getData(JdbcDataSource.jav > a:39) > at > org.apache.solr.handler.dataimport.SqlEntityProcessor.initQuery(SqlEntityPro > cessor.java:58) > at > org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProce > ssor.java:71) > at > org.apache.solr.handler.dataimport.EntityProcessorWrapper.nextRow(EntityProc > essorWrapper.java:237) > at > org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java: > 357) > ... 5 more > May 10, 2010 11:54:54 PM org.apache.solr.update.DirectUpdateHandler2 > rollback > INFO: start rollback > May 10, 2010 11:54:54 PM org.apache.solr.update.DirectUpdateHandler2 > rollback > INFO: end_rollback > > > > > I tried allocating 4 Gigs of memory to the VM but no luck. > Are the records cached before indexing or streamed? > any pointers to documents? > > thanks in anticipation, > umar > > > > _____ > > View message @ > http://lucene.472066.n3.nabble.com/DIH-full-import-memory-issue-tp809069p809 > 069.html > To start a new topic under Solr - User, email > ml-node+472068-464289649-124...@n3.nabble.com > To unsubscribe from Solr - User, click > < (link removed) > GZvcnRoZW90aGVyc3R1ZmZAZ21haWwuY29tfDQ3MjA2OHwtOTM0OTI1NzEx> here. > > > > > -- > View this message in context: > http://lucene.472066.n3.nabble.com/DIH-full-import-memory-issue-tp809069p809076.html > Sent from the Solr - User mailing list archive at Nabble.com. > -- Lance Norskog goks...@gmail.com
