Hi Group,
I have been trying index about 70 million records in the solr index, the
data is coming from the MySQL database, and I am using the DataImportHandler
with batchSize set to -1. When I perform a full-import, it indexes about 27
million records then throws the following exception:
Any help will be really appreciated!
thanks!
Giri
-------------------------------------------------------
WARNING: Error reading data
com.mysql.jdbc.CommunicationsException: Communications link failure due to
underlying exception:
** BEGIN NESTED EXCEPTION **
java.io.EOFException
STACKTRACE:
java.io.EOFException
at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1934)
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2433)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2909)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:798)
at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1316)
at com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:370)
at com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:360)
at com.mysql.jdbc.ResultSet.next(ResultSet.java:5897)
at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:265)
at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$500(JdbcDataSource.java:161)
at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:196)
at
org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229)
at
org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:77)
at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:285)
at
org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:178)
at
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:136)
at
org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:334)
at
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:386)
at
org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:377)
** END NESTED EXCEPTION **
Last packet sent to the server was 5359471 ms ago.
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2592)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2909)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:798)
at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1316)
at com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:370)
at com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:360)
at com.mysql.jdbc.ResultSet.next(ResultSet.java:5897)
at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:265)
at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$500(JdbcDataSource.java:161)
at
org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:196)
at
org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229)
at
org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:77)
at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:285)
at
org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:178)
at
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:136)
at
org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:334)
at
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:386)
at
org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:377)
