As the index gets larger, the underlying housekeeping of the Lucene index sometimes causes pauses in the indexing. The JDBC connection (and/or the underlying socket) to the MySql database can time out during these pauses.
- If it is not set, you should add this to your JCBD url: autoreconnect=true - "Increase the netTimoutForStreamingResults value" from http://lucene.grantingersoll.com/2008/07/16/mysql-solr-and-communications-link-failure/ See also: http://lucene.472066.n3.nabble.com/Recommended-MySQL-JDBC-driver-td817458.html -Glen Newton http://zzzoot.blogspot.com/ On 09/06/2010, Giri <giriprak...@gmail.com> wrote: > Hi Group, > > I have been trying index about 70 million records in the solr index, the > data is coming from the MySQL database, and I am using the DataImportHandler > with batchSize set to -1. When I perform a full-import, it indexes about 27 > million records then throws the following exception: > > Any help will be really appreciated! > > thanks! > > Giri > > ------------------------------------------------------- > > WARNING: Error reading data > com.mysql.jdbc.CommunicationsException: Communications link failure due to > underlying exception: > > ** BEGIN NESTED EXCEPTION ** > > java.io.EOFException > > STACKTRACE: > > java.io.EOFException > at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1934) > at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2433) > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2909) > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:798) > at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1316) > at com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:370) > at com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:360) > at com.mysql.jdbc.ResultSet.next(ResultSet.java:5897) > at > org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:265) > at > org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$500(JdbcDataSource.java:161) > at > org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:196) > at > org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229) > at > org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:77) > at > org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:285) > at > org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:178) > at > org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:136) > at > org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:334) > at > org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:386) > at > org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:377) > > > ** END NESTED EXCEPTION ** > > > > Last packet sent to the server was 5359471 ms ago. > at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2592) > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2909) > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:798) > at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1316) > at com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:370) > at com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:360) > at com.mysql.jdbc.ResultSet.next(ResultSet.java:5897) > at > org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.hasnext(JdbcDataSource.java:265) > at > org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator.access$500(JdbcDataSource.java:161) > at > org.apache.solr.handler.dataimport.JdbcDataSource$ResultSetIterator$1.hasNext(JdbcDataSource.java:196) > at > org.apache.solr.handler.dataimport.EntityProcessorBase.getNext(EntityProcessorBase.java:229) > at > org.apache.solr.handler.dataimport.SqlEntityProcessor.nextRow(SqlEntityProcessor.java:77) > at > org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:285) > at > org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:178) > at > org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:136) > at > org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:334) > at > org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:386) > at > org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:377) > -- -
