Full import is around 6M documents which when completed totals around
30GB in size.
Im guessing it could be a database connectivity problem because I also
see these types of errors on delta-imports which could be anywhere from
20K to 300K records.
On 2/7/11 8:15 AM, Gora Mohanty wrote:
On Mon, Feb 7, 2011 at 9:29 PM, Mark<static.void....@gmail.com> wrote:
I'm receiving the following exception when trying to perform a full-import
(~30 hours). Any idea on ways I could fix this?
Is there an easy way to use DIH to break apart a full-import into multiple
pieces? IE 3 mini-imports instead of 1 large import?
Thanks.
Feb 7, 2011 5:52:33 AM org.apache.solr.handler.dataimport.JdbcDataSource
closeConnection
SEVERE: Ignoring Error when closing connection
com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException:
Communications link failure during rollback(). Transaction resolution
unknown.
[...]
This looks like a network issue, or some other failure in communicating
with the mysql database. Is that a possibility? Also, how many records
are you importing, what is the data size, what is the quality of the network
connection, etc.?
One way to break up the number of records imported at a time is to
shard your data at at the database level, but the advisability of this
option depends on whether there is a more fundamental issue.
Regards,
Gora
