One of our Solr indexes is failing to import successfully on occasion,
without any useful errors being reported.
This is a small index (about 15k documents) in Solr 1.4, deployed in Tomcat
7. It is updated every six minutes using the DataImportHandler. These jobs
are managed by JobScheduler.
Lately it has been failing several times a day. It will perform the "clean"
successfully, not show any errors in the logs, and fail to import any (or
most) of the data. Sometimes when this happens that index ends up with zero
documents, other times there are a handful of documents.
Here is an example of the log output of the case where it fails. This log
output is produced by Java Service Wrapper, which we use to maintain our
production Tomcat instances.
INFO | jvm 1 | 2012/06/21 17:33:28 | INFO: [] webapp=/subscription
path=/dataimport params={optimize=true&clean=true&command=full-import}
status=0 QTime=22
INFO | jvm 1 | 2012/06/21 17:33:28 | Jun 21, 2012 5:33:28 PM
org.apache.solr.core.SolrCore execute
INFO | jvm 1 | 2012/06/21 17:33:28 | INFO: [] webapp=/subscription
path=/dataimport params={} status=0 QTime=0
INFO | jvm 1 | 2012/06/21 17:33:28 | Jun 21, 2012 5:33:28 PM
org.apache.solr.handler.dataimport.DataImporter doFullImport
INFO | jvm 1 | 2012/06/21 17:33:28 | INFO: Starting Full Import
INFO | jvm 1 | 2012/06/21 17:33:28 | Jun 21, 2012 5:33:28 PM
org.apache.solr.handler.dataimport.SolrWriter readIndexerProperties
INFO | jvm 1 | 2012/06/21 17:33:28 | INFO: Read dataimport.properties
INFO | jvm 1 | 2012/06/21 17:33:28 | Jun 21, 2012 5:33:28 PM
org.apache.solr.update.DirectUpdateHandler2 deleteAll
INFO | jvm 1 | 2012/06/21 17:33:28 | INFO: [] REMOVING ALL DOCUMENTS
FROM INDEX
INFO | jvm 1 | 2012/06/21 17:33:28 | Jun 21, 2012 5:33:28 PM
org.apache.solr.core.SolrDeletionPolicy onInit
INFO | jvm 1 | 2012/06/21 17:33:28 | INFO: SolrDeletionPolicy.onInit:
commits:num=1
INFO | jvm 1 | 2012/06/21 17:33:28 |
commit{dir=/rpt/src/solr-home/mst/pbl/solr/data/index,segFN=segments_7j,version=1340212196528,generation=271,filenames=[_cj.fdt,
_cj.tii, _cj.fnm, _cj.frq, segments_7j, _cj.nrm, _cj.tis, _cj.prx, _cj.fdx]
INFO | jvm 1 | 2012/06/21 17:33:28 | Jun 21, 2012 5:33:28 PM
org.apache.solr.core.SolrDeletionPolicy updateCommits
INFO | jvm 1 | 2012/06/21 17:33:28 | INFO: newest commit =
1340212196528
INFO | jvm 1 | 2012/06/21 17:33:28 | Jun 21, 2012 5:33:28 PM
org.apache.solr.handler.dataimport.JdbcDataSource$1 call
INFO | jvm 1 | 2012/06/21 17:33:28 | INFO: Creating a connection for
entity PBL with URL: jdbc:oracle:thin:@ecnext-ecn01:1545:ecn01
STATUS | wrapper | 2012/06/21 17:33:29 | TERM trapped. Shutting down.
INFO | jvm 1 | 2012/06/21 17:33:29 | Jun 21, 2012 5:33:29 PM
org.apache.solr.handler.dataimport.JdbcDataSource$1 call
INFO | jvm 1 | 2012/06/21 17:33:29 | INFO: Time taken for
getConnection(): 1187
INFO | jvm 1 | 2012/06/21 17:33:31 | Jun 21, 2012 5:33:31 PM
org.apache.solr.core.SolrCore close
INFO | jvm 1 | 2012/06/21 17:33:31 | INFO: [] CLOSING SolrCore
org.apache.solr.core.SolrCore@57044c5
INFO | jvm 1 | 2012/06/21 17:33:31 | Jun 21, 2012 5:33:31 PM
org.apache.solr.core.SolrCore closeSearcher
INFO | jvm 1 | 2012/06/21 17:33:31 | INFO: [] Closing main searcher on
request.
INFO | jvm 1 | 2012/06/21 17:33:31 | Jun 21, 2012 5:33:31 PM
org.apache.solr.search.SolrIndexSearcher close
INFO | jvm 1 | 2012/06/21 17:33:31 | INFO: Closing Searcher@192425amain
INFO | jvm 1 | 2012/06/21 17:33:31 |
fieldValueCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,warmupTime=0,cumulative_lookups=0,cumulative_hits=0,cumulative_hitratio=0.00,cumulative_inserts=0,cumulative_evictions=0}
INFO | jvm 1 | 2012/06/21 17:33:31 | Jun 21, 2012 5:33:31 PM
org.apache.solr.update.DirectUpdateHandler2 close
INFO | jvm 1 | 2012/06/21 17:33:31 | INFO: closing
DirectUpdateHandler2{commits=0,autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=41,adds=42,deletesById=0,deletesByQuery=1,errors=0,cumulative_adds=42,cumulative_deletesById=0,cumulative_deletesByQuery=1,cumulative_errors=0}
INFO | jvm 1 | 2012/06/21 17:33:31 | Jun 21, 2012 5:33:31 PM
org.apache.solr.core.SolrDeletionPolicy onCommit
INFO | jvm 1 | 2012/06/21 17:33:31 | INFO:
SolrDeletionPolicy.onCommit: commits:num=2
INFO | jvm 1 | 2012/06/21 17:33:31 |
commit{dir=/rpt/src/solr-home/mst/pbl/solr/data/index,segFN=segments_7j,version=1340212196528,generation=271,filenames=[_cj.fdt,
_cj.tii, _cj.fnm, _cj.frq, segments_7j, _cj.nrm, _cj.tis, _cj.prx, _cj.fdx]
INFO | jvm 1 | 2012/06/21 17:33:31 |
commit{dir=/rpt/src/solr-home/mst/pbl/solr/data/index,segFN=segments_7k,version=1340212196529,generation=272,filenames=[_ck.nrm,
_ck.prx, _ck.fdt, _ck.fnm, _ck.tis, _ck.fdx, segments_7k, _ck.tii, _ck.frq]
INFO | jvm 1 | 2012/06/21 17:33:31 | Jun 21, 2012 5:33:31 PM
org.apache.solr.core.SolrDeletionPolicy updateCommits
INFO | jvm 1 | 2012/06/21 17:33:31 | INFO: newest commit =
1340212196529
INFO | jvm 1 | 2012/06/21 17:33:31 | Jun 21, 2012 5:33:31 PM
org.apache.solr.update.DirectUpdateHandler2 close
INFO | jvm 1 | 2012/06/21 17:33:31 | INFO: closed
DirectUpdateHandler2{commits=0,autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=43,deletesById=0,deletesByQuery=1,errors=0,cumulative_adds=43,cumulative_deletesById=0,cumulative_deletesByQuery=1,cumulative_errors=0}
INFO | jvm 1 | 2012/06/21 17:33:31 | Jun 21, 2012 5:33:31 PM
org.apache.solr.core.SolrDeletionPolicy onInit
INFO | jvm 1 | 2012/06/21 17:33:31 | INFO: SolrDeletionPolicy.onInit:
commits:num=1
INFO | jvm 1 | 2012/06/21 17:33:31 |
commit{dir=/rpt/src/solr-home/mst/pbl/solr/data/index,segFN=segments_7k,version=1340212196529,generation=272,filenames=[_ck.nrm,
_ck.prx, _ck.fdt, _ck.fnm, _ck.tis, _ck.fdx, segments_7k, _ck.tii, _ck.frq]
INFO | jvm 1 | 2012/06/21 17:33:31 | Jun 21, 2012 5:33:31 PM
org.apache.solr.core.SolrDeletionPolicy updateCommits
INFO | jvm 1 | 2012/06/21 17:33:31 | INFO: newest commit =
1340212196529
STATUS | wrapper | 2012/06/21 17:33:32 | <-- Wrapper Stopped
We are confident (due to careful monitoring of the DB) that there has been
no data issue here.
I appreciate any help that anyone can provide in diagnosing this issue.
Thanks,
Eric
