We have an indexing script which has been running for a couple of weeks
now without problems. It indexes documents and then periodically commit
(which is a tad redundant I suppose) both via the HTTP interface.
All documents are indexed to a master and a slave rsyncs them off using
the standard 1.3.0 replication.
Recently the indexing script got into problems when the commit was
taking longer than the request timeout. I killed the script, did a
commit by hand (using
bin/commit) and then started to index again and it still wouldn't
commit. We then tried to go to the stats page and got the error
org.apache.lucene.index.CorruptIndexException:
doc counts differ for segment _mib: fieldsReader shows 1 but segmentInfo
shows 718 at
org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:960) at
org.apache.solr.core.SolrCore.<init>(SolrCore.java:470) at
This is a stock 1.3.0 running off tomcat 6.0.20 with
java version "1.6.0_13"
Java(TM) SE Runtime Environment (build 1.6.0_13-b03)
Java HotSpot(TM) 64-Bit Server VM (build 11.3-b02, mixed mode)
Linux solr.local 2.6.18-128.1.10.el5 #1 SMP Thu May 7 10:35:59 EDT
2009 x86_64 x86_64 x86_64 GNU/Linux
Plenty of RAM and disk space (usage is 31% - 353G used from 534G)
CheckIndex says
Opening index @ index/
Segments file=segments_c8z numSegments=28 version=FORMAT_HAS_PROX
[Lucene 2.4]
Checking only these segments: _mib:
22 of 28: name=_mib docCount=718
compound=false
hasProx=true
numFiles=9
size (MB)=0.029
has deletions [delFileName=_mib_1.del]
test: open reader.........FAILED
WARNING: fixIndex() would remove reference to this segment; full
exception:
org.apache.lucene.index.CorruptIndexException: doc counts differ for
segment _mib: fieldsReader shows 1 but segmentInfo shows 718
at
org.apache.lucene.index.SegmentReader$CoreReaders.openDocStores(SegmentReader.java:282)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:640)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:591)
at
org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:491)
at org.apache.lucene.index.CheckIndex.main(CheckIndex.java:903)
WARNING: 1 broken segments (containing 718 documents) detected
WARNING: would write new segments file, and 718 documents would be lost,
if -fix were specified
Any ideas? We can restore from back ups and back fill but really we'd
love to know what caused this so we can avoid a repetition.
Simon
