swapnilsvaidya opened a new issue, #13457:
URL: https://github.com/apache/lucene/issues/13457
### Description
We are using OpenSearch 1.2.3 to index our data. We are observing the
following CorruptIndexException frequently
---------------------------------------------------------------------------------------------------------------------------
org.apache.lucene.index.MergePolicy$MergeException:
org.apache.lucene.index.CorruptIndexException: docs out of order (46 <= 684 )
(resource=RateLimitedIndexOutput(FSIndexOutput(path="/search/nodes/0/indices/A7Yxq0gHSz2ktlIv695d2Q/0/index/_qwcf_Lucene84_0.doc")))
at
org.opensearch.index.engine.InternalEngine$EngineMergeScheduler$2.doRun(InternalEngine.java:2719)
[opensearch-1.2.3.jar:1.2.3]
at
org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:792)
[opensearch-1.2.3.jar:1.2.3]
at
org.opensearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:50)
[opensearch-1.2.3.jar:1.2.3]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source) [?:?]
at java.lang.Thread.run(Unknown Source) [?:?]
Caused by: org.apache.lucene.index.CorruptIndexException: docs out of order
(46 <= 684 )
(resource=RateLimitedIndexOutput(FSIndexOutput(path="/search/nodes/0/indices/A7Yxq0gHSz2ktlIv695d2Q/0/index/_qwcf_Lucene84_0.doc")))
at
org.apache.lucene.codecs.lucene84.Lucene84PostingsWriter.startDoc(Lucene84PostingsWriter.java:231)
~[lucene-core-8.10.1.jar:8.10.1 2f24e6a49d48a032df1f12e146612f59141727a9 -
mayyasharipova - 2021-10-12 15:13:05]
at
org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm(PushPostingsWriterBase.java:146)
~[lucene-core-8.10.1.jar:8.10.1 2f24e6a49d48a032df1f12e146612f59141727a9 -
mayyasharipova - 2021-10-12 15:13:05]
at
org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.write(BlockTreeTermsWriter.java:907)
~[lucene-core-8.10.1.jar:8.10.1 2f24e6a49d48a032df1f12e146612f59141727a9 -
mayyasharipova - 2021-10-12 15:13:05]
at
org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:318)
~[lucene-core-8.10.1.jar:8.10.1 2f24e6a49d48a032df1f12e146612f59141727a9 -
mayyasharipova - 2021-10-12 15:13:05]
at
org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:105)
~[lucene-core-8.10.1.jar:8.10.1 2f24e6a49d48a032df1f12e146612f59141727a9 -
mayyasharipova - 2021-10-12 15:13:05]
at
org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.merge(PerFieldPostingsFormat.java:197)
~[lucene-core-8.10.1.jar:8.10.1 2f24e6a49d48a032df1f12e146612f59141727a9 -
mayyasharipova - 2021-10-12 15:13:05]
at
org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:244)
~[lucene-core-8.10.1.jar:8.10.1 2f24e6a49d48a032df1f12e146612f59141727a9 -
mayyasharipova - 2021-10-12 15:13:05]
at
org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:139)
~[lucene-core-8.10.1.jar:8.10.1 2f24e6a49d48a032df1f12e146612f59141727a9 -
mayyasharipova - 2021-10-12 15:13:05]
at
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4757)
~[lucene-core-8.10.1.jar:8.10.1 2f24e6a49d48a032df1f12e146612f59141727a9 -
mayyasharipova - 2021-10-12 15:13:05]
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4361)
~[lucene-core-8.10.1.jar:8.10.1 2f24e6a49d48a032df1f12e146612f59141727a9 -
mayyasharipova - 2021-10-12 15:13:05]
at
org.apache.lucene.index.IndexWriter$IndexWriterMergeSource.merge(IndexWriter.java:5920)
~[lucene-core-8.10.1.jar:8.10.1 2f24e6a49d48a032df1f12e146612f59141727a9 -
mayyasharipova - 2021-10-12 15:13:05]
at
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:626)
~[lucene-core-8.10.1.jar:8.10.1 2f24e6a49d48a032df1f12e146612f59141727a9 -
mayyasharipova - 2021-10-12 15:13:05]
at
org.opensearch.index.engine.OpenSearchConcurrentMergeScheduler.doMerge(OpenSearchConcurrentMergeScheduler.java:118)
~[opensearch-1.2.3.jar:1.2.3]
at
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:684)
~[lucene-core-8.10.1.jar:8.10.1 2f24e6a49d48a032df1f12e146612f59141727a9 -
mayyasharipova - 2021-10-12 15:13:05]
---------------------------------------------------------------------------------------------------------------------------
I can see that a similar issue is filed on lucene 2.x version but we are
using the recent version and still facing this issue.
I checked the state of the Corrupted index using the CheckIndex utility but
it is not reporting any issue with the index.
---------------------------------------------------------------------------------------------------------------------------
java -cp lucene-core-8.10.1.jar org.apache.lucene.index.CheckIndex -slow
/search/nodes/0/indices/A7Yxq0gHSz2ktlIv695d2Q/0/index/
---------------------------------------------------------------------------------------------------------------------------
Opening index @ /search/nodes/0/indices/A7Yxq0gHSz2ktlIv695d2Q/0/index/
0.00% total deletions; 12038 documents; 0 deleteions
Segments file=segments_3m numSegments=11 version=8.10.1
id=8jru4jsnlmk6em810fnqva8cw userData={history_uuid=HJhQ41sOTDeVG_Jtb8yT1Q,
local_checkpoint=1473044, max_seq_no=1473044, max_unsafe_auto_id_timestamp=-1,
min_retained_seq_no=1461070, translog_uuid=TDZJxve2RLCYj-DpRFRAbw}
1 of 11: name=_qp5q maxDoc=5361
version=8.10.1
id=8jru4jsnlmk6em810fnqv8ihn
codec=Lucene87
compound=false
numFiles=20
size (MB)=21.958
diagnostics = {os=Linux, java.version=11.0.23, os.arch=amd64,
java.runtime.version=11.0.23+10-LTS, source=merge,
os.version=5.15.146-nn2-server, java.vendor=BellSoft,
java.vm.version=11.0.23+10-LTS, lucene.version=8.10.1, mergeMaxNumSegments=-1,
mergeFactor=10, timestamp=1717442290779}
no deletions
test: open reader.........OK [took 0.079 sec]
test: check integrity.....OK [took 0.014 sec]
test: check live docs.....OK [took 0.000 sec]
test: field infos.........OK [328 fields] [took 0.000 sec]
test: field norms.........OK [80 fields] [took 0.016 sec]
test: terms, freq, prox...OK [124918 terms; 2974090 terms/docs pairs;
3758504 tokens] [took 0.824 sec]
test: stored fields.......OK [10722 total field count; avg 2.0 fields
per doc] [took 0.159 sec]
test: term vectors........OK [0 total term vector count; avg 0.0
term/freq vector fields per doc] [took 0.000 sec]
test: docvalues...........OK [246 docvalues fields; 0 BINARY; 4 NUMERIC;
0 SORTED; 84 SORTED_NUMERIC; 158 SORTED_SET] [took 0.407 sec]
test: points..............OK [76 fields, 615710 points] [took 0.063 sec]
test: check soft deletes.....
2 of 11: name=_qri6 maxDoc=3313
version=8.10.1
id=8jru4jsnlmk6em810fnqv9ceu
codec=Lucene87
compound=false
numFiles=20
size (MB)=14.201
diagnostics = {os=Linux, java.version=11.0.23, os.arch=amd64,
java.runtime.version=11.0.23+10-LTS, source=merge,
os.version=5.15.146-nn2-server, java.vendor=BellSoft,
java.vm.version=11.0.23+10-LTS, lucene.version=8.10.1, mergeMaxNumSegments=-1,
mergeFactor=10, timestamp=1717443246184}
no deletions
test: open reader.........OK [took 0.013 sec]
test: check integrity.....OK [took 0.005 sec]
test: check live docs.....OK [took 0.000 sec]
test: field infos.........OK [328 fields] [took 0.000 sec]
test: field norms.........OK [80 fields] [took 0.001 sec]
test: terms, freq, prox...OK [104449 terms; 1838566 terms/docs pairs;
2323261 tokens] [took 0.363 sec]
test: stored fields.......OK [6626 total field count; avg 2.0 fields per
doc] [took 0.075 sec]
test: term vectors........OK [0 total term vector count; avg 0.0
term/freq vector fields per doc] [took 0.000 sec]
test: docvalues...........OK [246 docvalues fields; 0 BINARY; 4 NUMERIC;
0 SORTED; 84 SORTED_NUMERIC; 158 SORTED_SET] [took 0.137 sec]
test: points..............OK [76 fields, 380766 points] [took 0.025 sec]
test: check soft deletes.....
3 of 11: name=_qsvd maxDoc=1907
version=8.10.1
id=8jru4jsnlmk6em810fnqv9tyn
codec=Lucene87
compound=false
numFiles=20
size (MB)=8.642
diagnostics = {os=Linux, java.version=11.0.23, os.arch=amd64,
java.runtime.version=11.0.23+10-LTS, source=merge,
os.version=5.15.146-nn2-server, java.vendor=BellSoft,
java.vm.version=11.0.23+10-LTS, lucene.version=8.10.1, mergeMaxNumSegments=-1,
mergeFactor=10, timestamp=1717443857480}
no deletions
test: open reader.........OK [took 0.011 sec]
test: check integrity.....OK [took 0.003 sec]
test: check live docs.....OK [took 0.000 sec]
test: field infos.........OK [328 fields] [took 0.000 sec]
test: field norms.........OK [80 fields] [took 0.000 sec]
test: terms, freq, prox...OK [78363 terms; 1058585 terms/docs pairs;
1337263 tokens] [took 0.204 sec]
test: stored fields.......OK [3814 total field count; avg 2.0 fields per
doc] [took 0.045 sec]
test: term vectors........OK [0 total term vector count; avg 0.0
term/freq vector fields per doc] [took 0.000 sec]
test: docvalues...........OK [246 docvalues fields; 0 BINARY; 4 NUMERIC;
0 SORTED; 84 SORTED_NUMERIC; 158 SORTED_SET] [took 0.064 sec]
test: points..............OK [76 fields, 218938 points] [took 0.015 sec]
test: check soft deletes.....
4 of 11: name=_qtw1 maxDoc=1448
version=8.10.1
id=8jru4jsnlmk6em810fnqva88x
codec=Lucene87
compound=false
numFiles=20
size (MB)=6.901
diagnostics = {os=Linux, java.version=11.0.23, os.arch=amd64,
java.runtime.version=11.0.23+10-LTS, source=merge,
os.version=5.15.146-nn2-server, java.vendor=BellSoft,
java.vm.version=11.0.23+10-LTS, lucene.version=8.10.1, mergeMaxNumSegments=-1,
mergeFactor=10, timestamp=1717444263593}
no deletions
test: open reader.........OK [took 0.010 sec]
test: check integrity.....OK [took 0.002 sec]
test: check live docs.....OK [took 0.000 sec]
test: field infos.........OK [328 fields] [took 0.000 sec]
test: field norms.........OK [80 fields] [took 0.000 sec]
test: terms, freq, prox...OK [76024 terms; 804015 terms/docs pairs;
1015762 tokens] [took 0.161 sec]
test: stored fields.......OK [2896 total field count; avg 2.0 fields per
doc] [took 0.033 sec]
test: term vectors........OK [0 total term vector count; avg 0.0
term/freq vector fields per doc] [took 0.000 sec]
test: docvalues...........OK [246 docvalues fields; 0 BINARY; 4 NUMERIC;
0 SORTED; 84 SORTED_NUMERIC; 158 SORTED_SET] [took 0.049 sec]
test: points..............OK [76 fields, 166420 points] [took 0.023 sec]
test: check soft deletes.....
5 of 11: name=_qtw2 maxDoc=1
version=8.10.1
id=8jru4jsnlmk6em810fnqva895
codec=Lucene87
compound=true
numFiles=3
size (MB)=0.139
diagnostics = {java.vendor=BellSoft, os=Linux, java.version=11.0.23,
java.vm.version=11.0.23+10-LTS, lucene.version=8.10.1, os.arch=amd64,
java.runtime.version=11.0.23+10-LTS, source=flush,
os.version=5.15.146-nn2-server, timestamp=1717444263988}
no deletions
test: open reader.........OK [took 0.007 sec]
test: check integrity.....OK [took 0.000 sec]
test: check live docs.....OK [took 0.000 sec]
test: field infos.........OK [327 fields] [took 0.000 sec]
test: field norms.........OK [80 fields] [took 0.000 sec]
test: terms, freq, prox...OK [557 terms; 557 terms/docs pairs; 703
tokens] [took 0.007 sec]
test: stored fields.......OK [2 total field count; avg 2.0 fields per
doc] [took 0.000 sec]
test: term vectors........OK [0 total term vector count; avg 0.0
term/freq vector fields per doc] [took 0.000 sec]
test: docvalues...........OK [245 docvalues fields; 0 BINARY; 3 NUMERIC;
0 SORTED; 84 SORTED_NUMERIC; 158 SORTED_SET] [took 0.002 sec]
test: points..............OK [76 fields, 116 points] [took 0.001 sec]
6 of 11: name=_qtw3 maxDoc=1
version=8.10.1
id=8jru4jsnlmk6em810fnqva89g
codec=Lucene87
compound=true
numFiles=3
size (MB)=0.139
diagnostics = {java.vendor=BellSoft, os=Linux, java.version=11.0.23,
java.vm.version=11.0.23+10-LTS, lucene.version=8.10.1, os.arch=amd64,
java.runtime.version=11.0.23+10-LTS, source=flush,
os.version=5.15.146-nn2-server, timestamp=1717444264168}
no deletions
test: open reader.........OK [took 0.006 sec]
test: check integrity.....OK [took 0.000 sec]
test: check live docs.....OK [took 0.000 sec]
test: field infos.........OK [327 fields] [took 0.000 sec]
test: field norms.........OK [80 fields] [took 0.000 sec]
test: terms, freq, prox...OK [557 terms; 557 terms/docs pairs; 703
tokens] [took 0.005 sec]
test: stored fields.......OK [2 total field count; avg 2.0 fields per
doc] [took 0.000 sec]
test: term vectors........OK [0 total term vector count; avg 0.0
term/freq vector fields per doc] [took 0.000 sec]
test: docvalues...........OK [245 docvalues fields; 0 BINARY; 3 NUMERIC;
0 SORTED; 84 SORTED_NUMERIC; 158 SORTED_SET] [took 0.003 sec]
test: points..............OK [76 fields, 116 points] [took 0.001 sec]
7 of 11: name=_qtw4 maxDoc=1
version=8.10.1
id=8jru4jsnlmk6em810fnqva89p
codec=Lucene87
compound=true
numFiles=3
size (MB)=0.139
diagnostics = {java.vendor=BellSoft, os=Linux, java.version=11.0.23,
java.vm.version=11.0.23+10-LTS, lucene.version=8.10.1, os.arch=amd64,
java.runtime.version=11.0.23+10-LTS, source=flush,
os.version=5.15.146-nn2-server, timestamp=1717444264260}
no deletions
test: open reader.........OK [took 0.007 sec]
test: check integrity.....OK [took 0.000 sec]
test: check live docs.....OK [took 0.000 sec]
test: field infos.........OK [327 fields] [took 0.000 sec]
test: field norms.........OK [80 fields] [took 0.000 sec]
test: terms, freq, prox...OK [552 terms; 552 terms/docs pairs; 701
tokens] [took 0.008 sec]
test: stored fields.......OK [2 total field count; avg 2.0 fields per
doc] [took 0.000 sec]
test: term vectors........OK [0 total term vector count; avg 0.0
term/freq vector fields per doc] [took 0.000 sec]
test: docvalues...........OK [245 docvalues fields; 0 BINARY; 3 NUMERIC;
0 SORTED; 84 SORTED_NUMERIC; 158 SORTED_SET] [took 0.002 sec]
test: points..............OK [76 fields, 114 points] [took 0.001 sec]
8 of 11: name=_qtw5 maxDoc=2
version=8.10.1
id=8jru4jsnlmk6em810fnqva8ab
codec=Lucene87
compound=true
numFiles=3
size (MB)=0.15
diagnostics = {java.vendor=BellSoft, os=Linux, java.version=11.0.23,
java.vm.version=11.0.23+10-LTS, lucene.version=8.10.1, os.arch=amd64,
java.runtime.version=11.0.23+10-LTS, source=flush,
os.version=5.15.146-nn2-server, timestamp=1717444264887}
no deletions
test: open reader.........OK [took 0.006 sec]
test: check integrity.....OK [took 0.000 sec]
test: check live docs.....OK [took 0.000 sec]
test: field infos.........OK [327 fields] [took 0.000 sec]
test: field norms.........OK [80 fields] [took 0.000 sec]
test: terms, freq, prox...OK [659 terms; 1114 terms/docs pairs; 1406
tokens] [took 0.008 sec]
test: stored fields.......OK [4 total field count; avg 2.0 fields per
doc] [took 0.000 sec]
test: term vectors........OK [0 total term vector count; avg 0.0
term/freq vector fields per doc] [took 0.000 sec]
test: docvalues...........OK [245 docvalues fields; 0 BINARY; 3 NUMERIC;
0 SORTED; 84 SORTED_NUMERIC; 158 SORTED_SET] [took 0.002 sec]
test: points..............OK [76 fields, 232 points] [took 0.001 sec]
9 of 11: name=_qtw6 maxDoc=2
version=8.10.1
id=8jru4jsnlmk6em810fnqva8az
codec=Lucene87
compound=true
numFiles=3
size (MB)=0.152
diagnostics = {java.vendor=BellSoft, os=Linux, java.version=11.0.23,
java.vm.version=11.0.23+10-LTS, lucene.version=8.10.1, os.arch=amd64,
java.runtime.version=11.0.23+10-LTS, source=flush,
os.version=5.15.146-nn2-server, timestamp=1717444265208}
no deletions
test: open reader.........OK [took 0.006 sec]
test: check integrity.....OK [took 0.000 sec]
test: check live docs.....OK [took 0.000 sec]
test: field infos.........OK [327 fields] [took 0.000 sec]
test: field norms.........OK [80 fields] [took 0.000 sec]
test: terms, freq, prox...OK [716 terms; 1114 terms/docs pairs; 1402
tokens] [took 0.007 sec]
test: stored fields.......OK [4 total field count; avg 2.0 fields per
doc] [took 0.000 sec]
test: term vectors........OK [0 total term vector count; avg 0.0
term/freq vector fields per doc] [took 0.000 sec]
test: docvalues...........OK [245 docvalues fields; 0 BINARY; 3 NUMERIC;
0 SORTED; 84 SORTED_NUMERIC; 158 SORTED_SET] [took 0.002 sec]
test: points..............OK [76 fields, 228 points] [took 0.001 sec]
10 of 11: name=_qtw7 maxDoc=1
version=8.10.1
id=8jru4jsnlmk6em810fnqva8bm
codec=Lucene87
compound=true
numFiles=3
size (MB)=0.139
diagnostics = {java.vendor=BellSoft, os=Linux, java.version=11.0.23,
java.vm.version=11.0.23+10-LTS, lucene.version=8.10.1, os.arch=amd64,
java.runtime.version=11.0.23+10-LTS, source=flush,
os.version=5.15.146-nn2-server, timestamp=1717444265291}
no deletions
test: open reader.........OK [took 0.006 sec]
test: check integrity.....OK [took 0.000 sec]
test: check live docs.....OK [took 0.000 sec]
test: field infos.........OK [327 fields] [took 0.000 sec]
test: field norms.........OK [80 fields] [took 0.000 sec]
test: terms, freq, prox...OK [555 terms; 555 terms/docs pairs; 701
tokens] [took 0.006 sec]
test: stored fields.......OK [2 total field count; avg 2.0 fields per
doc] [took 0.000 sec]
test: term vectors........OK [0 total term vector count; avg 0.0
term/freq vector fields per doc] [took 0.000 sec]
test: docvalues...........OK [245 docvalues fields; 0 BINARY; 3 NUMERIC;
0 SORTED; 84 SORTED_NUMERIC; 158 SORTED_SET] [took 0.002 sec]
test: points..............OK [76 fields, 114 points] [took 0.001 sec]
11 of 11: name=_qtw8 maxDoc=1
version=8.10.1
id=8jru4jsnlmk6em810fnqva8cd
codec=Lucene87
compound=true
numFiles=3
size (MB)=0.139
diagnostics = {java.vendor=BellSoft, os=Linux, java.version=11.0.23,
java.vm.version=11.0.23+10-LTS, lucene.version=8.10.1, os.arch=amd64,
java.runtime.version=11.0.23+10-LTS, source=flush,
os.version=5.15.146-nn2-server, timestamp=1717444265405}
no deletions
test: open reader.........OK [took 0.006 sec]
test: check integrity.....OK [took 0.000 sec]
test: check live docs.....OK [took 0.000 sec]
test: field infos.........OK [327 fields] [took 0.000 sec]
test: field norms.........OK [80 fields] [took 0.000 sec]
test: terms, freq, prox...OK [554 terms; 554 terms/docs pairs; 703
tokens] [took 0.006 sec]
test: stored fields.......OK [2 total field count; avg 2.0 fields per
doc] [took 0.000 sec]
test: term vectors........OK [0 total term vector count; avg 0.0
term/freq vector fields per doc] [took 0.000 sec]
test: docvalues...........OK [245 docvalues fields; 0 BINARY; 3 NUMERIC;
0 SORTED; 84 SORTED_NUMERIC; 158 SORTED_SET] [took 0.002 sec]
test: points..............OK [76 fields, 116 points] [took 0.001 sec]
No problems were detected with this index.
Took 3.109 sec total.
I also tried to check the specific segment (_qwcf) reported by the exception
but no issue reported for this segment as well.
---------------------------------------------------------------------------------------------------------------------------
java -cp lucene-core-8.10.1.jar org.apache.lucene.index.CheckIndex -segment
_qwcf -slow /search/nodes/0/indices/A7Yxq0gHSz2ktlIv695d2Q/0/index/
---------------------------------------------------------------------------------------------------------------------------
Opening index @ /search/nodes/0/indices/A7Yxq0gHSz2ktlIv695d2Q/0/index/
0.00% total deletions; 12038 documents; 0 deleteions
Segments file=segments_3m numSegments=11 version=8.10.1
id=8jru4jsnlmk6em810fnqva8cw userData={history_uuid=HJhQ41sOTDeVG_Jtb8yT1Q,
local_checkpoint=1473044, max_seq_no=1473044, max_unsafe_auto_id_timestamp=-1,
min_retained_seq_no=1461070, translog_uuid=TDZJxve2RLCYj-DpRFRAbw}
Checking only these segments: _qwcf:
No problems were detected with this index.
Took 0.117 sec total.
### Version and environment details
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
