zacharymorn commented on pull request #128:
URL: https://github.com/apache/lucene/pull/128#issuecomment-851169333
Some test results with corrupted index (`_gx_Lucene90_0.dvd`):
### Full check
```
> Task :lucene:core:CheckIndex.main()
NOTE: testing will be more thorough if you run java with
'-ea:org.apache.lucene...', so assertions are enabled
Opening index @
/Users/xichen/IdeaProjects/benchmarks/indices/corrupted/index/
Checking index with async threadCount: 12
0.00% total deletions; 6647577 documents; 0 deletions
Segments file=segments_2 numSegments=15 version=9.0.0
id=59c6he3dhebad46x7proh30nq userData={userData=multi}
1 of 15: name=_32 maxDoc=1197893
version=9.0.0
id=59c6he3dhebad46x7proh2zhm
codec=Lucene90
compound=false
numFiles=17
size (MB)=2,531.843
diagnostics = {timestamp=1622100146526, lucene.version=9.0.0,
java.vm.version=11.0.9+11, java.version=11.0.9, mergeMaxNumSegments=-1,
os.version=10.15.5, java.vendor=AdoptOpenJDK, source=merge, os.arch=x86_64,
mergeFactor=10, java.runtime.version=11.0.9+11, os=Mac OS X}
no deletions
test: open reader.........OK [took 0.125 sec]
test: check integrity.....OK [took 20.451 sec]
test: check live docs.....OK [took 0.000 sec]
test: field infos.........OK [17 fields] [took 0.000 sec]
test: field norms.........OK [2 fields] [took 0.044 sec]
test: terms, freq, prox...OK [20065511 terms; 450728331 terms/docs
pairs; 1175837878 tokens] [took 109.702 sec]
test: stored fields.......OK [3593679 total field count; avg 3.0 fields
per doc] [took 0.967 sec]
test: term vectors........OK [0 total term vector count; avg 0.0
term/freq vector fields per doc] [took 0.000 sec]
test: docvalues...........OK [10 docvalues fields; 3 BINARY; 0 NUMERIC;
5 SORTED; 0 SORTED_NUMERIC; 2 SORTED_SET] [took 2.575 sec]
test: points..............OK [2 fields, 2395786 points] [took 0.204 sec]
test: vectors.............OK [0 fields, 0 vectors] [took 0.000 sec]
2 of 15: name=_65 maxDoc=1197893
version=9.0.0
id=59c6he3dhebad46x7proh2zqv
codec=Lucene90
compound=false
numFiles=17
size (MB)=1,539.981
diagnostics = {timestamp=1622100810971, lucene.version=9.0.0,
java.vm.version=11.0.9+11, java.version=11.0.9, mergeMaxNumSegments=-1,
os.version=10.15.5, java.vendor=AdoptOpenJDK, source=merge, os.arch=x86_64,
mergeFactor=10, java.runtime.version=11.0.9+11, os=Mac OS X}
no deletions
test: open reader.........OK [took 0.124 sec]
test: check integrity.....OK [took 13.612 sec]
test: check live docs.....OK [took 0.000 sec]
test: field infos.........OK [17 fields] [took 0.000 sec]
test: field norms.........OK [2 fields] [took 0.042 sec]
test: terms, freq, prox...OK [15042354 terms; 274837439 terms/docs
pairs; 686566591 tokens] [took 76.072 sec]
test: stored fields.......OK [3593679 total field count; avg 3.0 fields
per doc] [took 0.982 sec]
test: term vectors........OK [0 total term vector count; avg 0.0
term/freq vector fields per doc] [took 0.000 sec]
test: docvalues...........OK [10 docvalues fields; 3 BINARY; 0 NUMERIC;
5 SORTED; 0 SORTED_NUMERIC; 2 SORTED_SET] [took 2.351 sec]
test: points..............OK [2 fields, 2395786 points] [took 0.194 sec]
test: vectors.............OK [0 fields, 0 vectors] [took 0.000 sec]
...
...
10 of 15: name=_gx maxDoc=119789
version=9.0.0
id=59c6he3dhebad46x7proh30n7
codec=Lucene90
compound=false
numFiles=17
size (MB)=129.046
diagnostics = {timestamp=1622102767300, lucene.version=9.0.0,
java.vm.version=11.0.9+11, java.version=11.0.9, mergeMaxNumSegments=-1,
os.version=10.15.5, java.vendor=AdoptOpenJDK, source=merge, os.arch=x86_64,
mergeFactor=10, java.runtime.version=11.0.9+11, os=Mac OS X}
no deletions
test: open reader.........OK [took 0.125 sec]
test: check integrity.....FAILED
WARNING: exorciseIndex() would remove reference to this segment; full
exception:
org.apache.lucene.index.CorruptIndexException: checksum failed (hardware
problem?) : expected=87e2aa4 actual=7b3afcbd
(resource=BufferedChecksumIndexInput(MMapIndexInput(path="/Users/xichen/IdeaProjects/benchmarks/indices/corrupted/index/_gx_Lucene90_0.dvd")))
at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:440)
at
org.apache.lucene.codecs.CodecUtil.checksumEntireFile(CodecUtil.java:614)
at
org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer.checkIntegrity(Lucene90DocValuesProducer.java:1656)
at
org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsReader.checkIntegrity(PerFieldDocValuesFormat.java:364)
at
org.apache.lucene.index.CodecReader.checkIntegrity(CodecReader.java:252)
at
org.apache.lucene.index.SegmentReader.checkIntegrity(SegmentReader.java:391)
at org.apache.lucene.index.CheckIndex.testSegment(CheckIndex.java:913)
at
org.apache.lucene.index.CheckIndex.lambda$checkIndex$0(CheckIndex.java:742)
at
org.apache.lucene.index.CheckIndex.lambda$callableToSupplier$1(CheckIndex.java:842)
at
java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
11 of 15: name=_gy maxDoc=11979
version=9.0.0
id=59c6he3dhebad46x7proh30na
codec=Lucene90
compound=false
numFiles=17
size (MB)=13.473
diagnostics = {os.version=10.15.5, java.runtime.version=11.0.9+11,
os.arch=x86_64, source=flush, lucene.version=9.0.0, java.vm.version=11.0.9+11,
java.version=11.0.9, timestamp=1622102778710, os=Mac OS X,
java.vendor=AdoptOpenJDK}
no deletions
test: open reader.........OK [took 0.124 sec]
test: check integrity.....OK [took 0.065 sec]
test: check live docs.....OK [took 0.000 sec]
test: field infos.........OK [17 fields] [took 0.000 sec]
test: field norms.........OK [2 fields] [took 0.005 sec]
test: terms, freq, prox...OK [295343 terms; 1930448 terms/docs pairs;
4662478 tokens] [took 1.269 sec]
test: stored fields.......OK [35937 total field count; avg 3.0 fields
per doc] [took 0.077 sec]
test: term vectors........OK [0 total term vector count; avg 0.0
term/freq vector fields per doc] [took 0.000 sec]
test: docvalues...........OK [10 docvalues fields; 3 BINARY; 0 NUMERIC;
5 SORTED; 0 SORTED_NUMERIC; 2 SORTED_SET] [took 0.130 sec]
test: points..............OK [2 fields, 23958 points] [took 0.014 sec]
test: vectors.............OK [0 fields, 0 vectors] [took 0.000 sec]
...
...
15 of 15: name=_h2 maxDoc=11248
version=9.0.0
id=59c6he3dhebad46x7proh30nm
codec=Lucene90
compound=false
numFiles=17
size (MB)=10.617
diagnostics = {os.version=10.15.5, java.runtime.version=11.0.9+11,
os.arch=x86_64, source=flush, lucene.version=9.0.0, java.vm.version=11.0.9+11,
java.version=11.0.9, timestamp=1622102791291, os=Mac OS X,
java.vendor=AdoptOpenJDK}
no deletions
test: open reader.........OK [took 0.014 sec]
test: check integrity.....OK [took 0.155 sec]
test: check live docs.....OK [took 0.000 sec]
test: field infos.........OK [17 fields] [took 0.000 sec]
test: field norms.........OK [2 fields] [took 0.001 sec]
test: terms, freq, prox...OK [253387 terms; 1570705 terms/docs pairs;
3390075 tokens] [took 0.703 sec]
test: stored fields.......OK [33744 total field count; avg 3.0 fields
per doc] [took 0.015 sec]
test: term vectors........OK [0 total term vector count; avg 0.0
term/freq vector fields per doc] [took 0.000 sec]
test: docvalues...........OK [10 docvalues fields; 3 BINARY; 0 NUMERIC;
5 SORTED; 0 SORTED_NUMERIC; 2 SORTED_SET] [took 0.074 sec]
test: points..............OK [2 fields, 22496 points] [took 0.004 sec]
test: vectors.............OK [0 fields, 0 vectors] [took 0.000 sec]
WARNING: 1 broken segments (containing 119789 documents) detected
Took 134.361 sec total.
WARNING: would write new segments file, and 119789 documents would be lost,
if -exorcise were specified
> Task :lucene:core:CheckIndex.main() FAILED
Execution failed for task ':lucene:core:CheckIndex.main()'.
> Process 'command
'/Library/Java/JavaVirtualMachines/adoptopenjdk-11.jdk/Contents/Home/bin/java''
finished with non-zero exit value 1
```
### Check with `-segment _gx` flag
```
> Task :lucene:core:CheckIndex.main() FAILED
NOTE: testing will be more thorough if you run java with
'-ea:org.apache.lucene...', so assertions are enabled
Opening index @
/Users/xichen/IdeaProjects/benchmarks/indices/corrupted/index/
Checking index with async threadCount: 12
0.00% total deletions; 6647577 documents; 0 deletions
Segments file=segments_2 numSegments=15 version=9.0.0
id=59c6he3dhebad46x7proh30nq userData={userData=multi}
Checking only these segments: _gx:
10 of 15: name=_gx maxDoc=119789
version=9.0.0
id=59c6he3dhebad46x7proh30n7
codec=Lucene90
compound=false
numFiles=17
size (MB)=129.046
diagnostics = {timestamp=1622102767300, lucene.version=9.0.0,
java.vm.version=11.0.9+11, java.version=11.0.9, mergeMaxNumSegments=-1,
os.version=10.15.5, java.vendor=AdoptOpenJDK, source=merge, os.arch=x86_64,
mergeFactor=10, java.runtime.version=11.0.9+11, os=Mac OS X}
no deletions
test: open reader.........OK [took 0.086 sec]
test: check integrity.....FAILED
WARNING: exorciseIndex() would remove reference to this segment; full
exception:
org.apache.lucene.index.CorruptIndexException: checksum failed (hardware
problem?) : expected=87e2aa4 actual=7b3afcbd
(resource=BufferedChecksumIndexInput(MMapIndexInput(path="/Users/xichen/IdeaProjects/benchmarks/indices/corrupted/index/_gx_Lucene90_0.dvd")))
at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:440)
at
org.apache.lucene.codecs.CodecUtil.checksumEntireFile(CodecUtil.java:614)
at
org.apache.lucene.codecs.lucene90.Lucene90DocValuesProducer.checkIntegrity(Lucene90DocValuesProducer.java:1656)
at
org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsReader.checkIntegrity(PerFieldDocValuesFormat.java:364)
at
org.apache.lucene.index.CodecReader.checkIntegrity(CodecReader.java:252)
at
org.apache.lucene.index.SegmentReader.checkIntegrity(SegmentReader.java:391)
at org.apache.lucene.index.CheckIndex.testSegment(CheckIndex.java:913)
at
org.apache.lucene.index.CheckIndex.lambda$checkIndex$0(CheckIndex.java:742)
at
org.apache.lucene.index.CheckIndex.lambda$callableToSupplier$1(CheckIndex.java:842)
at
java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
WARNING: 1 broken segments (containing 119789 documents) detected
Took 0.734 sec total.
WARNING: would write new segments file, and 119789 documents would be lost,
if -exorcise were specified
Execution failed for task ':lucene:core:CheckIndex.main()'.
> Process 'command
'/Library/Java/JavaVirtualMachines/adoptopenjdk-11.jdk/Contents/Home/bin/java''
finished with non-zero exit value 1
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
