Michael Sokolov created LUCENE-9992:
---------------------------------------
Summary: During merging, write empty (headers-only) numeric
vectors when vector-valued FieldInfos is present
Key: LUCENE-9992
URL: https://issues.apache.org/jira/browse/LUCENE-9992
Project: Lucene - Core
Issue Type: Bug
Reporter: Michael Sokolov
When we merge segments having all documents with vectors in some field deleted,
we write FieldInfo for that field, but no index files, and then when we read
the index we get exceptions as shown by this ci test failure:
Build:
[https://ci-builds.apache.org/job/Lucene/job/Lucene-NightlyTests-main/294/]
1 tests failed.
FAILED:
org.apache.lucene.codecs.lucene90.TestLucene90HnswVectorFormat.testDeleteAllVectorDocs
Error Message:
org.apache.lucene.index.CorruptIndexException: Problem reading index from
RawDirectoryWrapper(ByteBuffersDirectory@70f29f52
lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@486fbd6)
(resource=RawDirectoryWrapper(ByteBuffersDirectory@70f29f52
lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@486fbd6))
Stack Trace:
org.apache.lucene.index.CorruptIndexException: Problem reading index from
RawDirectoryWrapper(ByteBuffersDirectory@70f29f52
lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@486fbd6)
(resource=RawDirectoryWrapper(ByteBuffersDirectory@70f29f52
lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@486fbd6))
at
__randomizedtesting.SeedInfo.seed([DA9C80A622E56DF7:82EB7A7E3E790C0C]:0)
at
org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:160)
at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:89)
at
org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:179)
at
org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:221)
at
org.apache.lucene.index.IndexWriter.lambda$getReader$0(IndexWriter.java:534)
at
org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:137)
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:596)
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:452)
at
org.apache.lucene.index.BaseVectorFormatTestCase.testDeleteAllVectorDocs(BaseVectorFormatTestCase.java:564)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1754)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:942)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:978)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:992)
at
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:44)
at
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
at
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
at
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
at
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:370)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:819)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:470)
at
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:951)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:836)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:887)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:898)
at
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
at
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
at
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
at
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
at
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:370)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:826)
at java.base/java.lang.Thread.run(Thread.java:834)
Suppressed: java.lang.RuntimeException: CheckIndex failed
at org.apache.lucene.util.TestUtil.checkIndex(TestUtil.java:331)
at org.apache.lucene.util.TestUtil.checkIndex(TestUtil.java:306)
at
org.apache.lucene.store.BaseDirectoryWrapper.close(BaseDirectoryWrapper.java:42)
at
org.apache.lucene.index.BaseVectorFormatTestCase.testDeleteAllVectorDocs(BaseVectorFormatTestCase.java:549)
... 39 more
Caused by: java.io.FileNotFoundException: No sub-file with id
_Lucene90HnswVectorFormat_0.vem found in compound file "_2.cfs"
(fileName=_2_Lucene90HnswVectorFormat_0.vem files: [.fnm, _Lucene90_0.tip,
_Lucene90_0.tmd, _Lucene90_0.doc, _Lucene90_0.tim, .fdm, .fdx, .fdt])
at
org.apache.lucene.codecs.lucene90.Lucene90CompoundReader.openInput(Lucene90CompoundReader.java:168)
at
org.apache.lucene.store.Directory.openChecksumInput(Directory.java:152)
at
org.apache.lucene.codecs.lucene90.Lucene90HnswVectorReader.readMetadata(Lucene90HnswVectorReader.java:98)
at
org.apache.lucene.codecs.lucene90.Lucene90HnswVectorReader.<init>(Lucene90HnswVectorReader.java:67)
at
org.apache.lucene.codecs.lucene90.Lucene90HnswVectorFormat.fieldsReader(Lucene90HnswVectorFormat.java:91)
at
org.apache.lucene.codecs.perfield.PerFieldVectorFormat$FieldsReader.<init>(PerFieldVectorFormat.java:209)
at
org.apache.lucene.codecs.perfield.PerFieldVectorFormat.fieldsReader(PerFieldVectorFormat.java:77)
at
org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:153)
... 47 more
Build Log:
[...truncated 664 lines...]
ERROR: The following test(s) have failed:
-
org.apache.lucene.codecs.lucene90.TestLucene90HnswVectorFormat.testDeleteAllVectorDocs
(:lucene:core)
Test output:
/home/jenkins/jenkins-slave/workspace/Lucene/Lucene-NightlyTests-main/checkout/lucene/core/build/test-results/test/outputs/OUTPUT-org.apache.lucene.codecs.lucene90.TestLucene90HnswVectorFormat.txt
Reproduce with: gradlew :lucene:core:test --tests
"org.apache.lucene.codecs.lucene90.TestLucene90HnswVectorFormat.testDeleteAllVectorDocs"
-Ptests.jvms=4 -Ptests.haltonfailure=false
-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -Ptests.seed=DA9C80A622E56DF7
-Ptests.multiplier=2 -Ptests.nightly=true -Ptests.badapples=false
-Ptests.file.encoding=UTF-8
-Ptests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene/Lucene-NightlyTests-main/test-data/enwiki.random.lines.txt
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]