Michael Sokolov created LUCENE-9992:
---------------------------------------

             Summary: During merging, write empty (headers-only) numeric 
vectors when vector-valued FieldInfos is present
                 Key: LUCENE-9992
                 URL: https://issues.apache.org/jira/browse/LUCENE-9992
             Project: Lucene - Core
          Issue Type: Bug
            Reporter: Michael Sokolov


When we merge segments having all documents with vectors in some field deleted, 
we write FieldInfo for that field, but no index files, and then when we read 
the index we get exceptions as shown by this ci test failure:

Build: 
[https://ci-builds.apache.org/job/Lucene/job/Lucene-NightlyTests-main/294/]

1 tests failed.
FAILED:  
org.apache.lucene.codecs.lucene90.TestLucene90HnswVectorFormat.testDeleteAllVectorDocs

Error Message:
org.apache.lucene.index.CorruptIndexException: Problem reading index from 
RawDirectoryWrapper(ByteBuffersDirectory@70f29f52 
lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@486fbd6) 
(resource=RawDirectoryWrapper(ByteBuffersDirectory@70f29f52 
lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@486fbd6))

Stack Trace:
org.apache.lucene.index.CorruptIndexException: Problem reading index from 
RawDirectoryWrapper(ByteBuffersDirectory@70f29f52 
lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@486fbd6) 
(resource=RawDirectoryWrapper(ByteBuffersDirectory@70f29f52 
lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@486fbd6))
        at 
__randomizedtesting.SeedInfo.seed([DA9C80A622E56DF7:82EB7A7E3E790C0C]:0)
        at 
org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:160)
        at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:89)
        at 
org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:179)
        at 
org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:221)
        at 
org.apache.lucene.index.IndexWriter.lambda$getReader$0(IndexWriter.java:534)
        at 
org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:137)
        at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:596)
        at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:452)
        at 
org.apache.lucene.index.BaseVectorFormatTestCase.testDeleteAllVectorDocs(BaseVectorFormatTestCase.java:564)
        at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1754)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:942)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:978)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:992)
        at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:44)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:370)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:819)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:470)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:951)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:836)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:887)
        at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:898)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
        at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
        at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:370)
        at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:826)
        at java.base/java.lang.Thread.run(Thread.java:834)
        Suppressed: java.lang.RuntimeException: CheckIndex failed
                at org.apache.lucene.util.TestUtil.checkIndex(TestUtil.java:331)
                at org.apache.lucene.util.TestUtil.checkIndex(TestUtil.java:306)
                at 
org.apache.lucene.store.BaseDirectoryWrapper.close(BaseDirectoryWrapper.java:42)
                at 
org.apache.lucene.index.BaseVectorFormatTestCase.testDeleteAllVectorDocs(BaseVectorFormatTestCase.java:549)
                ... 39 more
Caused by: java.io.FileNotFoundException: No sub-file with id 
_Lucene90HnswVectorFormat_0.vem found in compound file "_2.cfs" 
(fileName=_2_Lucene90HnswVectorFormat_0.vem files: [.fnm, _Lucene90_0.tip, 
_Lucene90_0.tmd, _Lucene90_0.doc, _Lucene90_0.tim, .fdm, .fdx, .fdt])
        at 
org.apache.lucene.codecs.lucene90.Lucene90CompoundReader.openInput(Lucene90CompoundReader.java:168)
        at 
org.apache.lucene.store.Directory.openChecksumInput(Directory.java:152)
        at 
org.apache.lucene.codecs.lucene90.Lucene90HnswVectorReader.readMetadata(Lucene90HnswVectorReader.java:98)
        at 
org.apache.lucene.codecs.lucene90.Lucene90HnswVectorReader.<init>(Lucene90HnswVectorReader.java:67)
        at 
org.apache.lucene.codecs.lucene90.Lucene90HnswVectorFormat.fieldsReader(Lucene90HnswVectorFormat.java:91)
        at 
org.apache.lucene.codecs.perfield.PerFieldVectorFormat$FieldsReader.<init>(PerFieldVectorFormat.java:209)
        at 
org.apache.lucene.codecs.perfield.PerFieldVectorFormat.fieldsReader(PerFieldVectorFormat.java:77)
        at 
org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:153)
        ... 47 more




Build Log:
[...truncated 664 lines...]
ERROR: The following test(s) have failed:
  - 
org.apache.lucene.codecs.lucene90.TestLucene90HnswVectorFormat.testDeleteAllVectorDocs
 (:lucene:core)
    Test output: 
/home/jenkins/jenkins-slave/workspace/Lucene/Lucene-NightlyTests-main/checkout/lucene/core/build/test-results/test/outputs/OUTPUT-org.apache.lucene.codecs.lucene90.TestLucene90HnswVectorFormat.txt
    Reproduce with: gradlew :lucene:core:test --tests 
"org.apache.lucene.codecs.lucene90.TestLucene90HnswVectorFormat.testDeleteAllVectorDocs"
 -Ptests.jvms=4 -Ptests.haltonfailure=false 
-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -Ptests.seed=DA9C80A622E56DF7 
-Ptests.multiplier=2 -Ptests.nightly=true -Ptests.badapples=false 
-Ptests.file.encoding=UTF-8 
-Ptests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene/Lucene-NightlyTests-main/test-data/enwiki.random.lines.txt

 

  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to