[
https://issues.apache.org/jira/browse/LUCENE-9309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17081395#comment-17081395
]
Simon Willnauer commented on LUCENE-9309:
-----------------------------------------
the push above was to fix a test failure we have on elastic CI:
{noformat}
15:10:24 [junit4] Suite: org.apache.lucene.index.TestIndexWriterMerging
15:10:24 [junit4] 1>
org.apache.lucene.index.MergePolicy$MergeAbortedException: merge is aborted:
_2u(9.0.0):c199/18:[diagnostics={source=flush, os.arch=amd64,
java.runtime.version=11.0.2+7, os.version=4.15.0-1060-gcp, os=Linux,
java.vendor=Oracle Corporation, timestamp=1586617824513, java.version=11.0.2,
java.vm.version=11.0.2+7,
lucene.version=9.0.0}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}]:delGen=1
_2x(9.0.0):c425:[diagnostics={source=flush, os.arch=amd64,
java.runtime.version=11.0.2+7, os.version=4.15.0-1060-gcp, os=Linux,
java.vendor=Oracle Corporation, timestamp=1586617824590, java.version=11.0.2,
java.vm.version=11.0.2+7,
lucene.version=9.0.0}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}]
15:10:24 [junit4] 1> at
org.apache.lucene.index.IndexWriter.registerMerge(IndexWriter.java:4125)
15:10:24 [junit4] 1> at
org.apache.lucene.index.IndexWriter.updatePendingMerges(IndexWriter.java:2203)
15:10:24 [junit4] 1> at
org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2160)
15:10:24 [junit4] 1> at
org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:5142)
15:10:24 [junit4] 1> at
org.apache.lucene.index.IndexWriter.maybeProcessEvents(IndexWriter.java:5132)
15:10:24 [junit4] 1> at
org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1360)
15:10:24 [junit4] 1> at
org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1647)
15:10:24 [junit4] 1> at
org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1289)
15:10:24 [junit4] 1> at
org.apache.lucene.index.TestIndexWriterMerging$1.run(TestIndexWriterMerging.java:416)
15:10:24 [junit4] 2> NOTE: reproduce with: ant test
-Dtestcase=TestIndexWriterMerging -Dtests.method=testNoWaitClose
-Dtests.seed=1D4119EDE2022625 -Dtests.slow=true -Dtests.badapples=true
-Dtests.locale=sv-AX -Dtests.timezone=Europe/Busingen -Dtests.asserts=true
-Dtests.file.encoding=UTF-8
15:10:24 [junit4] ERROR 0.90s J2 | TestIndexWriterMerging.testNoWaitClose
<<<
15:10:24 [junit4] > Throwable #1:
org.apache.lucene.index.MergePolicy$MergeAbortedException: merge is aborted:
_2u(9.0.0):c199/18:[diagnostics={source=flush, os.arch=amd64,
java.runtime.version=11.0.2+7, os.version=4.15.0-1060-gcp, os=Linux,
java.vendor=Oracle Corporation, timestamp=1586617824513, java.version=11.0.2,
java.vm.version=11.0.2+7,
lucene.version=9.0.0}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}]:delGen=1
_2x(9.0.0):c425:[diagnostics={source=flush, os.arch=amd64,
java.runtime.version=11.0.2+7, os.version=4.15.0-1060-gcp, os=Linux,
java.vendor=Oracle Corporation, timestamp=1586617824590, java.version=11.0.2,
java.vm.version=11.0.2+7,
lucene.version=9.0.0}]:[attributes={Lucene50StoredFieldsFormat.mode=BEST_SPEED}]
15:10:24 [junit4] > at
__randomizedtesting.SeedInfo.seed([1D4119EDE2022625:3AB0B83C3114F6D1]:0)
15:10:24 [junit4] > at
org.apache.lucene.index.IndexWriter.registerMerge(IndexWriter.java:4125)
15:10:24 [junit4] > at
org.apache.lucene.index.IndexWriter.updatePendingMerges(IndexWriter.java:2203)
15:10:24 [junit4] > at
org.apache.lucene.index.IndexWriter.maybeMerge(IndexWriter.java:2160)
15:10:24 [junit4] > at
org.apache.lucene.index.IndexWriter.processEvents(IndexWriter.java:5142)
15:10:24 [junit4] > at
org.apache.lucene.index.IndexWriter.maybeProcessEvents(IndexWriter.java:5132)
15:10:24 [junit4] > at
org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1360)
15:10:24 [junit4] > at
org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1647)
15:10:24 [junit4] > at
org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1289)
15:10:24 [junit4] > at
org.apache.lucene.index.TestIndexWriterMerging$1.run(TestIndexWriterMerging.java:416)
15:10:24 [junit4] 2> NOTE: test params are: codec=Asserting(Lucene84),
sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@285b2588),
locale=sv-AX, timezone=Europe/Busingen
15:10:24 [junit4] 2> NOTE: Linux 4.15.0-1060-gcp amd64/Oracle Corporation
11.0.2 (64-bit)/cpus=32,threads=1,free=387385984,total=536870912
15:10:24 [junit4] 2> NOTE: All tests run in this JVM:
[TestNRTReaderCleanup, TestMutableValues, TestDistributionSPL,
TestBinaryDocument, TestRollback, TestXYMultiPolygonShapeQueries,
TestTwoPhaseCommitTool, TestDocIDMerger, TestFieldType,
TestMultiTermConstantScore, TestWildcard, TestLMDirichletSimilarity,
TestIndexWriterFromReader, TestRegexpRandom, TestLatLonPoint,
TestIndexOrDocValuesQuery, TestBasicModelIne, TestByteSlices,
TestFieldValueQuery, TestFSTDirectAddressing, TestSortedSetSelector,
TestPagedBytes, TestTotalHitCountCollector, TestDocIdSetIterator,
TestIndexOptions, TestTopDocsCollector, TestLongRange, TestDocValuesIndexing,
TestStressIndexing, TestNorms, TestMinimize, TestLongValuesSource,
TestDocumentsWriterDeleteQueue, TestDoubleValuesSource,
TestCompetitiveFreqNormAccumulator, TestIndexWriterOnVMError,
TestBoolean2ScorerSupplier, TestGraphTokenizers,
LimitedFiniteStringsIteratorTest, TestAttributeSource, TestIndexCommit,
TestLatLonPolygonShapeQueries, Test2BBKDPoints, TestFieldReuse,
TestMmapDirectory, TestAutomatonQueryUnicode, TestGeoUtils,
TestFlushByRamOrCountsPolicy, TestLatLonLineShapeQueries,
TestBooleanQueryVisitSubscorers, TestXYPolygon, TestTermVectorsReader,
TestNewestSegment, TestOmitTf, TestSimilarity, TestBagOfPostings,
TestLucene50StoredFieldsFormatMergeInstance, TestMaxClauseLimit,
TestSpanNotQuery, TestNativeFSLockFactory, TestStringMSBRadixSorter,
TestLogMergePolicy, TestSoftDeletesRetentionMergePolicy, TestSwappedIndexFiles,
Test2BNumericDocValues, TestSmallFloat, TestByteArrayDataInput,
TestFixedBitDocIdSet, TestDocIdsWriter, TestBytesRefHash, TestAssertions,
TestDemo, TestExternalCodecs, TestMergeSchedulerExternal, TestSearch,
TestSearchForDuplicates, TestAnalyzerWrapper, TestCachingTokenFilter,
TestCharFilter, TestCharacterUtils, TestDelegatingAnalyzerWrapper,
TestGraphTokenFilter, TestStopFilter, TestToken, TestWordlistLoader,
TestStandardAnalyzer, TestPackedTokenAttributeImpl, TestSimpleAttributeImpl,
TestCodecUtil, TestFastDecompressionMode, TestHighCompressionMode,
TestLucene50LiveDocsFormat, TestLucene50StoredFieldsFormat,
TestLucene50StoredFieldsFormatHighCompression, TestLucene70SegmentInfoFormat,
TestIndexedDISI, TestFeatureSort, TestLatLonMultiPointShapeQueries,
TestXYShapeEncoding, TestCircle, TestPoint, TestPoint2D, TestPolygon,
TestPolygon2D, TestAllFilesHaveChecksumFooter, TestBagOfPositions,
TestBufferedUpdates, TestConcurrentMergeScheduler, TestFieldInfos,
TestFilterCodecReader, TestFilterLeafReader, TestIndexManyDocuments,
TestIndexWriterMaxDocs, TestIndexWriterMerging]
{noformat}
> IW#addIndices(CodecReader) might delete files concurrently to IW#rollback
> -------------------------------------------------------------------------
>
> Key: LUCENE-9309
> URL: https://issues.apache.org/jira/browse/LUCENE-9309
> Project: Lucene - Core
> Issue Type: Bug
> Reporter: Simon Willnauer
> Assignee: Simon Willnauer
> Priority: Major
> Fix For: master (9.0), 8.6
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> During work on LUCENE-9304 [~mikemccand] ran into a failure:
> {noformat}
> org.apache.lucene.index.TestAddIndexes > test suite's output saved to
> /home/mike/src/simon/lucene/core/build/test-results/test/outputs/OUTPUT-org.apache.lucene.index.TestAddIndexes.txt,
> copied below:
> > java.nio.file.NoSuchFileException:
> _gt_Lucene85FieldsIndex-doc_ids_6u.tmp
> > at
> __randomizedtesting.SeedInfo.seed([4760FA81FBD4B2CE:A147156E5F7BD9B0]:0)
> > at
> org.apache.lucene.store.ByteBuffersDirectory.deleteFile(ByteBuffersDirectory.java:148)
> > at
> org.apache.lucene.store.MockDirectoryWrapper.deleteFile(MockDirectoryWrapper.java:607)
> > at
> org.apache.lucene.store.LockValidatingDirectoryWrapper.deleteFile(LockValidatingDirectoryWrapper.java:38)
> > at
> org.apache.lucene.index.IndexFileDeleter.deleteFile(IndexFileDeleter.java:696)
> > at
> org.apache.lucene.index.IndexFileDeleter.deleteFiles(IndexFileDeleter.java:690)
> > at
> org.apache.lucene.index.IndexFileDeleter.refresh(IndexFileDeleter.java:449)
> > at
> org.apache.lucene.index.IndexWriter.rollbackInternalNoCommit(IndexWriter.java:2334)
> > at
> org.apache.lucene.index.IndexWriter.rollbackInternal(IndexWriter.java:2275)
> > at
> org.apache.lucene.index.IndexWriter.rollback(IndexWriter.java:2268)
> > at
> org.apache.lucene.index.TestAddIndexes.testAddIndexesWithRollback(TestAddIndexes.java:974)
> 2> NOTE: reproduce with: ant test -Dtestcase=TestAddIndexes
> -Dtests.method=testAddIndexesWithRollback -Dtests.seed=4760FA81FBD4B2CE
> -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=fr-GP -Dtests.t\
> imezone=Asia/Tbilisi -Dtests.asserts=true -Dtests.file.encoding=UTF-8
> 2> NOTE: test params are: codec=Asserting(Lucene84):
> {c=PostingsFormat(name=LuceneFixedGap),
> id=PostingsFormat(name=LuceneFixedGap),
> f1=PostingsFormat(name=LuceneFixedGap), f2=BlockTreeOrds(blocksize=128)\
> , version=BlockTreeOrds(blocksize=128), content=FST50},
> docValues:{dv=DocValuesFormat(name=Lucene80),
> soft_delete=DocValuesFormat(name=Lucene80),
> doc=DocValuesFormat(name=Lucene80), id=DocValuesFormat(name=\
> Asserting), content=DocValuesFormat(name=Asserting),
> doc2d=DocValuesFormat(name=Lucene80)}, maxPointsInLeafNode=982,
> maxMBSortInHeap=5.837219998050092,
> sim=Asserting(org.apache.lucene.search.similarities.As\
> sertingSimilarity@6ce38471), locale=fr-GP, timezone=Asia/Tbilisi
> {noformat}
> While this unfortunately doesn't reproduce it's likely a bug that exists for
> quite some time but never showed up until LUCENE-9147 which uses a temporary
> output. That's fine but with IW#addIndices(CodecReader...) not registering
> the merge it does in the IW we never wait for the merge to finish while
> rollback and if that merge finishes concurrently it will also remove these
> .tmp files.
> There are many ways to fix this and I can work on a patch, but hey do we
> really need to be able to add indices while we index and do that on an open
> and live IW or can it be a tool on top of it?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
