guojialiang92 opened a new issue, #14423: URL: https://github.com/apache/lucene/issues/14423
### Description ### Description I found that Test `TestIndexWriterWithThreads#testIOExceptionDuringWriteSegmentWithThreadsOnlyOnce` may fail in rare cases. Exception information is as follows: `java.lang.RuntimeException: MockDirectoryWrapper: cannot close: there are still 108 open files` I looked up some other related issues and finally located the problem. At the same time, I also added a test that can stably reproduce the problem. ### Analysis The root cause is that `IndexWriter#ensureOpen()` throws `AlreadyClosedException` when `IndexWriter#closed` or `IndexWriter#closing` is `true`. In the test code, after throwing `AlreadyClosedException` when executing `IndexWriter#commit`, the `IndexWriter#close` is not called again to ensure that `IndexWriter` closes normally. ``` try { writer.commit(); writer.close(); success = true; } catch ( @SuppressWarnings("unused") AlreadyClosedException ace) { // OK: abort closes the writer assertTrue(writer.isDeleterClosed()); } catch ( @SuppressWarnings("unused") IOException ioe) { writer.rollback(); failure.clearDoFail(); } ``` ### To Reproduce In order to stabilize the reproduce problem, I added a test `TestIndexWriterWithThreads#testIOExceptionWithMergeNotEndLongTime`. The code will be executed in the following order: 1. Set `LiveIndexWriterConfig#readerPooling` to `true` to ensure that `ReaderPool#release` does not release `ReadersAndUpdates` 2. Start an `IndexerThread` and start writing data 3. Wait for a merge thread to start working (using `mergeCountDownLatch`), then simulate write failure via `MockDirectoryWrapper#failOn` 4. Block the execution of the merge thread until `IndexWriter#updateDocument` throws an exception due to write failure (using `updateDocumentFailCountDownLatch`). 5. Merge thread and IW thread will both call `IndexWriter#maybeCloseOnTragicEvent`, and I control to let Merge thread execute `IndexWriter#rollbackInternalNoCommit` first, and the IW thread will skip it. (using `mergeCloseCountDownLatch`) 6. Because the execution of the merge thread is asynchronous, the test will continue until `IndexWriter#commit` is called and a `AlreadyClosedException` is throw. 7. In order to prevent `IndexInput` from being closed before calling `MockDirectoryWrapper#close`, I let the `ConcurrentMergeScheduler#close` sleep for 5s. ### Solution The closure of `IndexInput` needs to wait until the execution of `ReaderPool#close` in `IndexWriter#rollbackInternalNoCommit`. After calling `IndexWriter#commit` to throw `AlreadyClosedException`, It is necessary to call `IndexWriter#close` in finalize. ``` try { writer.commit(); writer.close(); success = true; } catch ( @SuppressWarnings("unused") AlreadyClosedException ace) { // OK: abort closes the writer assertTrue(writer.isDeleterClosed()); } catch ( @SuppressWarnings("unused") IOException ioe) { writer.rollback(); failure.clearDoFail(); } finally { writer.close(); } ``` ### Related issues [10930](https://github.com/apache/lucene/issues/10930), [13552](https://github.com/apache/lucene/issues/13552) ### Version and environment details _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org