guojialiang92 opened a new issue, #14423:
URL: https://github.com/apache/lucene/issues/14423
### Description
### Description
I found that Test
`TestIndexWriterWithThreads#testIOExceptionDuringWriteSegmentWithThreadsOnlyOnce`
may fail in rare cases. Exception information is as follows:
`java.lang.RuntimeException: MockDirectoryWrapper: cannot close: there are
still 108 open files`
I looked up some other related issues and finally located the problem. At
the same time, I also added a test that can stably reproduce the problem.
### Analysis
The root cause is that `IndexWriter#ensureOpen()` throws
`AlreadyClosedException` when `IndexWriter#closed` or `IndexWriter#closing` is
`true`. In the test code, after throwing `AlreadyClosedException` when
executing `IndexWriter#commit`, the `IndexWriter#close` is not called again to
ensure that `IndexWriter` closes normally.
```
try {
writer.commit();
writer.close();
success = true;
} catch (
@SuppressWarnings("unused")
AlreadyClosedException ace) {
// OK: abort closes the writer
assertTrue(writer.isDeleterClosed());
} catch (
@SuppressWarnings("unused")
IOException ioe) {
writer.rollback();
failure.clearDoFail();
}
```
### To Reproduce
In order to stabilize the reproduce problem, I added a test
`TestIndexWriterWithThreads#testIOExceptionWithMergeNotEndLongTime`.
The code will be executed in the following order:
1. Set `LiveIndexWriterConfig#readerPooling` to `true` to ensure that
`ReaderPool#release` does not release `ReadersAndUpdates`
2. Start an `IndexerThread` and start writing data
3. Wait for a merge thread to start working (using `mergeCountDownLatch`),
then simulate write failure via `MockDirectoryWrapper#failOn`
4. Block the execution of the merge thread until
`IndexWriter#updateDocument` throws an exception due to write failure (using
`updateDocumentFailCountDownLatch`).
5. Merge thread and IW thread will both call
`IndexWriter#maybeCloseOnTragicEvent`, and I control to let Merge thread
execute `IndexWriter#rollbackInternalNoCommit` first, and the IW thread will
skip it. (using `mergeCloseCountDownLatch`)
6. Because the execution of the merge thread is asynchronous, the test will
continue until `IndexWriter#commit` is called and a `AlreadyClosedException` is
throw.
7. In order to prevent `IndexInput` from being closed before calling
`MockDirectoryWrapper#close`, I let the `ConcurrentMergeScheduler#close` sleep
for 5s.
### Solution
The closure of `IndexInput` needs to wait until the execution of
`ReaderPool#close` in `IndexWriter#rollbackInternalNoCommit`. After calling
`IndexWriter#commit` to throw `AlreadyClosedException`, It is necessary to
call `IndexWriter#close` in finalize.
```
try {
writer.commit();
writer.close();
success = true;
} catch (
@SuppressWarnings("unused")
AlreadyClosedException ace) {
// OK: abort closes the writer
assertTrue(writer.isDeleterClosed());
} catch (
@SuppressWarnings("unused")
IOException ioe) {
writer.rollback();
failure.clearDoFail();
} finally {
writer.close();
}
```
### Related issues
[10930](https://github.com/apache/lucene/issues/10930),
[13552](https://github.com/apache/lucene/issues/13552)
### Version and environment details
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
