guojialiang92 opened a new issue, #14423:
URL: https://github.com/apache/lucene/issues/14423

   ### Description
   
   ### Description
   I found that Test 
`TestIndexWriterWithThreads#testIOExceptionDuringWriteSegmentWithThreadsOnlyOnce`
 may fail in rare cases. Exception information is as follows: 
   `java.lang.RuntimeException: MockDirectoryWrapper: cannot close: there are 
still 108 open files`
   
   I looked up some other related issues and finally located the problem. At 
the same time, I also added a test that can stably reproduce the problem.
   
   ### Analysis
   The root cause is that `IndexWriter#ensureOpen()` throws 
`AlreadyClosedException` when `IndexWriter#closed` or `IndexWriter#closing` is 
`true`. In the test code, after throwing `AlreadyClosedException` when 
executing `IndexWriter#commit`, the `IndexWriter#close` is not called again to 
ensure that `IndexWriter` closes normally.
   ```
   try {
     writer.commit();
     writer.close();
     success = true;
   } catch (
       @SuppressWarnings("unused")
       AlreadyClosedException ace) {
     // OK: abort closes the writer
     assertTrue(writer.isDeleterClosed());
   } catch (
       @SuppressWarnings("unused")
       IOException ioe) {
     writer.rollback();
     failure.clearDoFail();
   }
   ```
   
   ### To Reproduce
   In order to stabilize the reproduce problem, I added a test 
`TestIndexWriterWithThreads#testIOExceptionWithMergeNotEndLongTime`.
   
   The code will be executed in the following order:
   
   1. Set `LiveIndexWriterConfig#readerPooling` to `true` to ensure that 
`ReaderPool#release` does not release `ReadersAndUpdates`
   2. Start an `IndexerThread` and start writing data
   3. Wait for a merge thread to start working (using `mergeCountDownLatch`), 
then simulate write failure via `MockDirectoryWrapper#failOn`
   4. Block the execution of the merge thread until 
`IndexWriter#updateDocument` throws an exception due to write failure (using 
`updateDocumentFailCountDownLatch`).
   5. Merge thread and IW thread will both call 
`IndexWriter#maybeCloseOnTragicEvent`, and I control to let Merge thread 
execute `IndexWriter#rollbackInternalNoCommit` first, and the IW thread will 
skip it. (using `mergeCloseCountDownLatch`)
   6. Because the execution of the merge thread is asynchronous, the test will 
continue until `IndexWriter#commit` is called and a `AlreadyClosedException` is 
throw.
   7. In order to prevent `IndexInput` from being closed before calling 
`MockDirectoryWrapper#close`, I let the `ConcurrentMergeScheduler#close` sleep 
for 5s.
   
   ### Solution
   The closure of `IndexInput` needs to wait until the execution of 
`ReaderPool#close` in `IndexWriter#rollbackInternalNoCommit`. After calling 
`IndexWriter#commit` to throw `AlreadyClosedException`,  It is necessary to 
call `IndexWriter#close` in finalize.
   
   ```
   try {
     writer.commit();
     writer.close();
     success = true;
   } catch (
           @SuppressWarnings("unused")
           AlreadyClosedException ace) {
     // OK: abort closes the writer
     assertTrue(writer.isDeleterClosed());
   } catch (
           @SuppressWarnings("unused")
           IOException ioe) {
     writer.rollback();
     failure.clearDoFail();
   } finally {
     writer.close();
   }
   ```
   
   ### Related issues
   [10930](https://github.com/apache/lucene/issues/10930), 
[13552](https://github.com/apache/lucene/issues/13552)
   
   ### Version and environment details
   
   _No response_


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org

Reply via email to