uschindler commented on issue #13325: URL: https://github.com/apache/lucene/issues/13325#issuecomment-2230338249
To be clear: With raising number of threads, the process of closing an indexinput gets slower. With recent Lucene improve,ments this is mitigated a bit by grouping files together and read READONCE files with a confined arena, but for all other IndexInputs which are in multithreaded use, the closing thread has to wait for the handshake on ALL other threads, so it does not scale well when there are many threads. This is needed for safety to ensure no other thread is accesssing the index while we are closing. There is no way to work around that as it is the required safety applied by the JVM (that we are currently missing in Lucene, this is why you can crush the whole VM when you close an IndexReader while threads are accessing it). Because of this, the JDK maintainers recommend to possibly do the closing of indexes asynchronously in production environments. So basically in the normal processing of Solr, reopen the index as usual, but when actually closing the old IndexReader defer this to a single-threaded helper. This might also be a good idea for Elasticsearch. This would allow to open the fresh indexes as soon as possible, but the old segments get closed in a delayed way. We can't implement this inside Lucene as we have no access to thread pools, it is up to the user to possibly close indexes async. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
