uschindler commented on issue #13325: URL: https://github.com/apache/lucene/issues/13325#issuecomment-2238591954
Hi David, sorry for late response. In fact there problem is quite simple. Every IndexInput#close() costs time in the thread calling this. This time is linear to the number of threads (in plain Java 19-21 at moment). With the fixes by JDK people it will change a bit: - It will still be somehow linear to the total number of threads, but gets faster when not all threads are actually inside a MemorySement read - The will be no overhead anymore on consuming threads that don't use MemorySegments; at the moment it deoptimizes all threads. So with the JDK changes to be applied to Java 22+ (and maybe also 21) the overhead will be almost zero for threads querying Lucene. The work for closing only minimally changes (if at all). So the best we can do is: Don't concurently close IndexInputs in the main threads because on heavy loaded Solr clusters this can take a lot of time. So the idea is to do all "maintenance" on Lucene IndexSearchers/IndexReaders in a small & separate pool of threads. Actively reopen a DirectoryReader (after commit or softcommit) can be done in main thread, but closing the old one can easily be deferred to a small maintenance thread pool. There is no way around this, as the ByteBuffer based MMap will no lobger be possible with later Java versions, as sun.misc.Unsafe will be decommissioned in Java 24 latest. So the new code and its additional safety we have to live with. And that's good. My suggestion would be to to also rethink Solr's thread usage. 10.000 threads is incredibly wrong! Possibly use virtual threads. If you do this, the problem will also go away. Uwe -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
