BryceKan3 opened a new pull request, #15428: URL: https://github.com/apache/lucene/pull/15428
### Description Currently, as part of DirectoryReader.open() Lucene will [sequentially create segment readers for each segment](https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/index/StandardDirectoryReader.java#L91-L95). This can be a very slow operation due to the I/O on the SegmentReader creation. By adding support for an ExecutorService to be passed in to DirectoryReader.open() we can submit the segment reader creations into the threadpool and achieve significant performance gains in DirectoryReader.open() times. The implementation is fully backwards compatible and allows for the users to pass in their own executor services. I have tested the changes and validated the performance improvement that can be possible by utilizing parallelism for the opening of the directory readers. | Optimization | P50 (ms) | P90 (ms) | P99 (ms)| P50 Reduction %| |----------|----------|----------|------------------| ---------------| | Baseline | 995 | 1020 | 1041 | N/A | Concurrent SegmentReader Initialization | 171 | 178 | 188| 82.81%| Fixes #15387 <!-- If this is your first contribution to Lucene, please make sure you have reviewed the contribution guide. https://github.com/apache/lucene/blob/main/CONTRIBUTING.md --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
