BryceKan3 commented on code in PR #15428:
URL: https://github.com/apache/lucene/pull/15428#discussion_r2543108031
##########
lucene/core/src/java/org/apache/lucene/index/StandardDirectoryReader.java:
##########
@@ -88,10 +95,37 @@ protected DirectoryReader doBody(String segmentFileName)
throws IOException {
SegmentInfos.readCommit(directory, segmentFileName,
minSupportedMajorVersion);
final SegmentReader[] readers = new SegmentReader[sis.size()];
try {
- for (int i = sis.size() - 1; i >= 0; i--) {
- readers[i] =
- new SegmentReader(
- sis.info(i), sis.getIndexCreatedVersionMajor(),
IOContext.DEFAULT);
+ if (executor != null) {
+ List<Future<SegmentReader>> futures = new ArrayList<>();
+ for (int i = sis.size() - 1; i >= 0; i--) {
+ final int index = i;
+ // parallelize segment reader initialization
+ futures.add(
+ (executor)
+ .submit(
+ () ->
+ new SegmentReader(
+ sis.info(index),
+ sis.getIndexCreatedVersionMajor(),
+ IOContext.DEFAULT)));
+ }
+ RuntimeException firstException = null;
+ for (int i = 0; i < futures.size(); i++) {
+ try {
+ readers[sis.size() - 1 - i] = futures.get(i).get();
+ } catch (ExecutionException | InterruptedException e) {
+ // If there is an exception creating the reader we still
process
+ // the rest of the completed futures to allow us to close
created readers
+ if (firstException == null) firstException = new
RuntimeException(e);
+ }
+ }
+ if (firstException != null) throw firstException;
+ } else {
+ for (int i = sis.size() - 1; i >= 0; i--) {
+ readers[i] =
+ new SegmentReader(
+ sis.info(i), sis.getIndexCreatedVersionMajor(),
IOContext.DEFAULT);
+ }
Review Comment:
That's an interesting optimization! - I would like to keep the scope of
this PR to just implementing the base concurrency with the ExecutorService and
then we can have future improvements like this one that we can add. Let me know
your thoughts!
##########
lucene/CHANGES.txt:
##########
@@ -46,6 +46,8 @@ API Changes
* GITHUB#15187: Restrict visibility of PerFieldKnnVectorsFormat.FieldsReader
(Simon Cooper)
+* GITHUB##15428: Add support for ExecutorServices to be passed into
DirectoryReader.open() API (Bryce Kane)
Review Comment:
Will update!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
