jainankitk commented on code in PR #15428:
URL: https://github.com/apache/lucene/pull/15428#discussion_r2540003560
##########
lucene/core/src/java/org/apache/lucene/index/StandardDirectoryReader.java:
##########
@@ -88,10 +95,37 @@ protected DirectoryReader doBody(String segmentFileName)
throws IOException {
SegmentInfos.readCommit(directory, segmentFileName,
minSupportedMajorVersion);
final SegmentReader[] readers = new SegmentReader[sis.size()];
try {
- for (int i = sis.size() - 1; i >= 0; i--) {
- readers[i] =
- new SegmentReader(
- sis.info(i), sis.getIndexCreatedVersionMajor(),
IOContext.DEFAULT);
+ if (executor != null) {
+ List<Future<SegmentReader>> futures = new ArrayList<>();
+ for (int i = sis.size() - 1; i >= 0; i--) {
+ final int index = i;
+ // parallelize segment reader initialization
+ futures.add(
+ (executor)
+ .submit(
+ () ->
+ new SegmentReader(
+ sis.info(index),
+ sis.getIndexCreatedVersionMajor(),
+ IOContext.DEFAULT)));
+ }
+ RuntimeException firstException = null;
+ for (int i = 0; i < futures.size(); i++) {
+ try {
+ readers[sis.size() - 1 - i] = futures.get(i).get();
+ } catch (ExecutionException | InterruptedException e) {
+ // If there is an exception creating the reader we still
process
+ // the rest of the completed futures to allow us to close
created readers
+ if (firstException == null) firstException = new
RuntimeException(e);
+ }
+ }
+ if (firstException != null) throw firstException;
+ } else {
+ for (int i = sis.size() - 1; i >= 0; i--) {
+ readers[i] =
+ new SegmentReader(
+ sis.info(i), sis.getIndexCreatedVersionMajor(),
IOContext.DEFAULT);
+ }
Review Comment:
Right now, we are not using the caller thread to create segment readers. I
am wondering if we can/should do that similar to this PR -
https://github.com/apache/lucene/pull/13472. Although that makes the code
change slightly complex, but might add slightly better concurrency.
##########
lucene/CHANGES.txt:
##########
@@ -46,6 +46,8 @@ API Changes
* GITHUB#15187: Restrict visibility of PerFieldKnnVectorsFormat.FieldsReader
(Simon Cooper)
+* GITHUB##15428: Add support for ExecutorServices to be passed into
DirectoryReader.open() API (Bryce Kane)
Review Comment:
Are we not targeting this change for Lucene 10.4 release? If yes, we should
move the entry to 10.4 section.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
