mccullocht commented on code in PR #15224:
URL: https://github.com/apache/lucene/pull/15224#discussion_r2395370732
##########
lucene/misc/src/java/org/apache/lucene/misc/store/DirectIODirectory.java:
##########
@@ -554,8 +642,178 @@ public IndexInput slice(String sliceDescription, long
offset, long length) throw
"slice() " + sliceDescription + " out of bounds: " + this);
}
var slice = new DirectIOIndexInput(sliceDescription, this, this.offset +
offset, length);
+ // TODO figure out how to make this async
slice.seekInternal(0L);
return slice;
}
}
+
+ /** A prefetcher that can prefetch multiple chunks of data from a
FileChannel using direct IO. */
+ private static class DirectIOPrefetcher implements Closeable {
+ private final int maxConcurrentPrefetches;
+ private final int maxTotalPrefetches;
+ private final int blockSize;
+ private final long[] prefetchPos;
+ private final Future<?>[] prefetchThreads;
+ private final TreeMap<Long, Integer> posToSlot;
+ private final Deque<Integer> slots;
+ private final ByteBuffer[] prefetchBuffers;
+ private final IOException[] prefetchExceptions;
+ private final int prefetchBytesSize;
+ private final Deque<Long> pendingPrefetches = new ArrayDeque<>();
+ private final FileChannel channel;
+ private final ExecutorService executor =
Executors.newVirtualThreadPerTaskExecutor();
Review Comment:
Is the executor something you would want to share within a directory or
potentially even across directories? I can't find any documentation that
indicates that this pattern would be a problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
