BryceKan3 commented on PR #15428: URL: https://github.com/apache/lucene/pull/15428#issuecomment-3564338292
Hey dianjifzm, Thanks so much for the feedback! Very happy to hear that this feature will be something that you will benefit from as well. ForkJoinPool is an ExecutorService - with these changes you will be able to directly pass this in the commonPool() as your ExecutorService if you want! Some clients need fine grained threading control and for those they usually manage threadpools themselves, by allowing support to pass in an ExecutorService these users will be able to benefit from this change. As for using Arrays.parallelSetAll - this will utilize the ForkJoinPool, we don't want to do this by default as it can cause unexpected resource contention if the client is managing their own threads. As far as openIfChanged() - this method does [create SegmentReaders](https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/index/StandardDirectoryReader.java#L226) and as Mike and Vigya mentioned above there can be scenarios where if we have enough new segments we will be able to benefit from this change as well in that API. SegmentReader initialization taking 10 minutes seems extremely slow! In my profiling I have found most of the overhead is checksumming files, curious if that is the case for you as well... I wonder if there is some room on the table for some additional optimization in those code paths. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
