sohami opened a new issue, #12347: URL: https://github.com/apache/lucene/issues/12347
### Description For concurrent segment search, lucene uses the slices method to compute the number of work units which can be processed concurrently. a) It calculates slices in the [constructor of IndexSearcher](https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/search/IndexSearcher.java#L239) with default thresholds for document count and segment counts. b) Provides an implementation of [SliceExecutor (i.e. QueueSizeBasedExecutor)](https://github.com/apache/lucene/blob/main/lucene/core/src/java/org/apache/lucene/search/IndexSearcher.java#L1008) based on executor type which applies the backpressure in concurrent execution based on a limiting factor of 1.5 times the passed in threadpool maxPoolSize. In OpenSearch, there is a search threadpool which serves the search request to all the lucene indices (or OpenSearch shards) assigned to a node. Each node can get the requests to some or all the indices on that node. I am exploring a mechanism such that I can dynamically control the max slices for each lucene index search request. For example: search requests to some indices on that node to have max 4 slices each and others to have 2 slices each. Then the threadpool shared to execute these slices does not have any limiting factor. In this model the top level search threadpool will limit the number of active search requests which will limit the number of work units in the SliceExecutor threadpool. For this the derived implementation of IndexSearcher can get an input value in the constructor to control the slice count computation. Even though the `slice` method is `protected` it gets called from the constructor of base `IndexSearcher` class which prevents the derived class from using the passed in input. To achieve this I am making change along the lines as suggested on discussion thread in dev mailing list to get some feedback -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
