vigyasharma commented on PR #14900: URL: https://github.com/apache/lucene/pull/14900#issuecomment-3046266969
Based on all this, I think we get the following structure: 1. A merge scheduler that accepts an `ExecutorService` in its ctor. It will offload most of the scheduling / queueing / throttling logic to the executor service and its backing queue implementation. We will recommend using `MergeTasksExecutorService` or `IndexingMergeSharedExecutorService` but users are free to pass in their own executors. 2. A `MergeTasksExecutorService` which does CMS like scheduling but for merge tasks only. The fixed thread pool is shared across all submitted merges across all writers. This executor service ~~assumes~~ requires a separate indexing thread pool, and puts back-pressure on indexing when merges accumulate e.g. by making the merge run on the indexing thread which tries to submit it. 3. A `IndexingMergeSharedExecutorService` that can effectively handle sharing threads b/w indexing and merge tasks. FWIW, I'm not really sure if sharing a thread pool b/w indexing and merging would be simpler than having separate thread pools and applying backpressure on indexing. I still need to grok all the details. But if we think backpressure is the way to go, and we'll always only use the `MergeTasksExecutorService`, then writing this executor as part of the scheduler and making it a singleton might make the code simpler? OTOH, if we want a shared indexing/merging thread pool, or foresee a need to have different thread pools for sets of writers (shards in Elasticsearch / OpenSearch), then the ctor arg for executor service makes sense. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
