vigyasharma commented on PR #14900: URL: https://github.com/apache/lucene/pull/14900#issuecomment-3046261414
@jpountz I feel the scheduler should have some control on how the executor is created, its backing queue etc, so that it can prioritize how merges get scheduled, e.g. pick smaller merges before large ones, or have merges smaller than `MIN_BIG_MERGE_MB` so straight through without waiting. Otherwise, all the "merge scheduling" logic really gets offloaded to how the thread pool manages its task queue. That could be hard to get right, but maybe it's okay for expert users? We could have the multi-tenant CMS accept a constructor argument for a custom `MergeTasksExecutorService` defined in Lucene. It will maintain a fixed thread pool, use a fixed size priority blocking queue that prefers small merges over big ones, and add custom logic to directly execute small merges, have bounded wait time on merges, use calling thread when needed etc. Something on the lines of Lucene's `TaskExecutor` that you mentioned. On similar lines, we can also create an `IndexingMergeSharedExecutorService` to share threads across indexing and merge tasks. We'll probably need to write a custom executor service; I'm not sure if wrapping an executor will give us the hooks we need. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
