benwtrent commented on PR #13124: URL: https://github.com/apache/lucene/pull/13124#issuecomment-1959856815
> For instance the merge scheduler could return a custom Executor that dynamically decides to run a new task in the current thread or to fork to a separate thread depending on how many threads are currently busy across all merges vs. the current value of ConcurrentMergeScheduler.maxThreadCount? This makes me think that instead of it being a separate executor it should return a dynamic value for `numParallelMergeWorkers` or if `numParallelMergeWorkers==1` return `null` so that there is no parallelism. This would require some sort of "planning" on the merge scheduler (knowing how many are queued and how many `numParallelMergeWorkers` it provided to each executing merge action). I guess we could rely on an executor's `getActiveCount()` but that seems trappy. This all implies that the configuration will actually `maxParallelMergeWorkers` and the MergeScheduler is free to provide any number of workers up to that limit. The tricky part to me is determining 'how busy' the current merges are. The MergeScheduler could pass `numParallelMergeWorkers` to the `SegmentMerger`, but it goes completely unused. Maybe this is OK and we just assume it will be used. Another confusion is determining the total number of threads allowed for merging (inter/intra). We could default this to `maxParallelMergeWorkers * maxThreadCount`. In this instance `maxParallelMergeWorkers == 1` would behave as the current working. `maxParallelMergeWorkers == 2` would mean that we potentially use twice as many resources and the user should adjust `maxThreadCount` accordingly. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
