benwtrent commented on issue #12740: URL: https://github.com/apache/lucene/issues/12740#issuecomment-1954964162
@jpountz @zhaih Looking at @dweiss old proposal I am thinking we should add a `TaskExecutor` to `IndexWriter`. The only thing thats weird is that with `ConcurrentMergeScheduler`, we already have one layer of parallelism from the `IndexWriter`. Do we think its cool to have a `TaskExecutor parallelMergeExecutor` that is passed to the `IndexWriter` and then is passed to all merges kicked off by the `ConcurrentMergeScheduler`? My concern would be if the `parallelMergeExecutor.poolSize() <= ConcurrentMergeScheduler#maxThreadCount` & every merge had a vector field. Wouldn't this effectively make HNSW merging single threaded? Or do I misunderstand how `ConcurrentMergeScheduler` actually works? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
