msokolov commented on code in PR #13124:
URL: https://github.com/apache/lucene/pull/13124#discussion_r1513268905
##########
lucene/core/src/java/org/apache/lucene/index/ConcurrentMergeScheduler.java:
##########
@@ -910,4 +936,68 @@ public void setSuppressExceptions(ConcurrentMergeScheduler
cms) {
}
});
}
+
+ private class ScaledExecutor implements Executor {
+
+ private final AtomicInteger activeCount = new AtomicInteger(0);
+ private final ThreadPoolExecutor executor;
+
+ public ScaledExecutor() {
+ this.executor =
+ new ThreadPoolExecutor(
+ 0, Math.max(1, maxThreadCount), 1L, TimeUnit.MINUTES, new
SynchronousQueue<>());
+ }
+
+ void shutdown() {
+ executor.shutdown();
+ }
+
+ private void updatePoolSize() {
+ executor.setMaximumPoolSize(Math.max(1, maxThreadCount));
+ }
+
+ @Override
+ public void execute(Runnable command) {
+ assert mergeThreads.contains(Thread.currentThread()) : "caller is not a
merge thread";
+ Thread currentThread = Thread.currentThread();
+ if (currentThread instanceof MergeThread mergeThread) {
+ execute(mergeThread, command);
+ } else {
+ command.run();
+ }
+ }
+
+ private void execute(MergeThread mergeThread, Runnable command) {
+ // don't do multithreaded merges for small merges
+ if (mergeThread.merge.estimatedMergeBytes < MIN_BIG_MERGE_MB * 1024 *
1024) {
+ command.run();
+ } else {
+ final boolean isThreadAvailable;
+ // we need to check if a thread is available before submitting the
task to the executor
+ // synchronize on CMS to get an accurate count of current threads
+ synchronized (ConcurrentMergeScheduler.this) {
+ int max = maxThreadCount - mergeThreads.size();
+ int value = activeCount.get();
+ if (value < max) {
+ activeCount.incrementAndGet();
+ isThreadAvailable = true;
+ } else {
+ isThreadAvailable = false;
+ }
+ }
Review Comment:
I am going to just make stuff up here since I don't know this code well at
all -- but I think I was assuming there was a ThreadPoolExecutor somewhere that
was doling out threads for both of these operations. Now I'm realizing perhaps
that's not the case and CMS is doing all that tracking on its own. I wonder if
we should open a separate issue to migrate CMS to use JDK's thread pool
abstractions? It could make some of this easier to handle.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
