albertobastos commented on code in PR #15143:
URL: https://github.com/apache/pinot/pull/15143#discussion_r1975906078
##########
pinot-spi/src/main/java/org/apache/pinot/spi/executor/ExecutorServiceUtils.java:
##########
@@ -132,4 +133,14 @@ public static void close(ExecutorService executorService,
long terminationMillis
throw new RuntimeException(e);
}
}
+
+ /**
+ * Returns the hard limit of the number of threads that can be used by the
multi-stage executor.
+ * @param config Pinot configuration
+ * @return hard limit of the number of threads that can be used by the
multi-stage executor (no hard limit if <= 0)
+ */
+ public static int getMultiStageExecutorHardLimit(PinotConfiguration config) {
+ return
config.getProperty(CommonConstants.Helix.CONFIG_OF_MULTI_STAGE_ENGINE_MAX_SERVER_QUERY_THREADS,
0)
+ *
CommonConstants.Helix.MULTI_STAGE_ENGINE_MAX_SERVER_QUERY_THREADS_HARDLIMIT_FACTOR;
+ }
Review Comment:
Yeah, I agree, moved it to the Executor itself.
5a451ae6ca1239c679323725066c457b47d6e2b1
##########
pinot-common/src/main/java/org/apache/pinot/common/utils/HardLimitExecutorServicePlugin.java:
##########
@@ -0,0 +1,63 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.common.utils;
+
+import com.google.auto.service.AutoService;
+import java.util.concurrent.ExecutorService;
+import org.apache.pinot.spi.executor.ExecutorServicePlugin;
+import org.apache.pinot.spi.executor.ExecutorServiceProvider;
+import org.apache.pinot.spi.executor.ExecutorServiceUtils;
+import org.apache.pinot.spi.executor.HardLimitExecutor;
+import org.apache.pinot.spi.utils.CommonConstants;
+
+
+/**
+ * This is the plugin for the cached executor service with a hard limit for
the amount of created threads.
+ *
+ * The provider included in this plugin creates cached thread pools, which are
the recommended executor service for
+ * cases where the tasks are short-lived and not CPU bound.
+ *
+ * If that is not the case, this executor may create a large number of threads
that will be competing for CPU resources,
+ * which may lead to performance degradation and even system instability.
+ * In that case {@link FixedExecutorServicePlugin} could be used, but it may
need changes to the code to avoid
+ * deadlocks. Deployments using Java 21 or above could consider using a
virtual thread executor service plugin.
+ *
+ * @see org.apache.pinot.spi.executor.ExecutorServiceUtils
+ */
+@AutoService(ExecutorServicePlugin.class)
+public class HardLimitExecutorServicePlugin implements ExecutorServicePlugin {
+ @Override
+ public String id() {
+ return "hardlimit";
+ }
+
+ @Override
+ public ExecutorServiceProvider provider() {
+ return (conf, confPrefix, baseName) -> {
+ int limit = ExecutorServiceUtils.getMultiStageExecutorHardLimit(conf);
+ ExecutorService executorService = ExecutorServiceUtils.create(
+ conf, CommonConstants.Server.MULTISTAGE_EXECUTOR_CONFIG_PREFIX,
baseName,
+ CommonConstants.Server.DEFAULT_MULTISTAGE_EXECUTOR_TYPE);
+ if (limit > 0) {
+ executorService = new HardLimitExecutor(limit, executorService);
+ }
+ return executorService;
+ };
+ }
Review Comment:
Is no longer needed once we decide to not use the Provider API for that.
5a451ae6ca1239c679323725066c457b47d6e2b1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
