swaminathanmanish commented on code in PR #16571:
URL: https://github.com/apache/pinot/pull/16571#discussion_r2268973856
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/minion/PinotTaskManager.java:
##########
@@ -238,6 +238,15 @@ public Map<String, String> createTask(String taskType,
String tableName, @Nullab
LOGGER.warn("No ad-hoc task generated for task type: {}", taskType);
continue;
}
+ int maxNumberOfSubTasks = taskGenerator.getMaxNumSubTasks();
Review Comment:
For adhoc trigger, would it make sense to fail task generation to make it
explicit to the user that they need to take an action (bump global limit/limit
input ?
For adhoc call, we trigger all subtasks to get the job done. These triggers
can be from external systems like airflow and we will end up implicitly doing
partial work by limiting number of subtasks, which the user will not even know.
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/minion/generator/BaseTaskGenerator.java:
##########
@@ -81,6 +81,52 @@ public int getNumConcurrentTasksPerInstance() {
return JobConfig.DEFAULT_NUM_CONCURRENT_TASKS_PER_INSTANCE;
}
+ @Override
+ public int getMaxNumSubTasks() {
+ String configKey = MinionConstants.MAX_ALLOWED_SUB_TASKS_KEY;
Review Comment:
Can we name this as minion_global_max_sub_tasks_limit or something
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/minion/generator/BaseTaskGenerator.java:
##########
@@ -81,6 +81,52 @@ public int getNumConcurrentTasksPerInstance() {
return JobConfig.DEFAULT_NUM_CONCURRENT_TASKS_PER_INSTANCE;
}
+ @Override
+ public int getMaxNumSubTasks() {
Review Comment:
This can be private method right or folded into getNumSubTasks?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
