lucasbru opened a new pull request, #20458: URL: https://github.com/apache/kafka/pull/20458
In the original algorithm, standby tasks are assigned to a process that previously owned the task only if it is “load-balanced”, meaning the process has fewer tasks that members, or it is the least loaded process. This is strong requirement, and will cause standby tasks to often not get assigned to process that previously owned it. Furthermore, the condition “is the least loaded process” is hard to evaluate efficiently in this context. We propose to instead use the same “below-quota” condition as in active task assignment. We compute a quota for active and standby tasks, by definiing numOfTasks = numberOfActiveTasks+numberOfStandbyTasks and defining the quota as numOfTasks/numberOfMembers rounded up. Whenever a member becomes “full” (its assigned number of tasks is equal to numOfTasks) we deduct its tasks from numOfTasks and decrement numberOfMembers and recompute the quota, which means that the quota may be reduced by one during the assignment process, to avoid uneven assignments. A standby task can be assigned to a process that previously owned it, whenever the process has fewer than numOfMembersOfProcess*quota. This condition will, again, prioritize standby stickyness, and can be evaluated in constant time. In our worst-performing benchmark, this improves the runtime by 2.5x on top of the previous optimizations, but 5x on the more important incremental assignment case. # This is a stacked PR. Only review the last commit -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
