somandal opened a new pull request, #16857: URL: https://github.com/apache/pinot/pull/16857
Scheduled minion tasks are generated by the lead controller and the methods to generate the tasks are synchronized, ensuring that multiple tasks for the same table aren't generated. Pinot allows ad-hoc task creation as well, but this is performed on the controller that gets the request. This can result in scenarios where a task generation happens in parallel on multiple controllers. This PR addresses this by implementing a distributed locking mechanism to guard minion task creation across controllers vis the use of EPHEMERAL_SEQUENTIAL nodes. This PR also adds an API to forceRelease the lock for scenarios where the lock may be left behind incorrectly. Testing done using the HybridQuickStart to ensure that the features added work, and some unit tests have been added. Some other options considered: - Using a PERSISTENT lock with a timeout: https://github.com/apache/pinot/pull/16631 - task generation can take a long time, so choosing a timeout for clean-up is tricky - Forward all task generation requests to the lead controller of that table - this can provide automatic synchronization but has the disadvantage of how to handle lead controller changes for a given table. In such cases, it is still possible for the task generation to run on two different controllers. So some kind of distributed locking is still necessary cc @krishan1390 @shounakmk219 @swaminathanmanish @npawar -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
