somandal commented on PR #16857: URL: https://github.com/apache/pinot/pull/16857#issuecomment-3320806335
@krishan1390 one more thing I wanted to point out in my implementation - for scheduled tasks, I actually try to acquire all locks across all tables, and release them all after task generation is done for all tables Reason behind this is that we essentially perform two steps: - Generate all tasks for all tables of the given task type - Then we update Helix with the generated tasks as a separate step My understanding (and please correct me if I'm wrong) is that we want to hold the lock all the way up to when we actually update Helix with the task list, correct? Not just for generating the tasks, because if we happen to release the lock before we update Helix, that can causes races where another controller may update Helix before we get a chance to (in case of delays etc). If needed, we can perhaps combine the two steps (but prefer doing this a separate PR) before moving onto the next table, but not sure if there are caveats with that approach so wanted to discuss before trying that out. If we move to that model, we can always take just the single lock instead of taking all the table locks. I do handle the scenario where we cannot get a table lock by skipping task generation for that table. let me know your thoughts, thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
