C0urante commented on PR #16053:
URL: https://github.com/apache/kafka/pull/16053#issuecomment-2146086988
Thanks @gharris1727! I've updated the patch to apply to both distributed and
standalone mode, and fixed failing unit tests.
I've also tweaked the logic so that applied connector configs are tracked
regardless of whether the worker has completed startup or not, which should
eliminate unnecessary cluster churn on upgrades. This still "primes the pump"
for future connector config updates though, which I believe should make us both
very happy!
Going through the scenarios you listed in your analysis (which was correct
based on the implementation you reviewed):
- The first one (steps 1-3) should never occur now. If a connector has
generated task configs after its latest configuration was submitted by the
user, the worker should have an applied config for it; if the connector has not
generated task configs, then we should force a reconfiguration attempt, just
like we would today.
- The second one (steps 4-6) should remain correct ("Rebalancing a connector
among existing workers in a cluster won't force new task configs to be
generated"), because workers must be caught up on the config topic before
joining a group, and before resuming work after rejoining a group
- The third one (steps 7-8) should remain as correct as it was in the
original implementation. We eagerly transform applied connector configs as soon
as the corresponding set of task configs is generated, which does mean that
there may be some lag time between when the connector config was submitted and
when we performed the transform. Ultimately though, there is an eventual
consistency-esque guarantee that if a config provider changes periodically, new
task configs will at some point be forcibly written to the config topic.
- The fourth one (steps 9-11) should remain correct as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
