[
https://issues.apache.org/jira/browse/KAFKA-9076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16957016#comment-16957016
]
Ryanne Dolan commented on KAFKA-9076:
-------------------------------------
[~yangguo1220] I agree this would be an improvement, but there is some
complexity here:
- in order to write offsets for a consumer group, we need to know that the
group is not already running on the target cluster. Otherwise we'd be stepping
on that group's current offsets. The group coordinator won't allow this afaik.
- we could kick out a target group and force it to seek to the new offsets by
revoking group membership and forcing a rebalance etc. But we wouldn't want to
do this periodically.
- we could write offsets to a new group ID, eg. us-west group1, just like we do
with topics, s.t. we avoid the above issues. Then migrating groups would
involve changing the group ID. That works fine, but consumers would need a way
to determine which group ID to use. Translating group ID like that is more
cumbersome than translating offsets, since offsets can be altered using
existing tools, but there is no way to tell a consumer to change its group ID.
I think there are scenarios where automatically writing offsets as you propose
might make sense, e.g. in an active/standby scenario where consumers only
connect to one cluster at a time. But if you are automating that behavior, you
might as well automate the offset translation via RemoteClusterUtils, IMO.
My team has built external tooling using RemoteClusterUtils that works with
existing consumers. It's possible to fully automate failover and failback this
way. I'm skeptical that automatically writing offsets as you propose would make
this process simpler.
An alternative to automatically writing offsets is to provide such tooling,
e.g. as part of kafka-consumer-groups command. In addition to resetting
consumers to a particular timestamp or offset, the tool could cause consumers
to seek to the latest MM2 checkpoint.
> MirrorMaker 2.0 automated consumer offset sync
> ----------------------------------------------
>
> Key: KAFKA-9076
> URL: https://issues.apache.org/jira/browse/KAFKA-9076
> Project: Kafka
> Issue Type: Improvement
> Components: mirrormaker
> Affects Versions: 2.4.0
> Reporter: Ning Zhang
> Priority: Major
> Labels: mirrormaker, pull-request-available
> Fix For: 2.5.0
>
>
> To calculate the translated consumer offset in the target cluster, currently
> `Mirror-client` provides a function called "remoteConsumerOffsets()" that is
> used by "RemoteClusterUtils" for one-time purpose.
> In order to make the consumer migration from source to target cluster
> transparent and convenient, e.g. in event of source cluster failure, it is
> better to have a background job to continuously and periodically sync the
> consumer offsets from the source to target cluster, so that when the consumer
> switches to the target cluster, it will resume to consume from where it left
> off at source cluster.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
