[
https://issues.apache.org/jira/browse/KAFKA-15985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Philipp Dallig updated KAFKA-15985:
-----------------------------------
Description:
We are currently trying to migrate between two Kafka clusters using Mirrormaker2
new kafka cluster version: 7.5.2-ccs
old kafka cluster version: kafka_2.13-2.8.0
The Mirrormaker 2 process runs on the new cluster (target cluster)
My main problem: The lag in the target cluster is not the same as in the source
cluster.
target cluster
{code}
GROUP TOPIC PARTITION CURRENT-OFFSET
LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
test-sync-5 kafka-replication-test-5 0 3637 3668
31 - - -
{code}
source cluster
{code}
GROUP TOPIC PARTITION CURRENT-OFFSET
LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
test-sync-5 kafka-replication-test-5 0 3668 3668
0 - - -
{code}
MM2 configuration without connection properties.
{code}
t-kafka->t-extkafka.enabled = true
t-kafka->t-extkafka.topics = ops_filebeat, kafka-replication-.*
t-kafka->t-extkafka.sync.topic.acls.enabled = false
t-kafka->t-extkafka.sync.group.offsets.enabled = true
t-kafka->t-extkafka.sync.group.offsets.interval.seconds = 30
t-kafka->t-extkafka.refresh.groups.interval.seconds = 30
t-kafka->t-extkafka.offset-syncs.topic.location = target
t-kafka->t-extkafka.emit.checkpoints.interval.seconds = 30
t-kafka->t-extkafka.replication.policy.class =
org.apache.kafka.connect.mirror.IdentityReplicationPolicy
{code}
was:
We are currently trying to migrate between two Kafka clusters using Mirrormaker2
new kafka cluster version: 7.5.2-ccs
old kafka cluster version: kafka_2.13-2.8.0
The Mirrormaker 2 process runs on the new cluster (target cluster)
My main problem: The lag in the target cluster is not the same as in the source
cluster.
target cluster
{code}
GROUP TOPIC PARTITION CURRENT-OFFSET
LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
test-sync-5 kafka-replication-test-5 0 3637 3668
31 - - -
{code}
source cluster
{code}
GROUP TOPIC PARTITION CURRENT-OFFSET
LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
test-sync-5 kafka-replication-test-5 0 3668 3668
0 - - -
{code}
> Mirrormaker 2 offset sync is incomplete
> ---------------------------------------
>
> Key: KAFKA-15985
> URL: https://issues.apache.org/jira/browse/KAFKA-15985
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 3.5.1
> Reporter: Philipp Dallig
> Priority: Major
>
> We are currently trying to migrate between two Kafka clusters using
> Mirrormaker2
> new kafka cluster version: 7.5.2-ccs
> old kafka cluster version: kafka_2.13-2.8.0
> The Mirrormaker 2 process runs on the new cluster (target cluster)
> My main problem: The lag in the target cluster is not the same as in the
> source cluster.
> target cluster
> {code}
> GROUP TOPIC PARTITION CURRENT-OFFSET
> LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
> test-sync-5 kafka-replication-test-5 0 3637 3668
> 31 - - -
> {code}
> source cluster
> {code}
> GROUP TOPIC PARTITION CURRENT-OFFSET
> LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
> test-sync-5 kafka-replication-test-5 0 3668 3668
> 0 - - -
> {code}
> MM2 configuration without connection properties.
> {code}
> t-kafka->t-extkafka.enabled = true
> t-kafka->t-extkafka.topics = ops_filebeat, kafka-replication-.*
> t-kafka->t-extkafka.sync.topic.acls.enabled = false
> t-kafka->t-extkafka.sync.group.offsets.enabled = true
> t-kafka->t-extkafka.sync.group.offsets.interval.seconds = 30
> t-kafka->t-extkafka.refresh.groups.interval.seconds = 30
> t-kafka->t-extkafka.offset-syncs.topic.location = target
> t-kafka->t-extkafka.emit.checkpoints.interval.seconds = 30
> t-kafka->t-extkafka.replication.policy.class =
> org.apache.kafka.connect.mirror.IdentityReplicationPolicy
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
