kaushik srinivas created KAFKA-17101:
----------------------------------------
Summary: Mirror maker internal topics cleanup policy changes to
'delete' from 'compact'
Key: KAFKA-17101
URL: https://issues.apache.org/jira/browse/KAFKA-17101
Project: Kafka
Issue Type: Bug
Affects Versions: 3.6.1, 3.5.1, 3.4.1
Reporter: kaushik srinivas
Scenario/Setup details
Kafka cluster 1: 3 replicas
Kafka cluster 2: 3 replicas
MM1 moving data from cluster 1 to cluster 2
MM2 moving data from cluster 2 to cluster 1
Sometimes with a reboot of the kafka cluster 1 and MM1 instance, we observe MM
failing to come up with below exception,
{code:java}
{"message":"DistributedHerder-connect-1-1 -
org.apache.kafka.connect.runtime.distributed.DistributedHerder - [Worker
clientId=connect-1, groupId=site1-mm2] Uncaught exception in herder work
thread, exiting: "}}
org.apache.kafka.common.config.ConfigException: Topic
'mm2-offsets.site1.internal' supplied via the 'offset.storage.topic' property
is required to have 'cleanup.policy=compact' to guarantee consistency and
durability of source connector offsets, but found the topic currently has
'cleanup.policy=delete'. Continuing would likely result in eventually losing
source connector offsets and problems restarting this Connect cluster in the
future. Change the 'offset.storage.topic' property in the Connect worker
configurations to use a topic with 'cleanup.policy=compact'. {code}
Once the topic is altered with cleanup policy of compact. MM works just fine.
This is happening on our setups sporadically and across varieties of scenarios.
Not been successful in identifying the exact reproduction steps as of now.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
