[
https://issues.apache.org/jira/browse/KAFKA-17445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohit Bobade updated KAFKA-17445:
---------------------------------
Description:
We recently upgraded Kafka streams version to 3.8.0 and are seeing that the
streams app keeps rebalancing and does not process any events
We have explicitly set the config
GROUP_INSTANCE_ID_CONFIG
This is what we see on the broker logs:
[GroupCoordinator 2]: Preparing to rebalance group \{consumer-group-name} in
state PreparingRebalance with old generation 24781 (__consumer_offsets-29)
(reason: Updating metadata for static member {} with instance id {}; client
reason: rebalance failed due to UnjoinedGroupException)
We also tried to remove the GROUP_INSTANCE_ID_CONFIG but then see these logs
and rebalancing and no processing still
sessionTimeoutMs=45000, rebalanceTimeoutMs=1800000,
supportedProtocols=List(stream)) has left group \{groupId} through explicit
`LeaveGroup`; client reason: the consumer unsubscribed from all topics
(kafka.coordinator.group.GroupCoordinator)
other logs show:
during Stable; client reason: need to revoke partitions and re-join)
client reason: triggered followup rebalance scheduled for 0
On the application logs we see:
1. state being restored from changelog topic
2. INFO org.apache.kafka.streams.processor.internals.StreamThread -
stream-thread at state RUNNING: partitions lost due to missed rebalance.
3. Task Migrated exceptions
org.apache.kafka.streams.errors.TaskMigratedException: Error encountered
sending record to topic
org.apache.kafka.common.errors.InvalidProducerEpochException: Producer with
transactionalId
attempted to produce with an old epoch
Written offsets would not be recorded and no more records would be sent since
the producer is fenced, indicating the task may be migrated out; it means all
tasks belonging to this thread should be migrated.
at
org.apache.kafka.streams.processor.internals.RecordCollectorImpl.recordSendError(RecordCollectorImpl.java:306)
~[kafka-streams-3.8.0.jar:?]
at
org.apache.kafka.streams.processor.internals.RecordCollectorImpl.lambda$send$1(RecordCollectorImpl.java:286)
~[kafka-streams-3.8.0.jar:?]
at
datadog.trace.instrumentation.kafka_clients.KafkaProducerCallback.onCompletion(KafkaProducerCallback.java:44)
~[?:?]
at
org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:1106)
~[kafka-clients-3.8.0.jar:?]
was:
We recently upgraded Kafka streams version to 3.8.0 and are seeing that the
streams app keeps rebalancing and does not process any events
We have explicitly set the config
GROUP_INSTANCE_ID_CONFIG
This is what we see on the broker logs:
[GroupCoordinator 2]: Preparing to rebalance group \{consumer-group-name} in
state PreparingRebalance with old generation 24781 (__consumer_offsets-29)
(reason: Updating metadata for static member {} with instance id {}; client
reason: rebalance failed due to UnjoinedGroupException)
We also tried to remove the GROUP_INSTANCE_ID_CONFIG but then see these logs
and rebalancing and no processing still
sessionTimeoutMs=45000, rebalanceTimeoutMs=1800000,
supportedProtocols=List(stream)) has left group \{groupId} through explicit
`LeaveGroup`; client reason: the consumer unsubscribed from all topics
(kafka.coordinator.group.GroupCoordinator)
other logs show:
during Stable; client reason: need to revoke partitions and re-join)
client reason: triggered followup rebalance scheduled for 0
On the application logs we see:
1. state being restored from changelog topic
2. Task Migrated exceptions
org.apache.kafka.streams.errors.TaskMigratedException: Error encountered
sending record to topic
org.apache.kafka.common.errors.InvalidProducerEpochException: Producer with
transactionalId
attempted to produce with an old epoch
Written offsets would not be recorded and no more records would be sent since
the producer is fenced, indicating the task may be migrated out; it means all
tasks belonging to this thread should be migrated.
at
org.apache.kafka.streams.processor.internals.RecordCollectorImpl.recordSendError(RecordCollectorImpl.java:306)
~[kafka-streams-3.8.0.jar:?]
at
org.apache.kafka.streams.processor.internals.RecordCollectorImpl.lambda$send$1(RecordCollectorImpl.java:286)
~[kafka-streams-3.8.0.jar:?]
at
datadog.trace.instrumentation.kafka_clients.KafkaProducerCallback.onCompletion(KafkaProducerCallback.java:44)
~[?:?]
at
org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:1106)
~[kafka-clients-3.8.0.jar:?]
> Kafka streams keeps rebalancing with the following reasons
> ----------------------------------------------------------
>
> Key: KAFKA-17445
> URL: https://issues.apache.org/jira/browse/KAFKA-17445
> Project: Kafka
> Issue Type: Bug
> Components: streams
> Affects Versions: 3.8.0
> Reporter: Rohit Bobade
> Priority: Major
>
> We recently upgraded Kafka streams version to 3.8.0 and are seeing that the
> streams app keeps rebalancing and does not process any events
> We have explicitly set the config
> GROUP_INSTANCE_ID_CONFIG
> This is what we see on the broker logs:
> [GroupCoordinator 2]: Preparing to rebalance group \{consumer-group-name} in
> state PreparingRebalance with old generation 24781 (__consumer_offsets-29)
> (reason: Updating metadata for static member {} with instance id {}; client
> reason: rebalance failed due to UnjoinedGroupException)
> We also tried to remove the GROUP_INSTANCE_ID_CONFIG but then see these logs
> and rebalancing and no processing still
> sessionTimeoutMs=45000, rebalanceTimeoutMs=1800000,
> supportedProtocols=List(stream)) has left group \{groupId} through explicit
> `LeaveGroup`; client reason: the consumer unsubscribed from all topics
> (kafka.coordinator.group.GroupCoordinator)
> other logs show:
> during Stable; client reason: need to revoke partitions and re-join)
> client reason: triggered followup rebalance scheduled for 0
> On the application logs we see:
> 1. state being restored from changelog topic
> 2. INFO org.apache.kafka.streams.processor.internals.StreamThread -
> stream-thread at state RUNNING: partitions lost due to missed rebalance.
> 3. Task Migrated exceptions
> org.apache.kafka.streams.errors.TaskMigratedException: Error encountered
> sending record to topic
> org.apache.kafka.common.errors.InvalidProducerEpochException: Producer with
> transactionalId
> attempted to produce with an old epoch
> Written offsets would not be recorded and no more records would be sent since
> the producer is fenced, indicating the task may be migrated out; it means all
> tasks belonging to this thread should be migrated.
> at
> org.apache.kafka.streams.processor.internals.RecordCollectorImpl.recordSendError(RecordCollectorImpl.java:306)
> ~[kafka-streams-3.8.0.jar:?]
> at
> org.apache.kafka.streams.processor.internals.RecordCollectorImpl.lambda$send$1(RecordCollectorImpl.java:286)
> ~[kafka-streams-3.8.0.jar:?]
> at
> datadog.trace.instrumentation.kafka_clients.KafkaProducerCallback.onCompletion(KafkaProducerCallback.java:44)
> ~[?:?]
> at
> org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:1106)
> ~[kafka-clients-3.8.0.jar:?]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
