[
https://issues.apache.org/jira/browse/KAFKA-12821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prakash Patel updated KAFKA-12821:
----------------------------------
Description:
I have logstash ( Java Kafka Consumer ) running on Kubernetes and consuming
events from 100+ topics.Each topic have 3+ partitions. Thus Single consumer
group subscribed to 100+ topics and consumes events from 500+ partitions. Below
are the Kafka consumers configuration.I am running 10 instances of logstash on
kubernetes so total 100 consumer threads consuming from 500 partitions.
Each consumer threads is assigned to 5+ topics/partitions. No of incoming
events to partition varies a lot.
{quote}auto_offset_reset => "earliest"
enable_auto_commit => "true"
group_id => "test_logstash"
consumer_threads => "10"
max_poll_records => "500"
heartbeat_interval_ms => "9000"
session_timeout_ms => "30000"
fetch_max_bytes => "10485760"
max_partition_fetch_bytes => "524288"
client_id => "test_logstash"
decorate_events => true
partition_assignment_strategy =>
"org.apache.kafka.clients.consumer.RoundRobinAssignor"
{quote}
intermittently Kafka consumer stops reading from single partition although its
subscribed and assigned to consumer group.
There is no rebalance or any error or info message Kafka consumer throws on
client side.
Restating the consumer solve the problem. This is intermittent issue and it can
happen with any partition of any topic consumer is subscribed/assigned.
was:
I have logstash ( Java Kafka Consumer ) running on Kubernetes and consuming
events from 100+ topics.Each topic have 3+ partitions. Thus Single consumer
group subscribed to 100+ topics and consumes events from 500+ partitions. Below
are the Kafka consumers configuration.I am running 10 instances of logstash on
kubernetes so total 100 consumer threads consuming from 500 partitions.
{quote}auto_offset_reset => "earliest"
enable_auto_commit => "true"
group_id => "test_logstash"
consumer_threads => "10"
max_poll_records => "500"
heartbeat_interval_ms => "9000"
session_timeout_ms => "30000"
fetch_max_bytes => "10485760"
max_partition_fetch_bytes => "524288"
client_id => "test_logstash"
decorate_events => true
partition_assignment_strategy =>
"org.apache.kafka.clients.consumer.RoundRobinAssignor"
{quote}
intermittently Kafka consumer stops reading from single partition although its
subscribed and assigned to consumer group.
There is no rebalance or any error or info message Kafka consumer throws on
client side.
Restating the consumer solve the problem. This is intermittent issue and it can
happen with any partition of any topic consumer is subscribed/assigned.
> Kafka consumer skip reading from single partition
> --------------------------------------------------
>
> Key: KAFKA-12821
> URL: https://issues.apache.org/jira/browse/KAFKA-12821
> Project: Kafka
> Issue Type: Bug
> Components: clients
> Affects Versions: 2.3.0
> Reporter: Prakash Patel
> Priority: Major
>
> I have logstash ( Java Kafka Consumer ) running on Kubernetes and consuming
> events from 100+ topics.Each topic have 3+ partitions. Thus Single consumer
> group subscribed to 100+ topics and consumes events from 500+ partitions.
> Below are the Kafka consumers configuration.I am running 10 instances of
> logstash on kubernetes so total 100 consumer threads consuming from 500
> partitions.
> Each consumer threads is assigned to 5+ topics/partitions. No of incoming
> events to partition varies a lot.
>
> {quote}auto_offset_reset => "earliest"
> enable_auto_commit => "true"
> group_id => "test_logstash"
> consumer_threads => "10"
> max_poll_records => "500"
> heartbeat_interval_ms => "9000"
> session_timeout_ms => "30000"
> fetch_max_bytes => "10485760"
> max_partition_fetch_bytes => "524288"
> client_id => "test_logstash"
> decorate_events => true
> partition_assignment_strategy =>
> "org.apache.kafka.clients.consumer.RoundRobinAssignor"
> {quote}
> intermittently Kafka consumer stops reading from single partition although
> its subscribed and assigned to consumer group.
> There is no rebalance or any error or info message Kafka consumer throws on
> client side.
> Restating the consumer solve the problem. This is intermittent issue and it
> can happen with any partition of any topic consumer is subscribed/assigned.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
