Michael Arndt created KAFKA-9043:
------------------------------------
Summary: Problem committing offsets when using consumer subscribe
and assign method on different topics with same group.id
Key: KAFKA-9043
URL: https://issues.apache.org/jira/browse/KAFKA-9043
Project: Kafka
Issue Type: Bug
Components: consumer
Affects Versions: 2.3.0
Environment: not relevant
Reporter: Michael Arndt
I am using a consumer with group.id "myGroup" on an *_empty_* topic "election"
with 24 partitions in subscribe mode. The poll loop is a noop and is for
rebalancing only. On new assigments (only the partitions are relevant) I start
two consumers per assigned partition on other topics ("foo" and "bar") using
assignments on a specific partition/topic.
Summary:
consumer1: election-0, election-1 (subscribe)
consumer2: foo-0 (assign)
consumer3: bar-0 (assign)
consumer4: foo-1 (assign)
consumer5: bar-1 (assign)
The reason for this setup is a join on time and a certain key in the topics foo
and bar, where the content in one partition is complete for the key.
However, consumer1 never commits since the topic is empty and just for
rebalancing issues that will guarantee that I will always read from the same
partitions from foo and bar in this service instance.
When trying to commit consumer2-4 (they are using the same group-id as
consumer1) the ClientCoordinator reports an error telling the consumer is not
responsible for this topic and partition. This is an error raised at the server.
When using different group-ids for consumer1 and 2-4 the commit works fine.
This behaviour is somehow reasonable since the server only knows about the
subscription, but since assignment and subscription run on different topics
it's at least a lack in documentation.
As this will also happen when using assign and subscribe from different clients
on differnt topis one would not be able to use a common group-id for
identification issues (e.g. team related; useful when working with ACLs in an
environment where on kafka serves many teams and you want to limit the number
of group ids managed in the ACLs).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
