Lucas Brutschy created KAFKA-19044:
--------------------------------------
Summary: Handle tasks that are not present in the current topology
Key: KAFKA-19044
URL: https://issues.apache.org/jira/browse/KAFKA-19044
Project: Kafka
Issue Type: Sub-task
Reporter: Lucas Brutschy
A heartbeat might be sent to the group coordinator, claiming to own tasks that
we do not know about. This may happen for a variety of reasons. We need some
logic to handle those requests. In KIP-1071, we propose to return
`INVALID_REQUEST` error whenever this happens, effectively letting the clients
crash. This is safe, and may be a valid approach for the first version of the
protocol.
This behavior will, however, make topology updates impossible. [~cadonna]
proposed to only check that owned tasks match our set of expected tasks if the
topology epochs between the group and the client match. However, even if we let
clients crash only when they use the same topology epoch, that would mean that
scaling down the number of input partitions for a stateless application would
become impossible without crashing the applications.
Long term, I am wondering if it isn't a better course of action to just ignore
tasks that we do not expect. After all, the client is just saying what he
thinks he owns. If we'd just ignore those tasks, the next target assignment
would not include the tasks, so the client would automatically be instructed to
revoke whatever tasks it thinks it currently has.
The aim of this ticket is to implement a check and a behavior for the first
version of the protocol, and create follow-up ticket(s) for any extensions for
partition scaling and topology updating that we may want to add.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
