Lucas Brutschy created KAFKA-20170:
--------------------------------------
Summary: Support topology updates for streams groups without
requiring a new group
Key: KAFKA-20170
URL: https://issues.apache.org/jira/browse/KAFKA-20170
Project: Kafka
Issue Type: Task
Components: streams, group-coordinator
Reporter: Lucas Brutschy
The streams rebalance protocol \(KIP\-1071\) does not currently support
topology updates. If a topology is changed significantly \(e.g., by adding new
source topics or changing the number of subtopologies\), users must create a
new streams group. This is a significant limitation for applications that need
to evolve over time.
As specified in KIP\-1071, topology updates should be supported through
explicit versioning via a topology.epoch configuration. When a member joins
with a different topology, the broker should compare epochs: if the epoch is
the same but metadata differs, return STREAMS\_INVALID\_TOPOLOGY\_EPOCH; if the
epoch is lower, return STREAMS\_TOPOLOGY\_FENCED; if the epoch is bumped by
one, accept and update the group topology. Members with stale topologies should
receive a STALE\_TOPOLOGY status in heartbeat responses but continue processing
with their current tasks until upgraded. This enables rolling upgrades of the
topology across application instances.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
