[
https://issues.apache.org/jira/browse/KAFKA-20167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lucas Brutschy updated KAFKA-20167:
-----------------------------------
Description:
Currently, {{KafkaStreams}} allows users to specify how to close the
application using {{{}CloseOptions{}}}. The existing options are
{{LEAVE_GROUP}} and {{{}REMAIN_IN_GROUP{}}}.
The default behavior (when no options are passed) is {{{}REMAIN_IN_GROUP{}}}.
However, with the introduction of the Streams Protocol (KIP-1071), the behavior
of {{REMAIN_IN_GROUP}} has become inconsistent:
* *Classic Protocol:* The member actually remains in the group.
* *Streams Protocol (KIP-1071):* The member leaves the group unconditionally
(as staying in the group for non-static member provides no benefits).
To resolve this ambiguity and provide a cleaner API for the long term, we
should introduce a new enum value: {{{}CloseOptions.DEFAULT{}}}. This allows
the "default" behavior to be adaptive based on the underlying protocol, while
strictly reserving {{REMAIN_IN_GROUP}} for when the user explicitly requests
that intent.
h3. *Proposed Changes*
*1. Update {{org.apache.kafka.streams.CloseOptions}}*
Add a new enum value {{{}DEFAULT{}}}.
*2. Update {{KafkaStreams.close()}} Behavior*
* *Current:* {{KafkaStreams.close()}} calls
{{{}close(CloseOptions.REMAIN_IN_GROUP){}}}.
* *Proposed:* {{KafkaStreams.close()}} calls
{{{}close(CloseOptions.DEFAULT){}}}.
*3. Define Logic for {{CloseOptions.DEFAULT}}*
The new {{DEFAULT}} option should act as a "smart" selection that adheres to
the best practices of the specific protocol version being used:
* *If Classic Protocol:* Map to Consumer {{CloseOptions.REMAIN_IN_GROUP}}
(Preserves legacy behavior).
* *If Streams Protocol:* Map to Consumer {{CloseOptions.DEFAULT}} (Typically
leaves the group for dynamic members).
*4. Redefine {{CloseOptions.REMAIN_IN_GROUP}}*
Currently, {{REMAIN_IN_GROUP}} is the implicit default. It should become an
explicit instruction.
* It should always map to Consumer {{{}CloseOptions.REMAIN_IN_GROUP{}}},
regardless of the protocol.
*5. Implement Consumer {{CloseOptions.REMAIN_IN_GROUP}} support in
KIP-1071{{{}{}}}*
We should implementation the missing support for *{{REMAIN_IN_GROUP}}* in the
StreamsGroupHeartbeatManager
h3. *Behavior Matrix*
|*Streams CloseOption*|*Classic Protocol Behavior*|*Streams Protocol (KIP-1071)
Behavior*|
|*DEFAULT* (New)|Uses Consumer {{REMAIN}}|Uses Consumer {{DEFAULT}}|
|*REMAIN_IN_GROUP*|Uses Consumer {{REMAIN}}|Uses Consumer {{REMAIN}}|
|*LEAVE_GROUP*|Uses Consumer {{LEAVE}}|Uses Consumer {{LEAVE}}|
h3. *Motivation*
* *Clarity:* {{REMAIN_IN_GROUP}} should mean exactly what it says.
* *Future Proofing:* By using {{{}DEFAULT{}}}, we allow the "standard" close
behavior to evolve with future protocols (e.g., KIP-848/1071 evolutions)
without breaking users who explicitly asked to {{{}REMAIN_IN_GROUP{}}}.
h3. *Compatibility*
* This requires a KIP as it changes the {{CloseOptions}} public API.
* The default behavior for Classic Protocol users remains unchanged (they
still remain in the group).
* The default behavior for Streams Protocol users becomes cleaner, mapping to
the Consumer's default rather than a forced mismatch.
was:
Currently, {{KafkaStreams}} allows users to specify how to close the
application using {{{}CloseOptions{}}}. The existing options are
{{LEAVE_GROUP}} and {{{}REMAIN_IN_GROUP{}}}.
The default behavior (when no options are passed) is {{{}REMAIN_IN_GROUP{}}}.
However, with the introduction of the Streams Protocol (KIP-1071), the behavior
of {{REMAIN_IN_GROUP}} has become inconsistent:
* *Classic Protocol:* The member actually remains in the group.
* *Streams Protocol (KIP-1071):* The member leaves the group unconditionally
(as staying in the group for non-static member provides no benefits).
To resolve this ambiguity and provide a cleaner API for the long term, we
should introduce a new enum value: {{{}CloseOptions.DEFAULT{}}}. This allows
the "default" behavior to be adaptive based on the underlying protocol, while
strictly reserving {{REMAIN_IN_GROUP}} for when the user explicitly requests
that intent.
h3. *Proposed Changes*
*1. Update {{org.apache.kafka.streams.CloseOptions}}*
Add a new enum value {{{}DEFAULT{}}}.
*2. Update {{KafkaStreams.close()}} Behavior*
* *Current:* {{KafkaStreams.close()}} calls
{{{}close(CloseOptions.REMAIN_IN_GROUP){}}}.
* *Proposed:* {{KafkaStreams.close()}} calls
{{{}close(CloseOptions.DEFAULT){}}}.
*3. Define Logic for {{CloseOptions.DEFAULT}}*
The new {{DEFAULT}} option should act as a "smart" selection that adheres to
the best practices of the specific protocol version being used:
* *If Classic Protocol:* Map to Consumer {{CloseOptions.REMAIN_IN_GROUP}}
(Preserves legacy behavior).
* *If Streams Protocol:* Map to Consumer {{CloseOptions.DEFAULT}} (Typically
leaves the group for dynamic members).
*4. Redefine {{CloseOptions.REMAIN_IN_GROUP}}*
Currently, {{REMAIN_IN_GROUP}} is the implicit default. It should become an
explicit instruction.
* It should always map to Consumer {{{}CloseOptions.REMAIN_IN_GROUP{}}},
regardless of the protocol.
* _Note:_ Even if the Streams protocol currently forces a leave, passing this
explicit option ensures the intent is passed down to the consumer client,
rather than being intercepted by Streams logic.
h3. *Behavior Matrix*
|*Streams CloseOption*|*Classic Protocol Behavior*|*Streams Protocol (KIP-1071)
Behavior*|
|*DEFAULT* (New)|Uses Consumer {{REMAIN}}|Uses Consumer {{DEFAULT}}|
|*REMAIN_IN_GROUP*|Uses Consumer {{REMAIN}}|Uses Consumer {{REMAIN}}|
|*LEAVE_GROUP*|Uses Consumer {{LEAVE}}|Uses Consumer {{LEAVE}}|
h3. *Motivation*
* *Clarity:* {{REMAIN_IN_GROUP}} should mean exactly what it says. If the
underlying protocol cannot support it, it is better for the Consumer client to
handle that rejection/fallback rather than Kafka Streams masking it.
* *Future Proofing:* By using {{{}DEFAULT{}}}, we allow the "standard" close
behavior to evolve with future protocols (e.g., KIP-848/1071 evolutions)
without breaking users who explicitly asked to {{{}REMAIN_IN_GROUP{}}}.
h3. *Compatibility*
* This requires a KIP as it changes the {{CloseOptions}} public API.
* The default behavior for Classic Protocol users remains unchanged (they
still remain in the group).
* The default behavior for Streams Protocol users becomes cleaner, mapping to
the Consumer's default rather than a forced mismatch.
> Introduce CloseOptions.DEFAULT for Kafka Streams
> ------------------------------------------------
>
> Key: KAFKA-20167
> URL: https://issues.apache.org/jira/browse/KAFKA-20167
> Project: Kafka
> Issue Type: Task
> Components: streams
> Reporter: Lucas Brutschy
> Priority: Major
> Labels: kip, kip1071
>
> Currently, {{KafkaStreams}} allows users to specify how to close the
> application using {{{}CloseOptions{}}}. The existing options are
> {{LEAVE_GROUP}} and {{{}REMAIN_IN_GROUP{}}}.
> The default behavior (when no options are passed) is {{{}REMAIN_IN_GROUP{}}}.
> However, with the introduction of the Streams Protocol (KIP-1071), the
> behavior of {{REMAIN_IN_GROUP}} has become inconsistent:
> * *Classic Protocol:* The member actually remains in the group.
> * *Streams Protocol (KIP-1071):* The member leaves the group unconditionally
> (as staying in the group for non-static member provides no benefits).
> To resolve this ambiguity and provide a cleaner API for the long term, we
> should introduce a new enum value: {{{}CloseOptions.DEFAULT{}}}. This allows
> the "default" behavior to be adaptive based on the underlying protocol, while
> strictly reserving {{REMAIN_IN_GROUP}} for when the user explicitly requests
> that intent.
> h3. *Proposed Changes*
> *1. Update {{org.apache.kafka.streams.CloseOptions}}*
> Add a new enum value {{{}DEFAULT{}}}.
> *2. Update {{KafkaStreams.close()}} Behavior*
> * *Current:* {{KafkaStreams.close()}} calls
> {{{}close(CloseOptions.REMAIN_IN_GROUP){}}}.
> * *Proposed:* {{KafkaStreams.close()}} calls
> {{{}close(CloseOptions.DEFAULT){}}}.
> *3. Define Logic for {{CloseOptions.DEFAULT}}*
> The new {{DEFAULT}} option should act as a "smart" selection that adheres to
> the best practices of the specific protocol version being used:
> * *If Classic Protocol:* Map to Consumer {{CloseOptions.REMAIN_IN_GROUP}}
> (Preserves legacy behavior).
> * *If Streams Protocol:* Map to Consumer {{CloseOptions.DEFAULT}} (Typically
> leaves the group for dynamic members).
> *4. Redefine {{CloseOptions.REMAIN_IN_GROUP}}*
> Currently, {{REMAIN_IN_GROUP}} is the implicit default. It should become an
> explicit instruction.
> * It should always map to Consumer {{{}CloseOptions.REMAIN_IN_GROUP{}}},
> regardless of the protocol.
> *5. Implement Consumer {{CloseOptions.REMAIN_IN_GROUP}} support in
> KIP-1071{{{}{}}}*
> We should implementation the missing support for *{{REMAIN_IN_GROUP}}* in the
> StreamsGroupHeartbeatManager
> h3. *Behavior Matrix*
> |*Streams CloseOption*|*Classic Protocol Behavior*|*Streams Protocol
> (KIP-1071) Behavior*|
> |*DEFAULT* (New)|Uses Consumer {{REMAIN}}|Uses Consumer {{DEFAULT}}|
> |*REMAIN_IN_GROUP*|Uses Consumer {{REMAIN}}|Uses Consumer {{REMAIN}}|
> |*LEAVE_GROUP*|Uses Consumer {{LEAVE}}|Uses Consumer {{LEAVE}}|
> h3. *Motivation*
> * *Clarity:* {{REMAIN_IN_GROUP}} should mean exactly what it says.
> * *Future Proofing:* By using {{{}DEFAULT{}}}, we allow the "standard" close
> behavior to evolve with future protocols (e.g., KIP-848/1071 evolutions)
> without breaking users who explicitly asked to {{{}REMAIN_IN_GROUP{}}}.
> h3. *Compatibility*
> * This requires a KIP as it changes the {{CloseOptions}} public API.
> * The default behavior for Classic Protocol users remains unchanged (they
> still remain in the group).
> * The default behavior for Streams Protocol users becomes cleaner, mapping
> to the Consumer's default rather than a forced mismatch.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
