dajac commented on code in PR #18756:
URL: https://github.com/apache/kafka/pull/18756#discussion_r1936945456
##########
docs/upgrade.html:
##########
@@ -89,6 +89,27 @@ <h5><a id="upgrade_400_notable"
href="#upgrade_400_notable">Notable changes in 4
</li>
<li><b>Broker</b>
<ul>
+ <li>The new Consumer Group Rebalance Protocol is now a
production ready feature, enabled by default on the broker.
Review Comment:
nit: Should we add `(KIP-848)` after `Consumer Group Rebalance Protocol` and
link the KIP?
##########
docs/upgrade.html:
##########
@@ -89,6 +89,27 @@ <h5><a id="upgrade_400_notable"
href="#upgrade_400_notable">Notable changes in 4
</li>
<li><b>Broker</b>
<ul>
+ <li>The new Consumer Group Rebalance Protocol is now a
production ready feature, enabled by default on the broker.
+ The supported group protocols are defined with the
config <code>group.coordinator.rebalance.protocols=classic,consumer</code>,
Review Comment:
We may want to explain the `group.version` feature flag. It will be
automatically set when the upgrade to 4.0 is finalized with the
`kafka-features` command. Until this is done, the new protocol is not enabled.
##########
docs/upgrade.html:
##########
@@ -89,6 +89,27 @@ <h5><a id="upgrade_400_notable"
href="#upgrade_400_notable">Notable changes in 4
</li>
<li><b>Broker</b>
<ul>
+ <li>The new Consumer Group Rebalance Protocol is now a
production ready feature, enabled by default on the broker.
+ The supported group protocols are defined with the
config <code>group.coordinator.rebalance.protocols=classic,consumer</code>,
+ with both protocols supported. Client applications
should select the protocol to use (see Consumer section below).
+ <ul>
+ <li>
+ The supported partition assignment
strategies are defined on the broker now, via the config
<code>group.consumer.assignors</code>.
+ It defaults to
<code>org.apache.kafka.coordinator.group.assignor.UniformAssignor</code>, so
this will be the assignor used unless the consumer selects a different one
(Consumer section below).
+ See <a
href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=217387038#KIP848:TheNextGenerationoftheConsumerRebalanceProtocol-AssignorSelection";>Assignor
selection</a> for more details.
+ </li>
Review Comment:
I wonder if we should put a bullet point about the new server side configs
(session, heartbeat interval). What do you think?
##########
docs/upgrade.html:
##########
@@ -89,6 +89,27 @@ <h5><a id="upgrade_400_notable"
href="#upgrade_400_notable">Notable changes in 4
</li>
<li><b>Broker</b>
<ul>
+ <li>The new Consumer Group Rebalance Protocol is now a
production ready feature, enabled by default on the broker.
+ The supported group protocols are defined with the
config <code>group.coordinator.rebalance.protocols=classic,consumer</code>,
+ with both protocols supported. Client applications
should select the protocol to use (see Consumer section below).
+ <ul>
+ <li>
+ The supported partition assignment
strategies are defined on the broker now, via the config
<code>group.consumer.assignors</code>.
+ It defaults to
<code>org.apache.kafka.coordinator.group.assignor.UniformAssignor</code>, so
this will be the assignor used unless the consumer selects a different one
(Consumer section below).
+ See <a
href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=217387038#KIP848:TheNextGenerationoftheConsumerRebalanceProtocol-AssignorSelection";>Assignor
selection</a> for more details.
+ </li>
+ <li>
+ Static membership behaviour changes to
improve the experience and avoid conflicting members continuously affecting
each other.
+ A static consumer trying to join a group
with an instance ID that is already use will now fail with
<code>org.apache.kafka.common.errors.UnreleasedInstanceIdException</code>.
+ The initial static consumer using the
instance ID is not affected.
+ </li>
+ <li>
+ A new config
<code>group.coordinator.threads</code> is added to control the number of
threads used by the new group coordinator.
+ It defaults to 1, and it should be
adjusted as needed.
+ See <a
href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=217387038#KIP848:TheNextGenerationoftheConsumerRebalanceProtocol-GroupCoordinator";>New
Group Coordinator</a> for more details.
+ </li>
+ </ul>
Review Comment:
I wonder wether we should bring this as its own bullet point before the one
about the new rebalance protocol because this is important regardless of the
new protocol. What do you think? We could perhaps explain that 4.0 ships with a
new group coordinator, etc.
##########
docs/upgrade.html:
##########
@@ -89,6 +89,27 @@ <h5><a id="upgrade_400_notable"
href="#upgrade_400_notable">Notable changes in 4
</li>
<li><b>Broker</b>
<ul>
+ <li>The new Consumer Group Rebalance Protocol is now a
production ready feature, enabled by default on the broker.
+ The supported group protocols are defined with the
config <code>group.coordinator.rebalance.protocols=classic,consumer</code>,
+ with both protocols supported. Client applications
should select the protocol to use (see Consumer section below).
+ <ul>
+ <li>
+ The supported partition assignment
strategies are defined on the broker now, via the config
<code>group.consumer.assignors</code>.
Review Comment:
Should we mention the two assignors configured by default?
##########
docs/upgrade.html:
##########
@@ -89,6 +89,27 @@ <h5><a id="upgrade_400_notable"
href="#upgrade_400_notable">Notable changes in 4
</li>
<li><b>Broker</b>
<ul>
+ <li>The new Consumer Group Rebalance Protocol is now a
production ready feature, enabled by default on the broker.
+ The supported group protocols are defined with the
config <code>group.coordinator.rebalance.protocols=classic,consumer</code>,
+ with both protocols supported. Client applications
should select the protocol to use (see Consumer section below).
+ <ul>
+ <li>
+ The supported partition assignment
strategies are defined on the broker now, via the config
<code>group.consumer.assignors</code>.
+ It defaults to
<code>org.apache.kafka.coordinator.group.assignor.UniformAssignor</code>, so
this will be the assignor used unless the consumer selects a different one
(Consumer section below).
+ See <a
href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=217387038#KIP848:TheNextGenerationoftheConsumerRebalanceProtocol-AssignorSelection";>Assignor
selection</a> for more details.
+ </li>
+ <li>
+ Static membership behaviour changes to
improve the experience and avoid conflicting members continuously affecting
each other.
+ A static consumer trying to join a group
with an instance ID that is already use will now fail with
<code>org.apache.kafka.common.errors.UnreleasedInstanceIdException</code>.
+ The initial static consumer using the
instance ID is not affected.
+ </li>
+ <li>
+ A new config
<code>group.coordinator.threads</code> is added to control the number of
threads used by the new group coordinator.
+ It defaults to 1, and it should be
adjusted as needed.
+ See <a
href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=217387038#KIP848:TheNextGenerationoftheConsumerRebalanceProtocol-GroupCoordinator";>New
Group Coordinator</a> for more details.
+ </li>
+ </ul>
+ </li>
Review Comment:
I wonder if we should put a few words about the downgrade path after the new
protocol is enabled and used. What do you think?
##########
docs/upgrade.html:
##########
@@ -207,6 +228,38 @@ <h5><a id="upgrade_400_notable"
href="#upgrade_400_notable">Notable changes in 4
</li>
<li><b>Consumer</b>
<ul>
+ <li>New Consumer Group Rebalance Protocol
+ <ul>
+ <li>Client support for the new Consumer Group
Rebalance Protocol is now production ready.
+ To use it, set the new consumer config
<code>group.protocol=CONSUMER</code>.
+ This will make the consumer benefit from
the next generation of the rebalance protocol
+ (see <a
href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-848%3A+The+Next+Generation+of+the+Consumer+Rebalance+Protocol";>KIP-848</a>),
+ and also the improved consumer threading
model.
+ </li>
+ <li>
+ <code>group.remote.assignor</code> is
introduced as an optional config, supported only if using the Consumer protocol.
+ It allows clients to overwrite the default
assignment strategy configured on the broker.
+ </li>
+ <li>The following are not supported if using
the new Consumer protocol:
+ <ul>
+
<li><code>heartbeat.interval.ms</code></li>
+
<li><code>session.timeout.ms</code></li>
+
<li><code>partition.assignment.strategy</code></li>
+
<li><code>enforceRebalance(String)</code> and
<code>enforceRebalance()</code></li>
+ All continue to be supported if using
the Classic protocol.
+ </ul>
+ </li>
+ <li>
+ Support for RE2J regex subscription with
broker-side resolution is added to the Consumer. It can be used via the newly
added subscribe functions:
+
<code>subscribe(SubscriptionPattern)</code> and
<code>subscribe(SubscriptionPattern, ConsumerRebalanceListener)</code>.
+ A command line tool option has been added
to validate RE2J regular expressions to be provided as
<code>SubscriptionPattern</code>. See <a
href="https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=217387038#KIP848:TheNextGenerationoftheConsumerRebalanceProtocol---validate-regex";>RE2J
Validation Tool</a>
+ </li>
+ <li>
+ New metrics have been added to the
Consumer when using the new rebalance protocol, mainly providing visibility
over the improved threading model. See <a
href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-1068%3A+New+metrics+for+the+new+KafkaConsumer";>New
Consumer Metrics</a>
+ </li>
+ </ul>
Review Comment:
* Should we mention the upgrade path too?
* Should we mention the limitations (e.g. client side assignors are not
supported)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
