Hi Jiunn-Yang, Thanks for your response. AS1-AS6: lgtm
AS7: I'll have one last try at persuasion here, but I expect I won't succeed. In KIP-1274, the default group protocol for KafkaConsumer is changing to "consumer". It is true that "classic" is still supported, but users will need to explicitly enable it by setting "group.protocol=classic". I think this means we could change the default for "auto.offset.reset" to "by_start_time" in AK 5.0. Whether we want to do it is a different matter. AS8, AS9: lgtm AS10: I'm not a fan of the "Epoch-ms" wording, particularly in the Admin API javadoc. I would just use "The creation time of the group." or similar. AS11: I suspect you need to change StreamsGroupHeartbeatResponse too. Probably need a Kafka Streams expert to weigh in how `auto.offset.reset=by_start_time` would work for the consumer in a streams group. Thanks, Andrew On 2026/04/03 01:18:02 黃竣陽 wrote: > Hello all, > > AS4 > > I agree that bumping the version for request/response protocols provides a > meaningful benefit: it allows the consumer to distinguish between "the broker > does not support this feature" and "the broker supports it, but the group has > no > creation time." This enables more actionable error messages. > > For __consumer_offsets records (ConsumerGroupMetadataValue and > StreamsGroupMetadataValue), I will keep tagged fields. These records have no > version > negotiation. They are written and read internally by the broker. Tagged > fields allow old brokers > to safely ignore unknown tags during log replay, and avoid the need to > rewrite existing records > on upgrade. > > Best Regards, > Jiunn-Yang > > > Chia-Ping Tsai <[email protected]> 於 2026年4月3日 上午8:01 寫道： > > > > AS4 > > > > Bumping the version actually provides a significant benefit. It allows the > > new consumer to offer more actionable error messages to users, because it > > can clearly distinguish between "the broker is too old to support this > > feature" and "the broker supports it, but the group has no creation time." > > > > Furthermore, by bumping the version, the auto-generated protocol code can > > gracefully handle the serialization (e.g., ignoring the field if the > > client/broker is communicating with an older version). > > > > On 2026/04/02 15:04:39 黃竣陽 wrote: > >> Hi Andrew, > >> > >> Thanks for the thorough review. > >> > >> AS1: > >> The updated KIP now explicitly addresses these scenarios: > >> - Configuring auto.offset.reset=by_start_time with a classic consumer > >> group throws > >> ConfigException at startup. > >> - Classic groups upgraded to modern groups cannot use by_start_time > >> immediately, > >> because the group creation timestamp was not recorded at the time the > >> group was originally > >> created. The consumer throws `GroupCreationTimeNotAvailableException` in > >> this case. > >> Users must delete and recreate the group to obtain a fresh creation > >> timestamp. > >> > >> AS2, AS3, AS5: Updated the KIP accordingly. > >> > >> AS4: > >> > >> Tagged fields fit well because GroupCreationTimeMs / CreationTimeMs is > >> optional, > >> with a default value of -1 ("unknown") that has clear semantics, > >> especially for pre-existing > >> groups. They are only serialized when non-default, so they add no > >> overhead. Older > >> brokers simply ignore unknown tags when replaying __consumer_offsets, > >> requiring no > >> record rewriting during upgrades. > >> > >> In contrast, a version bump would require expanding validVersions, adding > >> version checks > >> across read/write paths, and provides no real benefit given the field’s > >> optional nature and > >> well-defined default. This follows Kafka’s convention: use tagged fields > >> for optional additions > >> with meaningful defaults, and version bumps for mandatory or semantic > >> changes. > >> > >> AS6: > >> > >> Thanks for the clarification. I’ve incorporated this into the KIP. > >> > >> AS7: > >> > >> I agree that by_start_time better matches user expectations for the > >> default offset reset policy. > >> However, since it is not supported by classic consumer groups and would > >> raise a ConfigException > >> at startup, changing the default is only safe after the classic protocol > >> is removed from KafkaConsumer. > >> Per KIP-1274, classic protocol support will be deprecated in Kafka 6.0. > >> After that, the > >> default can be changed from latest to by_start_time. > >> > >> AS8: > >> > >> InvalidOffsetException is an abstract class that requires subclasses to > >> implement partitions(), > >> implying the error is tied to specific partitions with invalid offsets. > >> > >> GroupCreationTimeNotAvailableException does not fit this model. The root > >> cause is a missing > >> group-level timestamp, not an issue with any particular partition. > >> Implementing partitions() > >> would require returning an empty or artificial set, which violates the > >> API’s semantics. > >> > >> Therefore, I believe extending KafkaException is the better choice. > >> > >> AS9: > >> > >> Renamed to GroupCreationTimeNotAvailableException throughout the KIP. This > >> better > >> describes the state ("not available") rather than the cause. > >> > >> Happy to hear any further feedback or suggestions. > >> > >> Best Regards, > >> Jiunn-Yang > >> > >> > >>> Andrew Schofield <[email protected]> 於 2026年4月2日 晚上8:34 寫道： > >>> > >>> Hi Jiunn-Yang, > >>> Thanks for this great KIP. Some comments. > >>> > >>> AS1: A limitation of this KIP is that it will not support classic > >>> consumer groups. I think this is not a problem and it will be a motivator > >>> for users to adopt modern consumer groups, but it will be necessary to > >>> test attempting to use the new policy with classic consumer groups and > >>> make sure that the ConsumerGroupDescription lacks the creation timestamp > >>> and so on. There is also the question of classic groups which are > >>> upgraded to modern consumer groups. > >>> AS2: It seems to me that you should also support streams groups, both in > >>> a StreamsGroupMetadataValue and StreamsGroupDescription. > >>> AS3: What you are describing as metadata log changes are actually records > >>> on the __consumer_offsets topic. > >>> AS4: While I understand the need to use tagged fields in the records, I > >>> do not see why you cannot bump the versions for ConsumerGroupHeartbeat > >>> and ConsumerGroupDescribe in order to support the new fields. > >>> AS5: In the Admin API, we already have an optional timestamp in > >>> TransactionDescription.transactionStartTimeMs which uses the type > >>> OptionalLong. Any reason not to use the same in ConsumerGroupDescription? > >>> AS6: I don't think share group support is needed. "latest" already starts > >>> at offset 0 for newly added partitions for share groups. > >>> AS7: Might you consider making "by_start_time" the default value for the > >>> auto.offset.reset consumer config in the next major release? I know > >>> that's a bold move, but I think what you're getting with the new option > >>> is what people expected "latest" to behave like. > >>> AS8: I wonder whether GroupCreationTimeUnknownException should extend the > >>> abstract InvalidOffsetException. > >>> AS9: To follow existing exception names, I think > >>> UnknownGroupCreationTimeException, GroupCreationTimeNotAvailableException > >>> (my favourite) or GroupCreationTimeNotFoundException would better match > >>> what we already have. wdyt? > >>> > >>> Thanks, > >>> Andrew > >>> > >>> On 2026/04/02 11:34:10 黃竣陽 wrote: > >>>> Hi chia, > >>>> > >>>> chia_13: > >>>> `ConsumerGroupDescription` now gains a new method > >>>> `groupCreationTimeMs()`: > >>>> > >>>> `groupCreationTimeMs()` (`Optional<Long>`): the epoch-ms when this > >>>> consumer group > >>>> was first created on the broker. `Optional.empty()` if unknown. > >>>> > >>>> Please take a look at the updated KIP and let us know if you have > >>>> further questions. > >>>> > >>>> Best Regards, > >>>> Jiunn-Yang > >>>> > >>>>> Chia-Ping Tsai <[email protected]> 於 2026年4月2日 上午9:58 寫道： > >>>>> > >>>>> hi Jiunn > >>>>> > >>>>> chia_13: should we update the public API: ConsumerGroupDescription as > >>>>> well? > >>>>> > >>>>> Best, > >>>>> Chia-Ping > >>>>> > >>>>> On 2026/04/01 11:59:46 黃竣陽 wrote: > >>>>>> Hi chia, > >>>>>> > >>>>>> Thank you for the thoughtful feedback! > >>>>>> > >>>>>> chia_10: I have added a clarification in the Proposed Changes section > >>>>>> to address this. In this case, > >>>>>> ListOffsetsRequest will resolve to the Log End Offset, but this is not > >>>>>> a fallback to `latest`. The distinction > >>>>>> matters: `latest` is a direct seek to the Log End Offset, whereas > >>>>>> `by_start_time` always anchors to the > >>>>>> group creation timestamp and lets the lookup result follow naturally. > >>>>>> The consumer will begin consuming > >>>>>> any new records produced after this point normally. > >>>>>> > >>>>>> chia_11: I have introduced a new exception > >>>>>> `GroupCreationTimeUnknownException` to signal > >>>>>> this specific condition. > >>>>>> > >>>>>> chia_12: I have updated the KIP to expose the group creation timestamp > >>>>>> via `AdminClient.describeConsumerGroups()` > >>>>>> , accessible through `ConsumerGroupDescription.groupCreationTimeMs()`. > >>>>>> This is backed by a new > >>>>>> `GroupCreationTimeMs` field in `ConsumerGroupDescribeResponse. > >>>>>> > >>>>>> Best Regards, > >>>>>> Jiunn-Yang > >>>>>> > >>>>>>> Chia-Ping Tsai <[email protected]> 於 2026年4月1日 下午2:33 寫道： > >>>>>>> > >>>>>>> hi Jiunn > >>>>>>> > >>>>>>> thanks for updating KIP. I have a couple of questions. > >>>>>>> > >>>>>>> chia_10: What happen if the group creation time is strictly greater > >>>>>>> than the timestamp of the latest record in the partition? Will the > >>>>>>> consumer fall back to the "latest" offset behaviour in this case? It > >>>>>>> could be good to elucidate that in the KIP > >>>>>>> > >>>>>>> chia_11: Could you specify which exact exception will be thrown when > >>>>>>> the group creation time is unknown > >>>>>>> > >>>>>>> chia_12: Since the group creation time is now a critical semantic > >>>>>>> anchor, should we also expose it via the `DescribeGroupsResponse`? > >>>>>>> > >>>>>>> Best, > >>>>>>> Chia-Ping > >>>>>>> > >>>>>>> On 2026/03/05 11:14:31 黃竣陽 wrote: > >>>>>>>> Hello everyone, > >>>>>>>> > >>>>>>>> I would like to start a discussion on KIP-1282: Prevent data loss > >>>>>>>> during partition expansion for dynamically added partitions > >>>>>>>> <https://cwiki.apache.org/confluence/x/mIY8G> > >>>>>>>> > >>>>>>>> This proposal aims to introduce a new auto.offset.reset policy > >>>>>>>> by_start_time, anchoring the > >>>>>>>> offset reset to the consumer's startup timestamp rather than > >>>>>>>> partition discovery time, to prevent > >>>>>>>> silent data loss during partition expansion. > >>>>>>>> > >>>>>>>> Best regards, > >>>>>>>> Jiunn-Yang > >>>>>> > >>>>>> > >>>> > >>>> > >> > >> > >
